Firewall Log Review and Analysis After the decision has been made to log events from your firewall, the next step is determining what you should be looking for in the logs and how you should properly perform log analysis. The most important thing to remember is that firewall logs are virtually worthless if no one ever looks at the logs. Logging is merely a means to an end, namely knowing what is going on with your firewalls so that you can respond accordingly. Review of the logs should not be reserved for only when an incident has occurred. It should be a part of the weekly, if not daily, tasks that the firewall administrators perform. To help reduce the time and effort required to review the logs, many of the enterprise security incident management products provide tools and utilities that assist the firewall administrator in separating the wheat from the chaff, allowing the firewall administrator to spend less time reviewing the logs, while still providing the information necessary to help identify situations before they become a problem. Another aspect of reviewing the logs that should not be overlooked is the need to define a log archive and normalization policy. Too many organizations do not store their firewall logs long enough to adhere to regulations (some of which such as Sarbanes-Oxley are generally accepted to require seven years of log data to be stored). This creates situations where data from the logs may be necessary, but the logs themselves have been destroyed. In conjunction with this, it is important to normalize your log data. Normalization just means converting your logs into a standard format that allows for easier review and correlation of data from different data sources (such as different firewall vendors). What to Look for in Firewall Logs After you have collected the firewall logs and begun the process of analyzing the logs, determine the data that you should be looking for in the logs. With that said, it is important to remember not to fall into the trap of looking in your firewall logs only for "bad" events. Yes, firewall logs can be the key element in discovering security incidents and compromises, but that is only one of the reasons for analyzing your logs. You also want to be able to use the log information to assist in defining the baselines and normal operations of the firewall. After all, one of the easiest ways to know whether behavior that has been logged is malicious is to know what the good things are and then note the exceptions. The simple fact of the matter is that certain events should always raise suspicion when they are detected. Ten of the most common events that warrant further investigation are as follows: • Authentication allowed. • Traffic dropped (not addressed to the firewall). • Firewall stop/start/restart. • Firewall configuration changed. • Interface up/down status changed. • Administrator access granted. • Connection was torn down. • Authentication failed. • Traffic dropped (addressed to the firewall). • Administrator session ended. The following sections explain these events in more detail. Authentication Allowed Although it may seem rather innocuous at first glance, it is important to look for authentication-allowed events because they can identify situations where access was granted by the firewall when it should not have been allowed. The reasons can range from legitimate administrators logging on when they should not have to malicious users logging on after compromising the account and password that they are using. In addition, if your firewall is configured to authenticate user access, this event can be used to identify users who have been authenticated for whatever function they are attempting to perform. Traffic Dropped (Not Addressed to the Firewall) Most firewalls will have some resources that they are protecting. Traffic addressed to these servers will typically be processed by the firewall and filtered accordingly. Although traffic-dropped messages can indicate that someone is attempting to access a protected resource in a manner other than what the firewall administrator has defined, a common cause of this event is a simple misconfiguration of the ruleset. Therefore, if users cannot access protected resources, it is important to review the logs to determine whether the firewall is dropping the traffic, thereby pointing you in the direction of what may need to be fixed to provide access to the resources requested. Firewall Stop/Start/Restart The firewall should never stop, start, or restart without the firewall administrator knowing in advance that the situation is going to occur. This event can be caused by non-firewall- specific issues such as power failures as well as by firewall-specific issues such as the firewall crashing or a high-availability failover, and therefore it should always be investigated in more detail to ascertain the root cause. Firewall Configuration Changed Almost all firewall configuration changes should be accompanied with the appropriate change control documentation. This event always warrants further investigation to ensure that the changes that were made are legitimate and in accordance with expected results. In fact, many SIM products can be configured to perform a comparison of the changed configuration against a "known good" configuration when a firewall configuration changed event occurs. In fact, some products such as NetIQ Security Manager can actually use that information to attempt to undo the changes that were made if they are found to be out of compliance with the known good configuration. Interface Up/Down Status Changed Firewall interfaces transitioning from an up to a down status and vice versa can indicate problems with the underlying network configuration. This information can prove particularly helpful in situations where redundant firewalls are implemented, because the network interfaces transitioning to a down state could cause the firewall failover process to occur. Administrator Access Granted Whenever administrator access is granted, the corresponding event should be investigated. Although similar to monitoring for authentication, in this case we are looking explicitly at gaining administrator access. Most likely the access is expected, and there is nothing suspicious or out of order that warrants further review. However, if that is not the case, this event rapidly becomes an extremely high-priority situation that must be investigated because the implication can be that an administrator account has been compromised. Connection Was Torn Down The termination of connections is a relatively routine process that is a part of normal communications. Where this event is particularly important, however, is in listing the reason why the connection was torn down. For example, the connection may have been torn down as a result of SYN timeout, which can be an indicator that someone is attempting to cause a denial of service, especially if there are a lot of events of that nature. In determining the cause of the connection tear down, it is important to review the firewall documentation for the teardown causes. For example, Cisco Secure PIX Firewall version 7.0 message ID 302014 lists the potential reasons for a TCP connection being torn down as shown in Table 12-3 . Table 12-3. TCP Connection Teardown Reasons Reason Description Conn-timeout Connection ended because it was idle longer than the configured idle timeout. Deny Terminate Flow was terminated by application inspection. Failover primary closed The standby unit in a failover pair deleted a connection because of a message received from the active unit FIN Timeout Force termination after 10 minutes awaiting the last ACK or after half-closed timeout. Flow closed by inspection Flow was terminated by inspection feature. Flow terminated by IPS Flow was terminated by IPS. Flow reset by IPS Flow was reset by IPS Flow terminated by TCP intercept Flow was terminated by TCP Intercept. Invalid SYN SYN packet not valid. Idle Timeout Connection timed out because it was idle longer than timeout value. IPS fail-close Flow was terminated due to IPS card down. SYN Control Back channel initiation from wrong side. SYN Timeout Force termination after 2 minutes awaiting three-way handshake completion. TCP bad retransmission Connection terminated because of bad TCP retransmission. TCP FINs Normal close-down sequence. TCP Invalid SYN Invalid TCP SYN packet. TCP Reset-I Reset was from the inside. TCP Reset-O Reset was from the outside. TCP segment partial overlap Detected a partially overlapping segment. TCP unexpected window size variation Connection terminated due to variation in the TCP window size. Tunnel has been torn down Flow terminated because tunnel is down. Unauth Deny Denied by URL filter. Unknown Catchall error. Xlate Clear Command-line removal As you can see, reasons such as "Unauth Deny" or "Flow closed by inspection" can be indicators of malicious traffic and warrant more concern and investigation than a reason such as "TCP ResetI" (which is a normal method of applications terminating their communications session). Authentication Failed Authentication-failed events can be indicators of everything from users making a typo when they enter their password to malicious users making a brute-force attack in an attempt to determine the password. Authentication-failed events should be examined in particular detail when the account in question is a privileged or administrator-level account. Traffic Dropped (Addressed to the Firewall) These events are similar to the traffic dropped that is not addressed to the firewall, with the obvious difference being that in this case the traffic is addressed to the firewall. As a general rule, the firewall should not have any traffic addressed directly to it on the external interface; instead, all traffic should be destined for the resources being protected by the firewall. These events can be indicators of malicious users attempting to gain access to the firewall or a misconfiguration of things such as ICMP, IPsec, or management or routing protocols and therefore should be investigated in more detail to determine the exact nature of why the traffic was dropped. Administrator Session Ended Similar to administrator access being granted, administrator sessions ending should be monitored to ensure that the administrator who had access was supposed to have access. This type of event can also be used as a time benchmark because only administrators should be able to make changes to the firewall, and therefore the logs should be investigated in more detail for the time preceding the administrator session ending to see exactly what commands may have been run. Cisco Secure PIX Firewall Syslog Event Baseline The following syslog events constitute a good baseline of events that should be monitored and paid careful attention to in most environments. In essence, this list is here to answer this question: What specific kinds of events should I look for? It is not meant to be an exhaustive list of all syslog message IDs or the only syslog message IDs that you should be filtering for. You can use this information to help build filtering rules for your particular logging softwarefor example, to identify the messages that administrators should get a page or e- mail notification over (for instance, message %PIX-3-201008) versus messages that can j ust be logged without any special notification occurring. This can be done by using the message ID (for example, %PIX-3-201008) in your logging software's filtering syntax/search strings. In general, every time Cisco releases a new version of software, syslog events are added/deleted from the list of events. Therefore, your particular version of software may or may not include all of these events, or it may have events that are not listed here. Obviously, not all events are relevant for all environments, but this list provides a sound starting point of events to be on the look out for, from which you can further customize to meet the logging requirements in your environment. This list can be easily modified to cover both the Cisco Adaptive Security Appliance (ASA) and Cisco Firewall Services Module (FWSM) by just replacing the %PIX syntax with either a %ASA or %FWSM, respectively (in fact, the log messages use %PIX|ASA to mean that either %PIX or %ASA can be used): • All severity level 1 messages (use the string %PIX|ASA-1 for the filter) • %PIX|ASA-2-106016: Deny IP spoof from (IP_address) to IP_address on interface interface_name • %PIX|ASA-2-106017: Deny IP due to Land Attack from IP_address to IP_address • %PIX|ASA-2-106018: ICMP packet type ICMP_type denied by outbound list acl_ID src inside_address dest outside_address • %PIX|ASA-2-106020: Deny IP teardrop fragment (size = number, offset = number) from IP_address to IP_address • %PIX|ASA-2-201003: Embryonic limit exceeded nconns/elimit for outside_address/outside_port (global_address) inside_address/inside_port on interface interface_name • %PIX|ASA-2-304007: URL Server IP_address not responding, ENTERING ALLOW mode. • %PIX|ASA-3-316001: Denied new tunnel to IP_address. VPN peer limit (platform_vpn_peer_limit) exceeded • %PIX|ASA-3-201002: Too many TCP connections on {static|xlate} global_address! econns nconns • %PIX|ASA-3-201004: Too many UDP connections on {static|xlate} global_address! udp connections limit • %PIX|ASA-3-201008: The PIX is disallowing new connections. • %PIX|ASA-3-201009: TCP connection limit of number for host IP_address on interface_name exceeded • %PIX|ASA-3-202001: Out of address translation slots! • %PIX|ASA-3-211001: Memory allocation error • %PIX|ASA-3-211003: CPU utilization for number seconds = percent • %PIX|ASA-3-302302: ACL = deny; no sa created • %PIX|ASA-3-304003: URL Server IP_address timed out URL url • %PIX|ASA-3-304006: URL Server IP_address not responding • %PIX|ASA-3-315004: Fail to establish SSH session because PIX RSA host key retrieval failed. • %PIX|ASA-3-317004: IP routing table limit warning • %PIX|ASA-3-322001: Deny MAC address MAC_address, possible spoof attempt on interface interface • %PIX|ASA-3-322002: ARP inspection check failed for arp {request|response} received from host MAC_address on interface interface. This host is advertising MAC Address MAC_address_1 for IP Address IP_address, which is {statically|dynamically} bound to MAC Address MAC_address_2. • %PIX|ASA-3-404102: ISAKMP: Exceeded embryonic limit • %PIX|ASA-3-407002: Embryonic limit nconns/elimit for through connections exceeded. outside_address/outside_port to global_address (inside_address)/inside_port on interface interface_name • %PIX|ASA-3-710003: {TCP|UDP} access denied by ACL from source_address/source_port to interface_name:dest_address/service • %PIX|ASA-4-106023: Deny protocol src [interface_name:source_address/source_port] dst interface_name:dest_address/dest_port [type {string}, code {code}] by access_group acl_ID • %PIX|ASA-4-209003: Fragment database limit of number exceeded: src = IP_address, dest = IP_address, proto = protocol, id = number • %PIX|ASA-4-209004: Invalid IP fragment, size = bytes exceeds maximum size = bytes: src = IP_address, dest = IP_address, proto = protocol, id = number • %PIX|ASA-4-209005: Discard IP fragment set with more than number elements: src = IP_address, dest = IP_address, proto = protocol, id = number • %PIX|ASA-4-401004: Shunned packet: IP_address ==> IP_address on interface interface_name • %PIX|ASA-4-402103: identity does not match negotiated identity (ip) dest_address= dest_address, src_addr= source_address, prot= protocol, (ident) local=inside_address, remote=remote_address, local_proxy=IP_address/IP_address/port/port, remote_proxy=IP_address/IP_address/port/port • %PIX|ASA-4-405001: Received ARP {request | response} collision from IP_address/MAC_address on interface interface_name • %PIX|ASA-4-405002: Received mac mismatch collision from IP_address/MAC_address for authenticated host • %PIX|ASA-4-407001: Deny traffic for local-host interface_name:inside_address, license limit of number exceeded • %PIX|ASA-4-415012: internal_sig_id HTTP Deobfuscation signature detected - action HTTP deobfuscation detected IPS evasion technique from source_address to source_address • %PIX|ASA-4-415014: internal_sig_id Maximum of 10 unanswered HTTP requests exceeded from source_address to dest_address • %PIX|ASA-5-111001: Begin configuration: IP_address writing to device • %PIX|ASA-5-111003: IP_address Erase configuration • %PIX|ASA-5-111004: IP_address end configuration: {FAILED|OK} • %PIX|ASA-5-111005: IP_address end configuration: OK • %PIX|ASA-5-111007: Begin configuration: IP_address reading from device. • %PIX|ASA-5-111008: User user executed the command string • %PIX|ASA-5-199001: PIX reload command executed from Telnet (remote IP address) • %PIX|ASA-5-199006: Orderly reload started at when by whom. Reload reason: reason • %PIX|ASA-5-304001: User source address accessed {JAVA URL|URL} dest_address: url. • %PIX|ASA-5-304002: Access denied URL url SRC IP_address DEST IP_address: url • %PIX|ASA-5-415007: internal_sig_id HTTP Extension method illegal - action 'method_name' from source_address to dest_address • %PIX|ASA-5-415008: internal_sig_id HTTP RFC method illegal - action 'method_name' from source_address to dest_address • %PIX|ASA-5-415010: internal_sig_id HTTP protocol violation detected - action HTTP Protocol not detected from source_address to dest_address • %PIX|ASA-5-415013: internal_sig_id HTTP Transfer encoding violation detected - action Xfer_encode Transfer encoding not allowed from source_address to dest_address • %PIX|ASA-5-500001: ActiveX content modified src IP_address dest IP_address on interface interface_name. • %PIX|ASA-5-500002: Java content modified src IP_address dest IP_address on interface interface_name. • %Pix|ASA-5-501101: User transitioning priv level • %PIX|ASA-5-502101: New user added to local dbase: Uname: user Priv: privilege_level Encpass: string • %PIX|ASA-5-502102: User deleted from local dbase: Uname: user Priv: privilege_level Encpass: string • %PIX|ASA-5-502103: User priv level changed: Uname: user From: privilege_level To: privilege_level • %PIX|ASA-5-612001: Auto Update succeeded:filename, version:number • %PIX|ASA-6-109006: Authentication failed for user user from inside_address/inside_port to outside_address/outside_port on interface interface_name. • %PIX|ASA-6-106012: Deny IP from IP_address to IP_address, IP options hex • %PIX|ASA-6-106015: Deny TCP (no connection) from IP_address/port to IP_address/port flags tcp_flags on interface interface_name. • %PIX|ASA-6-109008: Authorization denied for user user from source_address/source_port to destination_address/destination_port on interface interface_name.\ • %PIX|ASA-6-109024: Authorization denied from source_address/source_port to dest_address/dest_port (not authenticated) on interface interface_name using protocol • %PIX|ASA-6-109025: Authorization denied (acl=acl_ID) for user 'user' from source_address/source_port to dest_address/dest_port on interface interface_name using protocol • %PIX|ASA-6-113006: User user locked out on exceeding number successive failed authentication attempts • %PIX|ASA-6-302014: Teardown TCP connection id for interface:real- address/real-port to interface:real-address/real-port duration hh:mm:ss bytes bytes [reason] [(user)] • %PIX|ASA-6-308001: PIX console enable password incorrect for number tries (from IP_address) • %PIX|ASA-6-309002: Permitted manager connection from IP_address. • %PIX|ASA-6-315011: SSH session from IP_address on interface interface_name for user user disconnected by SSH server, reason: reason • %PIX|ASA-6-415009: internal_sig_id HTTP Header length exceeded. Received length byte Header - action header length exceeded from source_address to dest_address • %PIX|ASA-6-415011: internal_sig_id HTTP URL Length exceeded. Received size byte URL - action URI length exceeded from source_address to dest_address • %PIX|ASA-6-605004: Login denied from {source_address/source_port | serial} to {interface_name:dest_address/service | console} for user "user" • %PIX|ASA-6-605005: Login permitted from {source_address/source_port | serial} to {interface_name:dest_address/service | console} for user "user" • %PIX|ASA-6-606001: ADSM session number number from IP_address started • %PIX|ASA-6-606002: ADSM session number number from IP_address ended • %PIX|ASA-6-610101: Authorization failed: Cmd: command Cmdtype: command_modifier • %PIX|ASA-6-611101: User authentication succeeded: Uname: user • %PIX|ASA-6-611102: User authentication failed: Uname: user • %PIX|ASA-6-611311: VNPClient: XAUTH Failed: Peer: IP_address • %PIX|ASA-7-111009: User user executed cmd:string • %PIX|ASA-7-304009: Ran out of buffer blocks specified by url-block command N ote For an exhaustive list of all Cisco PIX/ASA/FWSM syslog messages, see http://www.cisco.com/en/US/products/sw/secursw/ps2120/products_system_message_gu ides_list.html. . Firewall Log Review and Analysis After the decision has been made to log events from your firewall, the next step is determining. different firewall vendors). What to Look for in Firewall Logs After you have collected the firewall logs and begun the process of analyzing the logs, determine