Network Traffic Analysis Using tcpdump Real World Analysis Judy Novak jhnovak@ix.netcom.com All material Copyright  Novak, 2000, 2001 All rights reserved Real World Examples • Introduction to tcpdump • Writing tcpdump Filters • Examination of Datagram Fields • Beginning Analysis • Real World Examples • Step by Step Analysis This page intentionally left blank Objectives • Learn advanced analysis through examining real world examples The final challenge in intrusion detection was to synthesize confirmed anomalies to further understand the exploit or trends In this section, we demonstrate, using examples, how we delve into the analysis process and come up with a more complete explanation of the activity You’ve Had a Compromise!!!! • Notification from parent CERT organization of compromised host • The notifier only knew: • IP numbers of the hostile host and compromised host • Day and approximate time of incident • What tools have we learned about that might help us validate and investigate this incident? For starters, we need to examine tcpdump output for the time of the reported event to determine out what happened From there, we can use the new scripts we created if more detail is required So, tcpdump records were dumped for the site: tcpdump –r tcpdumpfile ‘hostile-IP and compromised-IP’ where hostile-IP and compromised-IP were the actual IP’s What a “Normal” TCP Connection Session Looks Like • Before looking at output from the tcpdump output from the break-in, let’s examine typical output from a telnet session boulder.myplace.com.38060 > aspen.myplace.com.telnet: S 3774957990: 3774957990(0) win 8760 (DF) aspen.myplace.com.telnet > boulder.myplace.com.38060: S 2009600000: 2009600000(0) ack 3774957991 win 1024 boulder.myplace.com.38060 > aspen.myplace.com.telnet: ack win 8760 (DF) boulder.myplace.com.38060 > aspen.myplace.com.telnet: P 1:28(27) ack win 8760 (DF) aspen.myplace.com.telnet > boulder.myplace.com.38060: P 1:14(13) ack win 1024 aspen.myplace.com.telnet > boulder.myplace.com.38060: P 14:23(9) ack 28 win 1024 aspen.myplace.com.telnet > boulder.myplace.com.38060: F 4289:4289(0) ack 92 win 1024 boulder.myplace.com.38060 > aspen.myplace.com.telnet: ack 4290 win 8760 (DF) boulder.myplace.com.38060 > aspen.myplace.com.telnet: F 92:92(0) ack 4290 win 8760(DF) aspen.myplace.com.telnet > boulder.myplace.com.38060: ack 93 win 1024 A TCP session where connectivity has been established and data has been exchanged has the pattern of the above records Namely: • The connection is established by: • The client (boulder) sending a SYN to the server (aspen) to the desired port • The server (aspen) sending a SYN/ACK back to the client (boulder) acknowledging the request • The client (boulder) sending an ACK to the server (aspen) to complete the three-way handshake • Data is exchanged: • The PUSH flag is set for data transmission • Actual data bytes are sent (this can be seen before the ACK on records with the PUSH set, you see values in parentheses, i.e ( 27), (13), (9)) • The connection is stopped: • Gracefully with the two-way FIN exchange • Aborted with a RESET Note: timestamps have been removed from the tcpdump records to make the records more readable tcpdump Output from the Break-in whatsup.net.24997 > dns.myplace.com.sunrpc: S 2368718861:2368718861(0) win 512 whatsup.net.25002 > dns.myplace.com.139: S 4067302570:4067302570(0) win 512 whatsup.net.25075 > dns.myplace.com.ftp: S 1368714289:1368714289(0) win 512 dns.myplace.com.ftp > whatsup.net.25075: R 0:0(0) ack 1368714290 win (DF) whatsup.net.25177 > dns.myplace.com.1114: S 3231175487:3231175487(0) win 512 whatsup.net.25189 > dns.myplace.com.tcpmux: S 368146356:368146356(0) win 512 whatsup.net.25118 > dns.myplace.com.22: S 2035824356:2035824356(0) win 512 dns.myplace.com.1114 > whatsup.net.25177: R 0:0(0) ack 3231175488 win0 (DF) dns.myplace.com.22 > whatsup.net.25118: R 0:0(0) ack 2035824357 win (DF) dns.myplace.com is the alleged compromised host and whatsup.net is the hostile IP What’s wrong with the break-in theory? Well, no connections were established, no data exchanged, and no connections were stopped after being established This looks like a typical scan from whatsup.net Occasionally, you’ll see dns.myplace.com respond with a RESET/ACK meaning that the port that is being scanned is inactive Other than that, there is no evidence of any two-way conversation If no RESET/ACK is seen that means that the traffic never actually made it to the inactive port to be rejected In other words it was blocked by an exterior router The sensor for this site resides outside the filtering router This, of course, assumes that there is no backdoor entry to the network/host that the sensor does not see It turns out that this incident was a communication problem among the CERT organizations One reported it as a scan and somehow several phone calls later, it got reported as a break-in Because we had the tcpdump output, we were confident that a break-in did not occur Had we not had tcpdump or some other software that had an audit trail of activity, we would not have any concrete evidence to refute the break-in claim You should have an IDS or software that is capable of maintaining a history of activity such as tcpdump Scan of High-Numbered Port • Hostile Activity: • Scan to TCP port 32775 • Site router blocked much of activity • Internal sensor detected initial SYN activity to many hosts Examining tcpdump output for a given hour, a scan appeared of high-numbered port 32775 The sensor that picked this up was inside the packet-filtering device signaling that some traffic got through to internal hosts At this point, all we could see was that there were initial SYN connections attempted We would need to further investigate to discover if there were any responses to these connections Network View of Activity Filtering Router Hostile host 1.1.1.1 TCP port 32775 scan sensor Semi-protected subnet What we see above is a site that has a filtering router that blocks some traffic and allows in others Specifically, they allow in some high-numbered ports for certain subnets to accommodate users or system administrators who need to use outside services such as active FTP that might try to open up these ports The sensor collecting the traffic is inside the packet filtering router so anything we see has gotten through the packet filtering device A scan from hostile host 1.1.1.1 was received directed to destination port 32775 Port Scan 16:46:42.690168 scanner.net.4838 > 192.168.5.118.32775: S 2943632224:2943632224(0) win 32120 (DF) 16:46:43.270573 scanner.net.2077 > 192.168.5.120.32775: S 2940858778:2940858778(0) win 32120 (DF) 16:46:43.270737 scanner.net.2080 > 192.168.5.121.32775: S 2944647564:2944647564(0) win 32120 (DF) 16:46:43.272415 scanner.net.2083 > 192.168.5.124.32775: S 2938904083:2938904083(0) win 32120 (DF) 16:46:43.272906 scanner.net.2086 > 192.168.5.125.32775: S 2941596782:2941596782(0) win 32120 (DF) 16:46:43.275637 scanner.net.2091 > 192.168.5.128.32775: S 2950442877:2950442877(0) win 32120 (DF) 16:46:43.275813 scanner.net.2092 > 192.168.5.129.32775: S 2952745733:2952745733(0) win 32120 (DF) 16:46:43.276139 scanner.net.2097 > 192.168.5.132.32775: S 2950485912:2950485912(0) win 32120 (DF) This is the type of traffic that appeared on the sensor Many connections were allowed through the filtering router destined for TCP port 32775 It can be surmised that the scanner actually probed the entire class B 192.168 subnet, but since our vantage point is from the inside, we can only guess this We know that all subnets and hosts that had access rules that allowed access to ports greater than 1023 were scanned That is why it is suspected that the entire class B network was probed Either that or the scanner did some prior reconnaissance and knew that there were some subnets that allowed traffic through to high-numbered ports TCP port 32775 doesn’t have a well-known service associated with it, but it is in the range of ports considered to be Remote Procedure Call (RPC) services This range is 32770 – 34000 both TCP and UDP So, we begin to surmise that the prober is looking for some sort of RPC service being offered – either one that we know is vulnerable or an entirely new exploit that has yet to be reported Conventional inetd Services listens on port 23 listens on port 23 telnet server x telnet server q listens on port 23 telnet server z 10 A moment of digression to explain the difference in the way conventional internet services started by the Unix inetd daemon are identified versus the way RPC services must be identified A server can start any given internet service on any port, but convention dictates that if we want to offer a universal service, we must listen on a standard well-known port For telnet, this is port 23 Almost all servers offering the telnet service will listen on port 23 so the clients will know where to find the telnet port 10 .. .Real World Examples • Introduction to tcpdump • Writing tcpdump Filters • Examination of Datagram Fields • Beginning Analysis • Real World Examples • Step by Step Analysis This... by Step Analysis This page intentionally left blank Objectives • Learn advanced analysis through examining real world examples The final challenge in intrusion detection was to synthesize confirmed... understand the exploit or trends In this section, we demonstrate, using examples, how we delve into the analysis process and come up with a more complete explanation of the activity You’ve Had a Compromise!!!!