526 Chapter 10 • Incident Response, Forensics, and the Law Even companies with small IT departments will usually have a net- work person and a systems administrator, if not more than one of each. You’ll likely need to involve them.You’ll probably need a representative from your Legal department or attorney’s office. A system administrator or a dedicated security engineer can handle the forensics work. Finally, you’ll need a dedicated security function that will form the core of the team, and tie it together.This may be a dedicated person, or perhaps a portion of a person’s time, but the responsibility must belong to one or more individuals. The core person’s responsibility will be to call meetings, make sure representation is present from all concerned organizations, coordinate writing policy and getting agreement on policy, arrange for training as needed, and drive actual incident response when the occasion arises. It may be obvious, but it’s worth mentioning: Someone will need to be on call at all times to handle any incidents that arise. Hopefully, you will have enough team members that this duty can be rotated.This also implies that there is some mechanism to detect an incident, whether it’s an IDS or an e-mail address that people can use to report issues as they arise. Setting the Prosecution Boundaries At some point during an incident, a decision must be made as to whether you wish to pursue legal or civil action, or even just report the incident to a provider or third-party organization.This is not a trivial decision. Pursuing legal action is expensive, and you must consider what you will get out of it. Attackers Crossing the Line The first line you have to draw is the line between attempt and inci- dent. It doesn’t matter who you are, whether you’re tiny or huge, you will get attempts to penetrate your site. Some of these attempts will be downright useless or idiotic—in fact, most of them will be.This includes things like trying NT exploits against a Unix Web server, or trying to rsh to a Windows NT database server. At one large software company where www.syngress.com 134_ecomm_10 6/19/01 12:05 PM Page 526 Incident Response, Forensics, and the Law • Chapter 10 527 I managed the firewalls, there were over 1000 attempts to Telnet to the main Web server, per day—all day, every day. Every single attempt was stopped at the firewall and logged. One has to wonder what they would have tried to do if they were able to connect, try logging in as guest? Unfortunately, the reason that the stupid stuff is attempted is that many hosts are vulnerable to it.An attacker may figure that 5 percent of the hosts that he tries have some stupid misconfiguration or hole, so what does he have to lose by trying? So, all day long, you will receive probes and scans.These are people trying out new tools, or potential attackers gathering intelligence infor- mation, or even automated worms.You may get somewhere between dozens and thousands of these per day.You can’t possibly treat them all as full incidents, because no one has that kind of time to spend. You need a set of criteria for deciding how a particular event will be handled.Your possible response ranges from ignoring them, to taking legal action.There are a handful of options in between, such as notifying the attacker’s ISP or company.Your incident response team, or whoever writes policy for that team, has to decide where the boundaries are for each response. One fairly common response for relatively benign attempts (besides ignoring it, of course) is to report the attempts to the appropriate ISP, company, or their provider.This can be time consuming, and you may not get much response from the organizations to which you are reporting. Some IDS software includes a reporting mechanism to help generate reports and locate the proper e-mail addresses to contact.There are also services, such as the ARIS service offered by SecurityFocus.com (note that this author works on the ARIS project), which allows users to submit their IDS logs, and assists them in producing incident reports, and mailing them in.The reports can be done in a number of languages, appropriate to the address where the report is being sent.This service is free. Another recourse is posting the incidents to a community mailing list, such as the Incidents list, also hosted by SecurityFocus.com. Many subscribers to the Incidents list have been able to confirm that others are seeing the same new scan, or to get answers as to what an attacker is looking for. In the past, much of the traffic to such a list has consisted of “I’m seeing this scan ” followed by a number of lines from an IDS or www.syngress.com 134_ecomm_10 6/19/01 12:05 PM Page 527 528 Chapter 10 • Incident Response, Forensics, and the Law firewall log. It seems likely that services such as ARIS will eventually take over the mechanical function of correlating attacks among multiple users, and the mailing lists will remain an advice and discussion forum. What remains to decide is when to go into full-blown incident response mode. Clearly, that would almost always qualify in the event of an actual intrusion. An intrusion would consist of an attacker gaining a higher level of privilege that you intended for him or her to have.You would probably also want to treat a significant denial-of-service (DoS) attack in a similar manner.Will there be any cases where you would want to treat unsuccessful attempts the same way you would an actual intrusion? Possibly, if the attacker was persistent enough. For example, if you have an attacker who is doing password grinding—that is, trying usernames and passwords repeatedly—you might want to go to the trouble of tracking him or her down. For most people, this would prob- ably have to be an especially persistent attacker—for example, if every time you blocked the IP address or range that the attacker is coming from, he or she started again from a different one. An attacker who is coming at you from obviously compromised machines elsewhere on the Internet would probably be another case that you’d want to track down. (Note that for the latter, you would need to take care that the traffic seemed to have some human intelligence behind it.There are many worms in the wild that will come at you from compromised machines, but they have so far been very single-minded.) You should take into consideration all of the types of attacks and risks that are discussed in this book, and for each one, decide how you should respond to it.This would necessarily include different levels of severity as well. For example, you might find yourself the recipient of a bunch of traffic coming from a Smurf amplifier, but it may not even be impacting your service. How to calculate damages needs to be discussed.What is fair to include in damages? Most of the time, it’s only going to be the time spent dealing with the incident.This includes investigation, meetings, preparing documentation, and so forth. Businesses that have a docu- mented daily revenue amount, and those revenues wouldn’t be mostly recovered when service was restored, may be able to include that lost amount in the damages. For example, an online brokerage that does a www.syngress.com 134_ecomm_10 6/19/01 12:05 PM Page 528 Incident Response, Forensics, and the Law • Chapter 10 529 certain dollar amount in trades per trading day might be able to claim that very amount in damages if they lose an entire trading day to a DDoS attack. However, they likely wouldn’t be able to if it occurred over a weekend when the markets were closed. Understanding the Chain of Custody The chain of custody defines, quite simply, who has access to the evi- dence during the entire investigation process.The basic reason for this concerns tampering (it also makes sure that the evidence doesn’t show up missing). Starting from when it’s clear that evidence is present, a log needs to be kept of who has had access to it. If at all possible, you should also go back as far as when the evidence first became present (usually back to when the penetration took place). This will not always be possible; for example, some evidence may be months old. In our hypothetical scenario earlier in the chapter, careful record was kept when the investigation was being conducted according to a policy. However, one of the minor items of evidence that was col- lected was about six weeks old. For that one evidence item, it would probably not be possible to determine all the people who had access to the old Web logs. It would include all the people who have a login to that machine, as well as whoever has access to the room in which the physical machine resides.The chain of evidence is one of the many rea- sons why it’s important to conduct a careful investigation as soon after the incident occurs as possible. Maintaining a chain of custody list isn’t difficult; you just have to record several items: ■ Who was in custody (possession) of the evidence? ■ Where was the evidence? ■ What security measures are in place at that location? ■ What items of evidence existed at that time? www.syngress.com 134_ecomm_10 6/19/01 12:05 PM Page 529 530 Chapter 10 • Incident Response, Forensics, and the Law You must write down a new entry each time one of these things changes, such as turning the evidence over to a new person, adding a piece of evidence, or moving it. Where you maintain the evidence is obviously important, as it affects whether someone would have had an opportunity to tamper with it.You might wonder who would have access to your premises that you would have to worry about, but a significant portion of incidents are caused by insiders or someone in cooperation with an insider. Police property or evidence rooms are secured, which illustrates nicely the point about having a secured storage area, and the idea that you can’t always trust the other folks in the building with you. Police evidence security is in part to guard against tampering by other officers. The ideal place to maintain evidence is in a safe.That can be prob- lematic due to the size of the evidence to be maintained.The next-best option would be a locked room, with a limited number of folks who hold a key. If janitorial services has a key, then the room isn’t very secure. Surveillance of some sort would be ideal. You maintain a chain of evidence in case you have to use it.This could be for internal action, such as putting someone on report or firing an employee, or to turn over to law enforcement or enter into court evidence. Once the evidence has been turned over to law enforcement or a court, they are responsible for maintaining the chain of custody. Even if something goes wrong with the chain of custody, all is not lost.The evidence may still be perfectly usable, but if you’re in a situa- tion where the evidence is already in question, it won’t help if the chain of custody hasn’t been maintained. A court is much more likely to accept evidence that has had a proper chain of custody recorded. Establishing an Incident Response Process Once you have a policy in place that dictates how you will respond when an incident occurs, you need to build a set of processes to support your responses.This covers the range from really minor attempts, all the way up to full intrusions. Among the items you need to set up are your www.syngress.com 134_ecomm_10 6/19/01 12:05 PM Page 530 Incident Response, Forensics, and the Law • Chapter 10 531 forensics toolkit and skills, and your incident tracking system, which we’ll cover in the following sections. The most technical part of the whole incident handling process is the forensics aspect. For many security engineers, it’s also the most inter- esting. However, like most jobs, incident handling is 20 percent inter- esting work, and 80 percent grunt work. Probably one of the reasons that the forensics part of an investigation is so interesting is because it’s challenging. Computer forensics requires a deep understanding of how the operating system you’re investigating works.You will need to understand how the files are stored on disk, how the processes interact, how all the software is configured, and what log information is available to you.And you have to know this for each operating system you need to investigate. There is a practically infinite combination of operating systems, applications, and configurations. Each new application provides a new opportunity for forensic information to be collected.This section attempts to provide an introduction to computer forensics as a basis for further learning. Once you have your incident response procedures written down, and have your tools, inventories, and some training in place, you’ll need a system for tracking incidents that occur.This goes beyond actual intru- sions and cases that you investigate, and should include things like attempts, and interesting traffic patterns that your IDS picks up. Introduction to Forensic Computing The first step in any forensic investigation is to make a backup of all the information available to you, if possible. Unfortunately, this doesn’t just mean backing up the drives. Before you even get to that point, you have to decide how to examine what might be in memory when you arrive. There may be some evidence in memory that you want to get at, and not all operating systems have a provision for dumping RAM to disk. Even for those that do, you normally have to configure that ahead of time. There is a basic problem that you will face as an investigator; you need to do something with a machine that is under the control of an www.syngress.com 134_ecomm_10 6/19/01 12:05 PM Page 531 532 Chapter 10 • Incident Response, Forensics, and the Law attacker.The vast majority of the time, everything will be straightforward, and you will have no worries.There will be nothing special in memory, and you will be able to shut down without losing anything or causing damage. However, there is always the possibility that your system will get broken into by a very special attacker, one who cares a lot about what evidence he or she leaves behind. In such a situation, doing investigation of the machine is extremely tricky.You will probably be able to log on and poke around; however, the attacker controls your view of reality.A sophisticated attacker can go as far as to replace parts of the operating system, live, in memory. Such an attacker can hide processes, files, or anything he or she wants. As long as you are dependent on the running OS to provide you with informa- tion, the attacker can provide you with lies. Depending on which tricks were used, the tricks can be partially countered. For example, some rootkits will install a module that will allow modified executables to go undetected.When you run your MD5 hash calculation tool, the original file is presented, and it checks out. However, when you run the file, the replacement version is run. Dominique Brezinski gave a presentation on these topics at the Black Hat Briefings in 1999.This presentation has been provided for public viewing online, and can be found at www.blackhat.com/html/ bh-multi-media-archives.html. Search the page for “Building a Forensic Toolkit that Will Protect You from Evil Influences.”This situation can be a damned-if-you-do, damned-if-you-don’t situation. One choice is to shut the system down, and boot your own OS to do your investigation, thereby removing the possibility that the victimized operating system will be altering your view of reality. However, what if the attacker designed in a booby trap to erase everything if you try to shut down? Alternately, you can keep the machine up, and try to do an interactive investigation, but you’re at the mercy of the running OS, and run the risk of destroying date and timestamps with each command you type, with no backup. Even disconnecting from the network might possibly set off a booby trap.What if the attacker designed his or her compromise software to erase everything if it stops receiving a certain signal from the outside? That’s possibly a signal that you can’t replicate, because it’s encrypted. www.syngress.com 134_ecomm_10 6/19/01 12:05 PM Page 532 Incident Response, Forensics, and the Law • Chapter 10 533 You could monitor traffic to and from the machine, but that will be time consuming. There is no single right answer. Based on how most of the existing forensics tools work, current best practice is to just pull the plug on the machine.This allows for later backup, and doesn’t give any shutdown code an opportunity to execute.This is based solely on how often a really clever attack occurs (or more specifically, doesn’t occur).This opinion could easily change as time goes on, and will mostly be depen- dent on the state-of-the-art in rootkits.There have been viruses in the past that encrypt the FAT table of a hard drive, making it difficult to examine the hard drive without the virus running. A rootkit could act in a similar manner, but so far, none of them do. www.syngress.com Rootkits Briefly, a rootkit is a piece of software designed to be installed on a victimized machine that permits the attacker to burrow in and hide. Rootkits will allow for things such as hiding files, modifying the output of commands such as ps or netstat, and install various backdoors to let the attacker back in. The idea behind a rootkit is for the break-in to go undiscovered as long as possible. Rootkits are operating-system specific, as they need to hook into very specific OS functions, replace the exact proper binaries, and so on. Here’s a sample of several popular rootkits: ■ Rootkit for Windows NT www.rootkit.com ■ An analysis of t0rn www.securityfocus.com/focus/ids articles/t0rn.html ■ A large collection of Unix rootkits http://packetstorm .securify.com/UNIX/penetration/rootkits ■ A Linux rootkit detector www.chkrootkit.org (there’s a good set of links for more rootkit reading there, too) Tools & Traps… 134_ecomm_10 6/19/01 12:05 PM Page 533 534 Chapter 10 • Incident Response, Forensics, and the Law It bears repeating that backup is critical.You should back up first, investigate second. Unfortunately, this isn’t a terribly easy problem to solve.Well, let’s qualify that: It isn’t easy to solve if you want to grab a backup before shutting the compromised system down, especially on Windows. It’s generally quite easy to do a sector-by-sector backup of a hard drive if you can boot the machine to your own operating system, or if you can remove the drives and attach them to another system you control. The general problem with backing up a compromised system before you shut it down is that any use of the compromised system damages the evidence to some degree. Obviously, if you feel you need to do a backup prior to initial shutdown, you’ll have to do your best to mini- mize damage.You also need to not rely on the installed support libraries as much as possible, in case one of them has been replaced with a modi- fied version. For most operating systems, this means statically compiled binaries (no dynamic libraries).You’ll probably want to run off a CD- ROM, or similar read-only removable media. For Unix systems, the backup could be accomplished with minimal damage if you prepare the static binaries ahead of time.You can use a combination of dd and netcat to grab copies of entire partitions, including the “unused” sectors that the file system indicates don’t have any data on them.As shown in this section’s sidebar, this method was used by The Honeynet Project in their Forensic Challenge.The tradeoffs associated with this method are nicely summarized in the answer given in the sidebar; in particular, chain of custody is a little bit fuzzier here, because of the fact that the drive will keep changing.You can’t step into the same stream twice, and if the original computer is collected for evi- dence, and law enforcement does its own forensic analysis, there may be some question as to why there are differences.The Forensic Challenge wasn’t concerned with this, as prosecution wasn’t their goal.This is not to say that your evidence will be invalidated, but it’s another variable. Getting a backup of a running Windows system may be even more difficult. At present, all the prepackaged backup tools that I have found for Windows require an install step, and they are file based, meaning that they won’t get the “empty” drive space.They will also not take any care to not modify the file system being backed up, and will do things such www.syngress.com 134_ecomm_10 6/19/01 12:05 PM Page 534 Incident Response, Forensics, and the Law • Chapter 10 535 as modify the last accessed times and archive bits.There have been scat- tered reports of using a port of dd for Windows to do similar backups to the one illustrated in our Honeypot sidebar, but no one has written a clear procedure for its use, and it appears that dd can’t support all drive types on Windows, at least for the current ports. www.syngress.com The Honeynet Project Unix Backup Method As stated in this section, you can use a combination of dd and netcat to grab copies of entire partitions, including the “unused” sectors that the file system indicates don’t have any data on them. This method was used to prepare a backup of a compromised system for The Forensic Challenge, put on by The Honeynet Project (http://project.honeynet.org). The method they used is addressed in their FAQ, as shown in the following excerpt: Q: How did you make images of the compromised system? A: The file images.tar is an archive containing 6 GNU zip com- pressed files, taken from each of the systems’ active partitions at the time of compromise. It was created 08 November, 2000, at 21:00 CST. The following process was used to take the images, with minimum data pollution as a primary goal. We did not take the system down during the process. The following actions were taken while the system was still live. 1. Mounted cdrom containing forensic analysis tools (all statically compiled). 2. Used static binaries of dd(1M) and netcat(1M) from the cdrom to dd images of the hard drive to a trusted forensic system over the network. This is done by the following: Note from the Underground… Continued 134_ecomm_10 6/19/01 12:05 PM Page 535 [...]... the exact set of DIS and/or firewall you do, your ITS, and your idea of what constitutes an incident worth tracking Many of the things your IDS will report will be false alarms, but this is totally dependent on your particular environment.You will need to spend some quality time with your IDS, tuning the rules to reduce these as much as possible, and then documenting the ones you can’t eliminate entirely... www.syngress.com 5 49 134 _ecomm_ 10 550 6/ 19/ 01 12:05 PM Page 550 Chapter 10 • Incident Response, Forensics, and the Law Frequently Asked Questions The following Frequently Asked Questions, answered by the authors of this book, are designed to both measure your understanding of the concepts presented in this chapter and to assist you with real-life implementation of these concepts To have your questions... www.syngress.com 541 134 _ecomm_ 10 542 6/ 19/ 01 12:05 PM Page 542 Chapter 10 • Incident Response, Forensics, and the Law Resources Listed here are some resources available on the Internet for topics discussed in this chapter.We recommend that you take the time to go through these, and take note of the ones that best meet your needs to later referral Legal/Government/Law Enforcement The site www.cybercrime.gov... 551 134 _ecomm_ 10 6/ 19/ 01 12:05 PM Page 552 134 _ecomm_ appA 6/ 19/ 01 11:58 AM Page 553 Appendix A Cisco Solutions for Content Delivery Solutions in this appendix: s Improving Security Using Cisco LocalDirector s Securing Geographically Dispersed Server Farms Using Cisco DistributedDirector s Improving Security Using the Cisco Content Services Switch Summary Frequently Asked Questions 553 134 _ecomm_ appA... receptionist receives more telephone calls than can be handled—calls would be dropped, customers would get frustrated, and money would be lost.Today’s reality is that your Web site has become that window into your company Similarly, if your Web site gets too many hits, the same thing would happen—hits would be dropped, customers would get frustrated, and money would be lost As the World Wide Web grows and... www .site. com Port default Conns 892 Syn Count 500 LocalDirector(config)# Using Network Address Translation to Hide Real Addresses LocalDirector supports Network Address Translation (NAT).This allows you to use unregistered IP addresses on your inside network (usually the server farm) and prevents hackers from being able to directly target the real server’s IP address Request for Comments (RFC) 191 8... also conserves registered IP addresses www.syngress.com 5 59 134 _ecomm_ appA 560 6/ 19/ 01 11:58 AM Page 560 Appendix A • Cisco Solutions for Content Delivery Increased security is provided through NAT by hiding the internal IP address range and making it more difficult for potential hackers to access as well as learn about the internal structure of your network Figure A.2 shows an example of a device performing... difficult to hold you at fault A well-written policy will tell you what your responsibilities are, and what other people are on the hook for Establishing an Incident Response Team (IRT) You’ll likely need to involve a network person and a systems administrator.You’ll probably need a representative from your www.syngress.com 134 _ecomm_ 10 6/ 19/ 01 12:05 PM Page 547 Incident Response, Forensics, and the Law •... www.syngress.com 134 _ecomm_ 10 6/ 19/ 01 12:05 PM Page 5 49 Incident Response, Forensics, and the Law • Chapter 10 Tracking Incidents An incident tracking system (ITS) is a collection of programs designed to help an IRT manage the incidents that occur in their environment.These programs range from simple port scans that you do nothing about, to full-blown legal cases with appropriate legal documentation The... 134 _ecomm_ appA 6/ 19/ 01 11:58 AM Page 5 59 Cisco Solutions for Content Delivery • Appendix A LocalDirector(config)# show synguard Machine www.test.com Port SynGuard default Status 500 LocalDirector(config)# show syn Machine www.test.com Port Conns default 648 Syn Count 176 The following example shows synguard active Notice how the status changes to Active: LocalDirector(config)# show synguard Machine www .site. com . support your responses.This covers the range from really minor attempts, all the way up to full intrusions. Among the items you need to set up are your www.syngress.com 134 _ecomm_ 10 6/ 19/ 01 12:05. firewall you do, your ITS, and your idea of what constitutes an incident worth tracking. Many of the things your IDS will report will be false alarms, but this is totally dependent on your particular. online brokerage that does a www.syngress.com 134 _ecomm_ 10 6/ 19/ 01 12:05 PM Page 528 Incident Response, Forensics, and the Law • Chapter 10 5 29 certain dollar amount in trades per trading day