1002 Control (UAC) virtualization technology discussed in Chapter 6, but it applies to write operations as well. It applies if the following are true: ■ The application is a legacy application, meaning that it does not contain a manifest file compatible with Windows Vista or Windows Server 2008 with the requestedExecutionLevel value set. ■ The application is trying to modify a WRP-protected resource (the file or registry key contains the TrustedInstaller SID). ■ The application is being run under an administrator account (always true on systems with UAC enabled because of automatic installer program detection). WRP copies files that are needed to restart Windows to the cache directory located at \Windows\winsxs\Backup. Critical files that are not needed to restart Windows are not copied to the cache directory. The size of the cache directory and the list of files copied to the cache cannot be modified. To recover a file from the cache directory, you can use the System File Checker (Sfc.exe) tool, which can scan your system for modified protected files and restore them from a good copy. System Hive Corruption ■ Symptoms If the System registry hive (which is discussed along with hive files in the section “The Registry” in Chapter 4) is missing or corrupted, Winload will display the message “Windows could not start because the following file is missing or corrupt: \WINDOWS\SYSTEM32\CONFIG\SYSTEM,” on a black screen after the BIOS POST. ■ Causes The System registry hive, which contains configuration information necessary for the system to boot, has become corrupt or has been deleted. ■ Resolution Boot into the Windows Recovery Environment, choose the Command Prompt option, and then execute the chkdsk command. If the problem is not corrected, obtain a backup of the System registry hive. Windows makes copies of the registry hives every 12 hours (keeping the immediately previous copy with a .OLD extension) in a folder called \Windows\System32 \Config\RegBack, so copy the file named System to \Windows\System32\Config. If System Restore is enabled (System Restore is discussed in Chapter 11), you can often obtain a more recent backup of the registry hives, including the System hive, from the most recent restore point. You can choose System Restore from the Windows Recovery Environment to restore your registry from the last restore point. Post–Splash Screen Crash or Hang ■ Symptoms Problems that occur after the Windows splash screen displays, the desktop appears, or you log on fall into this category and can appear as a blue screen crash or a hang, where the entire system is frozen or the mouse cursor tracks the mouse but the system is otherwise unresponsive. ■ Causes These problems are almost always a result of a bug in a device driver, but they can sometimes be the result of corruption of a registry hive other than the System hive. 1003 ■ Resolution You can take several steps to try and correct the problem. The first thing you should try is the last known good configuration. Last known good (LKG), which is described earlier in this chapter and in the “Services” section of Chapter 4, consists of the registry control set that was last used to boot the system successfully. Because a control set includes core system configuration and the device driver and services registration database, using a version that does not reflect changes or newly installed drivers or services might avoid the source of the problem. You access last known good by pressing the F8 key early in the boot process to access the same menu from which you can boot into safe mode. As stated earlier in the chapter, when you boot into LKG, the system saves the control set that you are avoiding and labels it as the failed control set. You can leverage the failed control set in cases where LKG makes a system bootable to determine what was causing the system to fail to boot by exporting the contents of the current control set of the successful boot and the failed control set to .reg files. You do this by using the Regedit’s export functionality, which you access under the File menu: 1. Run Regedit, and select HKLM\SYSTEM\CurrentControlSet. 2. Select Export from the File menu, and save to a file named good.reg. 3. Open HKLM\SYSTEM\Select, read the value of Failed, and select the subkey named HKLM\SYSTEM\ControlXXX, where XXX is the value of Failed. 4. Export the contents of the control set to bad.reg. 5. Use WordPad (which is found under Accessories on the Start menu) to globally replace all instances of CurrentControlSet in good.reg with ControlSet. 6. Use WordPad to change all instances of ControlXXX (replacing XXX with the value of the Failed control set) in bad.reg with ControlSet. 7. Run Windiff from the Support Tools, and compare the two files. The differences between a failed control set and a good one can be numerous, so you should focus your examination on changes beneath the Control subkey as well as under the Parameters subkeys of drivers and services registered in the Services subkey. Ignore changes made to Enum subkeys of driver registry keys in the Services branch of the control set. If the problem you’re experiencing is caused by a driver or service that was present on the system since before the last successful boot, LKG will not make the system bootable. Similarly, if a problematic configuration setting changed outside the control set or was made before the last successful boot, LKG will not help. In those cases, the next option to try is safe mode (described earlier in this section). If the system boots successfully in safe mode and you know that particular driver was causing the normal boot to fail, you can disable the driver by using the Device Manager (accessible from the Hardware tab of the System Control Panel item). To do so, select the driver in question and choose Disable from the Action menu. If you recently updated the driver, and believe that the update introduced a bug, you can choose to roll back the driver to its previous version instead, also with the Device Manager. To restore a driver to its previous version, double-click on the device to open its Properties dialog box and click Roll Back Driver on the Driver tab. 1004 On systems with System Restore enabled, an option when LKG fails is to roll back all system state (as defined by System Restore) to a previous point in time. Safe mode detects the existence of restore points, and when they are present it will ask you whether you want to log on to the installation to perform a manual diagnosis and repair or launch the System Restore Wizard. Using System Restore to make a system bootable again is attractive when you know the cause of a problem and want the repair to be automatic or when you don’t know the cause but do not want to invest time to determine the cause. If System Restore is not an option or you want to determine the cause of a crash during the normal boot and the system boots successfully in safe mode, attempt to obtain a boot log from the unsuccessful boot by pressing F8 to access the special boot menu and choosing the boot logging option. As described earlier in this chapter, Session Manager (\Windows\System32\Smss.exe) saves a log of the boot that includes a record of device drivers that the system loaded and chose not to load to \Windows\ntbtlog.txt, so you’ll obtain a boot log if the crash or hang occurs after Session Manager initializes. When you reboot into safe mode, the system appends new entries to the existing boot log. Extract the portions of the log file that refer to the failed attempt and safe-mode boots into separate files. Strip out lines that contain the text “Did not load driver”, and then compare them with a text comparison tool such as Windiff. One by one, disable the drivers that loaded during the normal boot but not in the safe-mode boot until the system boots successfully again. (Then reenable the drivers that were not responsible for the problem.) If you cannot obtain a boot log from the normal boot (for instance, because the system is crashing before Session Manager initializes), if the system also crashes during the safe-mode boot, or if a comparison of boot logs from the normal and safe-mode boots do not reveal any significant differences (for example, when the driver that’s crashing the normal boot starts after Session Manager initializes), the next tool to try is the Driver Verifier combined with crash dump analysis. (See Chapter 14 for more information on both these topics.) 13.3 Shutdown If someone is logged on and a process initiates a shutdown by calling the Windows Exit-WindowsEx function, a message is sent to that session’s Csrss instructing it to perform the shutdown. Csrss in turn impersonates the caller and sends an RPC message to Winlogon, telling it to perform a system shutdown. Winlogon then impersonates the currently logged-on user (who might or might not have the same security context as the user who initiated the system shutdown) and calls ExitWindowsEx with some special internal flags. Again, this call causes a message to be sent to the Csrss process inside that session, requesting a system shutdown. This time, Csrss sees that the request is from Winlogon and loops through all the processes in the logon session of the interactive user (again, not the user who requested a shutdown) in reverse order of their shutdown level. A process can specify a shutdown level, which indicates to the system when they want to exit with respect to other processes, by calling SetProcessShutdownParameters. Valid shutdown levels are in the range 0 through 1023, and the default level is 640. Explorer, for example, sets its shutdown level to 2 and Task Manager specifies 1. For each process that owns a top-level window, Csrss sends the WM_QUERYEND 1005 SESSION message to each thread in the process that has a Windows message loop. If the thread returns TRUE, the system shutdown can proceed. Csrss then sends the WM_ENDSESSION Windows message to the thread to request it to exit. Csrss waits the number of seconds defined in HKCU\Control Panel\Desktop\HungAppTimeout for the thread to exit. (The default is 5000 milliseconds.) If the thread doesn’t exit before the timeout, Csrss fades out the screen and displays the hung-program screen shown in Figure 13-9. (You can disable this screen by changing the registry value HKCU\Control Panel\Desktop\AutoEndTasks to 1.) This screen indicates which programs are currently running and, if available, their current state. Windows indicates which program isn’t shutting down in a timely manner and gives the user a choice of either killing the process or aborting the shutdown. (There is no timeout on this screen, which means that a shutdown request could wait forever at this point.) Additionally, third-party applications can add their own specific information regarding state—for example, a virtualization product could display the number of actively running virtual machines. If the thread does exit before the timeout, Csrss continues sending the WM_QUERYEND SESSION/WM_ENDSESSION message pairs to the other threads in the process that own windows. Once all the threads that own windows in the process have exited, Csrss terminates the process and goes on to the next process in the interactive session. eXPerIMeNT: Witnessing the HungappTimeout You can see the use of the HungAppTimeout registry value by running Notepad, entering text into its editor, and then logging off. After the amount of time specified by the HungAppTimeout registry value has expired, Csrss.exe presents a prompt that asks you whether or not you want to end the Notepad process, which has not exited because it’s waiting for you to tell it whether or not to save the entered text to a file. If you click the Cancel button, Csrss.exe aborts the shutdown. As a second experiment, if you try shutting down again (with Notepad’s query dialog box still open), Notepad will display its own message box to inform you that shutdown cannot cleanly proceed. However, this dialog box is merely an informational message to help users—Csrss.exe will still consider that Notepad is “hung” and display the user interface to terminate unresponsive processes. If Csrss finds a console application, it invokes the console control handler by sending the CTRL_LOGOFF_EVENT event. (Only service processes receive the CTRL_SHUTDOWN_ EVENT event on shutdown.) If the handler returns FALSE, Csrss kills the process. If the handler returns TRUE or doesn’t respond by the number of seconds defined by HKCU\Control 1006 Panel\Desktop\WaitToKillAppTimeout (the default is 20,000 milliseconds), Csrss displays the hung-program screen shown in Figure 13-9. Next, Winlogon calls ExitWindowsEx to have Csrss terminate any COM processes that are part of the interactive user’s session. At this point, all the processes in the interactive user’s session have been terminated. Wininit next calls ExitWindowsEx, which this time executes within the system process context. This causes Wininit to send a message to the Csrss part of session 0, where the services live. Csrss then looks at all the processes belonging to the system context and performs and sends the WM_QUERYENDSESSION/WM_ENDSESSION messages to GUI threads (as before). Instead of sending CTRL_LOGOFF_EVENT, however, it sends CTRL_SHUTDOWN_EVENT to console applications that have registered control handlers. Note that the SCM is a console program that does register a control handler. When it receives the shutdown request, it in turn sends the service shutdown control message to all services that registered for shutdown notification. For more details on service shutdown (such as the shutdown timeout Csrss uses for the SCM), see the “Services” section in Chapter 4. Although Csrss performs the same timeouts as when it was terminating the user processes, it doesn’t display any dialog boxes and doesn’t kill any processes. (The registry values for the system process timeouts are taken from the default user profile.) These timeouts simply allow system processes a chance to clean up and exit before the system shuts down. Therefore, many system processes are in fact still running when the system shuts down, such as Smss, Wininit, Services, and Lsass. Once Csrss has finished its pass notifying system processes that the system is shutting down, Winlogon finishes the shutdown process by calling the executive subsystem function NtShutdownSystem. This function calls the function PoSetSystemPowerState to orchestrate the shutdown of drivers and the rest of the executive subsystems (Plug and Play manager, power manager, executive, I/O manager, configuration manager, and memory manager). 1007 For example, PoSetSystemPowerState calls the I/O manager to send shutdown I/O packets to all device drivers that have requested shutdown notification. This action gives device drivers a chance to perform any special processing their device might require before Windows exits. The stacks of worker threads are swapped in, the configuration manager flushes any modified registry data to disk, and the memory manager writes all modified pages containing file data back to their respective files. If the option to clear the paging file at shutdown is enabled, the memory manager clears the paging file at this time. The I/O manager is called a second time to inform the file system drivers that the system is shutting down. System shutdown ends in the power manager. The action the power manager takes depends on whether the user specified a shutdown, a reboot, or a power down. 13.4 Conclusion In this chapter, we’ve examined the detailed steps involved in starting and shutting down Windows (both normally and in error cases). We’ve examined the overall structure of Windows and the core system mechanisms that get the system going, keep it running, and eventually shut it down. The final chapter of this book explains how to deal with an unusual type of shutdown: system crashes. 1008 14. Crash Dump Analysis Almost every Windows user has heard of, if not experienced, the infamous “blue screen of death.” This ominous term refers to the blue screen that is displayed when Windows crashes, or stops executing, because of a catastrophic fault or an internal condition that prevents the system from continuing to run. In this chapter, we’ll cover the basic problems that cause Windows to crash, describe the information presented on the blue screen, and explain the various configuration options available to create a crash dump, a record of system memory at the time of a crash that can help you figure out which component caused the crash and why. This section is not intended to provide detailed troubleshooting information on how to analyze a Windows system crash. This section will also show you how to analyze a crash dump to identify a faulty driver or component. The effort required to perform basic crash dump analysis is minimal and takes a few minutes. Even if an analysis ascertains the problematic driver for only one out of every five or ten crash dumps, it’s still worth doing: one successful analysis can avoid future data loss, system downtime, and frustration. 14.1 Why Does Windows Crash? Windows crashes (stops execution and displays the blue screen) for many possible reasons. A common source is a reference to a memory address that causes an access violation, either a write operation to read-only memory or a read operation on an address that is not mapped. Another common cause is an unexpected exception or trap. Crashes also occur when a kernel subsystem (such as the memory manager and power manager) or a driver (such as a USB or display driver) detect inconsistencies in their operation. When a kernel-mode device driver or subsystem causes an illegal exception, Windows faces a difficult dilemma. It has detected that a part of the operating system with the ability to access any hardware device and any valid memory has done something it wasn’t supposed to do. But why does that mean Windows has to crash? Couldn’t it just ignore the exception and let the device driver or subsystem continue as if nothing had happened? The possibility exists that the error was isolated and that the component will somehow recover. But what’s more likely is that the detected exception resulted from deeper problems—for example, from a general corruption of memory or from a hardware device that’s not functioning properly. Permitting the system to continue operating would probably result in more exceptions, and data stored on disk or other peripherals could become corrupt—a risk that’s too high to take. So Windows adopts a fail fast policy in attempting to prevent the corruption in RAM from spreading to disk. 1009 14.2 The Blue Screen Regardless of the reason for a system crash, the function that actually performs the crash is KeBugCheckEx, documented in the Windows Driver Kit (WDK). This function takes a stop code (sometimes called a bugcheck code) and four parameters that are interpreted on a per– stop code basis. After KeBugCheckEx masks out all interrupts on all processors of the system, it switches the display into a low-resolution VGA graphics mode (one implemented by all Windows- supported video cards), paints a blue background, and then displays the stop code, followed by some text suggesting what the user can do. Finally, KeBugCheckEx calls any registered device driver bugcheck callbacks (registered by calling the KeRegisterBugCheckCallback function), allowing drivers an opportunity to stop their devices. It then calls registered reason callbacks (registered with KeRegisterBugCheckReasonCallback), which allow drivers to append data to the crash dump or write crash dump information to alternate devices. (It’s possible that system data structures have been so seriously corrupted that the blue screen isn’t displayed.) Figure 14-1 shows a sample Windows blue screen. KeBugCheckEx displays the textual representation of the stop code near the top of the blue screen and the numeric stop code and four parameters at the bottom of the blue screen. The first line in the Technical information section lists the stop code and the four additional parameters passed to KeBugCheckEx. A text line near the top of the screen provides the text equivalent of the stop code’s numeric identifier. According to the example in Figure 14-1, the stop code 0x000000D1 is a DRIVER_IRQL_NOT_LESS_OR_EQUAL crash. When a parameter contains an address of a piece of operating system or device driver code (as in Figure 14-1), Windows displays the base address of the module the address falls in, the date stamp, and the file name of the device driver. This information alone might help you pinpoint the faulty component. Although there are more than 300 unique stop codes, most are rarely, if ever, seen on production systems. Instead, just a few common stop codes represent the majority of Windows system crashes. Also, the meaning of the four additional parameters depends on the stop code (and not all stop codes have extended parameter information). Nevertheless, looking up the stop code 1010 and the meaning of the parameters (if applicable) might at least assist you in diagnosing the component that is failing (or the hardware device that is causing the crash). You can find stop code information in the section “Bug Checks (Blue Screens)” in the Debugging Tools for Windows help file. (For information on the Debugging Tools for Windows, see Chapter 1.) You can also search Microsoft’s Knowledge Base (http://support.microsoft.com) for the stop code and the name of the suspect hardware or application. You might find information about a workaround, an update, or a service pack that fixes the problem you’re having. The Bugcodes.h file in the WDK contains a complete list of the 300 or so stop codes, with some additional details on the reasons for some of them. Based on data collected from the release of Windows Vista through the release of Windows Vista SP1, the top 30 stop codes account for 96 percent of crashes and can be grouped into a dozen categories: ■ Page fault A page fault on memory backed by data in a paging file or a memorymapped file occurs at an IRQL of DPC/dispatch level or above, which would require the memory manager to have to wait for an I/O operation to occur. The kernel cannot wait or reschedule threads at an IRQL of DPC/dispatch level or higher. (See Chapter 3 for details on IRQLs.) This category also includes page faults in nonpaged areas. The common stop codes are:  0xA - IRQL_NOT_LESS_OR_EQUAL  0xD1 - DRIVER_IRQL_NOT_LESS_OR_EQUAL ■ Power management A device driver or an operating system function running in kernel mode is in an inconsistent or invalid power state. Most frequently, some component has failed to complete a power management I/O request operation within 10 minutes. This crash category is new in Windows Vista. In previous versions of the Windows operating system, these failures generally resulted in a system hang with no crash. The stop codes are:  0x9F - DRIVER_POWER_STATE_FAILURE  0xA0 - INTERNAL_POWER_ERROR ■ Exceptions and traps A device driver or an operating system function running in kernel mode incurs an unexpected exception or trap. The common stop codes are:  0x1E - KMODE_EXCEPTION_NOT_HANDLED  0x3B - SYSTEM_SERVICE_EXCEPTION  0x7E - SYSTEM_THREAD_EXCEPTION_NOT_HANDLED  0x7F - UNEXPECTED_KERNEL_MODE_TRAP  0x8E - KERNEL_MODE_EXCEPTION_NOT_HANDLED with P1 != 0xC0000005 STATUS_ACCESS_VIOLATION ■ Access violations A device driver or an operating system function running in kernel mode incurs a memory access violation, which is caused either by attempting to write to a read-only page or by attempting to read an address that isn’t currently mapped and therefore is not a valid memory location. The common stop codes are: 1011  0x50 - PAGE_FAULT_IN_NONPAGED_AREA  0x8E - KERNEL_MODE_EXCEPTION_NOT_HANDLED with P1 = 0xC0000005 STATUS_ACCESS_VIOLATION ■ Display The display device driver detects that it can no longer control the graphics processing unit or detects an inconsistency in video memory management. The common stop codes are:  0xEA - THREAD_STUCK_IN_DEVICE_DRIVER  0x10E - VIDEO_MEMORY_MANAGEMENT_INTERNAL  0x116 - VIDEO_TDR_FAILURE ■ Pool The kernel pool manager detects an improper pool reference. The common stop codes are:  0xC2 - BAD_POOL_CALLER  0xC5 - DRIVER_CORRUPTED_EXPOOL ■ Memory management The kernel memory manager detects a corruption of memory management data structures or an improper memory management request. The common stop codes are:  0x1A - MEMORY_MANAGEMENT  0x4E - PFN_LIST_CORRUPT ■ Consistency check This is a catch-all category for various other consistency checks performed by the kernel or device drivers. The common stop codes are:  0x18 - REFERENCE_BY_POINTER  0x35 - NO_MORE_IRP_STACK_LOCATIONS  0x44 - MULTIPLE_IRP_COMPLETE_REQUESTS  0xCE - DRIVER_UNLOADED_WITHOUT_CANCELLING_PENDING_OPERATIONS  0x8086 – This is a stop code used by the Intel storage driver iastor.sys ■ Hardware A hardware error, such as a machine check or a nonmaskable interrupt (NMI), occurs. This category also includes disk failures when the memory manager is attempting to read data to satisfy page faults. The common stop codes are:  0x77 – KERNEL_STACK_INPAGE_ERROR  0x7A - KERNEL_DATA_INPAGE_ERROR  0x124 - WHEA_UNCORRECTABLE_ERROR  0x101 - CLOCK_WATCHDOG_TIMEOUT (Software bugs can cause these errors too, but they are most common on over-clocked hardware systems.) [...]... operating system and drivers present on the machine This is the default setting for both Windows Vista and Windows Server 2008 ■ Small memory dump A small memory dump, which is typically between 128 KB and 1 MB in size and is also called a minidump or triage dump, contains the stop code and parameters, the list of loaded device drivers, the data structures that describe the current process and thread (called... analyzable crashes and, if so, you will need to execute manual analysis to try and determine what the problem is Here are some examples of basic commands that can provide clues during crash analysis The Debugging Tools for Windows help file provides complete documentation on these and other commands as well as examples of how to use them during crash analysis: ■ Use the !process 0 0 debugger command to look... to look at the processes running, and make sure that you understand the purpose of each one Try disabling or uninstalling unnecessary applications and services ■ Use the lm command with the kv option to list the loaded kernel-mode drivers Make sure that you understand the purpose of any third-party drivers and that you have the most recent versions ■ Use the !vm command to see whether the system has... reboot the crashed system and debug the problem offline 1036 The operating system code and data structures that handle processor exceptions can become corrupted such that a series of recursive faults occur One example of this would be if the operating system trap handler got corrupted and caused a page fault This would invoke the page fault handler, which would fault again, and so on If such a situation... reboots Obviously, before you spend time and energy making system configuration changes and analyzing crashes, you should ensure that your system’s kernel and drivers are the most recent available by using the services of Windows Update and third-party driver support sites Note If your system becomes unbootable because the Driver Verifier detects a driver error and crashes the system, then start in safe... different from the system that generated the dump However, the Microsoft symbol server contains images (and symbols) for all recent Windows versions, so you can set the image path in the debugger to point to the symbol server, and the debugger will automatically download the needed images (Of course, the Microsoft symbol server won’t have images for thirdparty drivers you have installed.) A more significant... required size, the system falls back to writing a minidump 1018 14.5 Windows error reporting As mentioned in Chapter 3, Windows includes a facility called Windows Error Reporting (WER), which facilitates the automatic submission of process and system failures (such as crashes and/ or hangs) to Microsoft (or an internal error reporting server) for analysis This feature is enabled by default, but it can... !analyze command with the –hang option This causes the debugger to examine the locks on the system and try to determine whether there’s a deadlock, and if so, what driver or drivers are involved However, for a hang like the one that Notmyfault’s Hang option generates, the !analyze analysis command will report nothing useful If the !analyze command doesn’t pinpoint the problem, execute !thread and !process... minidump from the dump file and stores it in the default location of \Windows\ Minidumps, unless otherwise configured through the MinidumpDir value in the HKLM\SYSTEM\CurrentControlSet \Control\CrashControl\ key 2 It writes the name of the minidump files to HKLM\SOFTWARE\Microsoft \Windows\ Windows Error Reporting\KernelFaults\Queue 3 It adds a command to execute WerFault.exe ( \Windows\ System32\WerFault.exe)... this: 1 2 3 Microsoft (R) Windows Debugger Version 6.9.0003.113 X86 Copyright (c) Microsoft Corporation All rights reserved Loading Dump File [C: \windows\ MEMORY.DMP] 1022 4 5 Kernel Summary Dump File: Only kernel address space is available Symbol search path is: srv*c:\programming\symbols\*http://msdl.microsoft.com /download/ 6 symbols 7 Executable search path is: 8 Windows Server 2008 Kernel Version 6001 . by the operating system and drivers present on the machine. This is the default setting for both Windows Vista and Windows Server 2008. ■ Small memory dump. involved in starting and shutting down Windows (both normally and in error cases). We’ve examined the overall structure of Windows and the core system mechanisms