Bugchecks Depicted 151 kd> kL Child-SP RetAddr Call Site fffffadf`dfcf19b8 fffffadf`dfee38c4 nt!KeBugCheck fffffadf`dfcf19c0 fffff800`012ce9cf userdump!UdIoctl+0x104 fffffadf`dfcf1a70 fffff800`012df026 nt!IopXxxControlFile+0xa5a fffffadf`dfcf1b90 fffff800`010410fd nt!NtDeviceIoControlFile+0x56 fffffadf`dfcf1c00 00000000`77ef0a5a nt!KiSystemServiceCopyEnd+0x3 00000000`01eadd58 00000001`0000a755 ntdll!NtDeviceIoControlFile+0xa 00000000`01eadd60 00000000`77ef30a5 userdump_100000000!UdServiceWorkerAPC+0x1005 00000000`01eaf970 00000000`77ef0a2a ntdll!KiUserApcDispatcher+0x15 00000000`01eafe68 00000001`00007fe2 ntdll!NtWaitForSingleObject+0xa 00000000`01eafe70 00000001`00008a39 userdump_100000000!UdServiceWorker+0xb2 00000000`01eaff20 000007ff`7fee4db6 userdump_100000000!UdServiceStart+0x139 00000000`01eaff50 00000000`77d6b6da ADVAPI32!ScSvcctrlThreadW+0x25 00000000`01eaff80 00000000`00000000 kernel32!BaseThreadStart+0x3a This might be useful if we want to see kernel data that happened to be at the exception time. In this case we can avoid requesting complete memory dump of physi- cal memory and ask for kernel memory dump only together with a user dump. Note: do not set this option if you are unsure. It can have your production servers bluescreen in the case of false positive dumps. Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 152 PART 2: Professional Crash Dump Analysis CF Bugcheck CF name is the second longest one: TERMINAL_SERVER_DRIVER_MADE_INCORRECT_MEMORY_REFERENCE (cf) Arguments: Arg1: a020b1d4, memory referenced Arg2: 00000000, value 0 = read operation, 1 = write operation Arg3: a020b1d4, If non-zero, the instruction address which referenced the bad memory address. Arg4: 00000000, Mm internal code. A driver has been incorrectly ported to Terminal Server. It is referencing session space addresses from the system process context. Probably from queueing an item to a system worker thread. The broken driver's name is displayed on the screen. Although bugcheck explanation mentions only system process context it can also happen in an arbitrary process context. Recall that kernel space address mapping is usually considered as persistent where virtual-to-physical mapping doesn’t change be- tween switching threads that belong to different processes. However there is so called session space in multi-user terminal services environments where different users can use different display drivers, for example: MS RDP users - RDPDD.DLL Citrix ICA users - VDTW30.DLL Vista users - TSDDD.DLL Console user - Some H/W related video driver like ATI or NVIDIA These drivers are not committed at the same time persistently since OS boot al- though their module addresses might remain fixed. Therefore when a new user session is created the appropriate display driver corresponding to terminal services protocol is loaded and mapped to the so called session space starting from A0000000 (x86) or FFFFF90000000000 (x64) after win32k.sys address range (on first usage) and then committed to physical memory by proper PTE entries in process page tables. During thread switch, if the new process context belongs to a different session with a different display driver, the current display driver is decommitted by clearing its PTEs and the new driver is committed by setting its proper PTE entries. Therefore in the system process context like worker threads virtual addresses corresponding to display driver code and data might be unknown. This can also hap- pen in an arbitrary process context if we access the code that belongs to a display driver Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. Bugchecks Depicted 153 that doesn’t correspond to the current session protocol. This can be illustrated with the following example where TSDD can be either RDP or ICA display driver. In the list of loaded modules we can see that ATI and TSDD drivers are loaded: 0: kd> lm start end module name . . . 77d30000 77d9f000 RPCRT4 (deferred) 77e10000 77e6f000 USER32 (deferred) 77f40000 77f7c000 GDI32 (deferred) 77f80000 77ffc000 ntdll (pdb symbols) 78000000 78045000 MSVCRT (deferred) 7c2d0000 7c335000 ADVAPI32 (deferred) 7c340000 7c34f000 Secur32 (deferred) 7c570000 7c624000 KERNEL32 (deferred) 7cc30000 7cc70000 winsrv (deferred) 80062000 80076f80 hal (deferred) 80400000 805a2940 nt (pdb symbols) a0000000 a0190ce0 win32k (pdb symbols) a0191000 a01e8000 ati2drad (deferred) a01f0000 a0296000 tsdd (no symbols) b4a60000 b4a72320 naveng (deferred) b4a73000 b4b44c40 navex15 (deferred) … … … The bugcheck happens in 3rd-partyApp process context running inside some terminal session: PROCESS_NAME: 3rd-partyApp.exe TRAP_FRAME: b475f84c -- (.trap 0xffffffffb475f84c) ErrCode = 00000000 eax=a020b1d4 ebx=00000000 ecx=04e0443b edx=ffffffff esi=a21b6778 edi=a201b018 eip=a020b1d4 esp=b475f8c0 ebp=b475f900 iopl=3 nv up ei pl zr na pe nc cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00013246 TSDD+0×1b1d4: a020b1d4 ?? ??? Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 154 PART 2: Professional Crash Dump Analysis Examining display driver virtual address shows that it is unknown (PTE is NULL): 0: kd> !pte a020b1d4 A020B1D4 - PDE at C0300A00 PTE at C028082C contains 14AB6863 contains 00000000 pfn 14ab6 –DA–KWV not valid ATI display driver address is unknown too: 0: kd> !pte a0191000 A0191000 - PDE at C0300A00 PTE at C0280644 contains 3E301863 contains 00000000 pfn 3e301 –DA–KWV not valid Let’s switch to another terminal session : PROCESS 87540a60 SessionId: 45 Cid: 3954 Peb: 7ffdf000 ParentCid: 0164 DirBase: 2473d000 ObjectTable: 889b2c48 TableSize: 182. Image: csrss.exe 0: kd> .process /r /p 87540a60 Implicit process is now 87540a60 Loading User Symbols Now TSDD display driver address is valid: 0: kd> !pte a020b1d4 A020B1D4 - PDE at C0300A00 PTE at C028082C contains 5D2C2863 contains 20985863 pfn 5d2c2 –DA–KWV pfn 20985 –DA–KWV but ATI driver address is not. It is unknown and this is expected because no real display hardware is used: 0: kd> !pte a0191000 A0191000 - PDE at C0300A00 PTE at C0280644 contains 5D2C2863 contains 00000000 pfn 5d2c2 –DA–KWV not valid Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. Bugchecks Depicted 155 Let’s switch to session 0 where the display is “physical”: PROCESS 8898a5e0 SessionId: 0 Cid: 0180 Peb: 7ffdf000 ParentCid: 0164 DirBase: 14c58000 ObjectTable: 8898b948 TableSize: 1322. Image: csrss.exe 0: kd> .process /r /p 8898a5e0 Implicit process is now 8898a5e0 Loading User Symbols TSDD driver address is unknown and this is expected too because we no longer use terminal services protocol: 0: kd> !pte a020b1d4 A020B1D4 - PDE at C0300A00 PTE at C028082C contains 14AB6863 contains 00000000 pfn 14ab6 –DA–KWV not valid However ATI display driver addresses are not unknown (not NULL) and their 2 se- lected pages are either in transition or in a page file: 0: kd> !pte a0191000 A0191000 - PDE at C0300A00 PTE at C0280644 contains 14AB6863 contains 156DD882 pfn 14ab6 –DA–KWV not valid Transition: 156dd Protect: 4 0: kd> !pte a0198000 A0198000 - PDE at C0300A00 PTE at C0280660 contains 14AB6863 contains 000B9060 pfn 14ab6 –DA–KWV not valid PageFile 0 Offset b9 Protect: 3 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 156 PART 2: Professional Crash Dump Analysis Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. Manual Stack Trace Reconstruction 157 MANUAL STACK TRACE RECONSTRUCTION This is a small case study to complement Incorrect Stack Trace pattern (page 288) and show how to reconstruct stack trace manually based on an example with complete source code. For it I created a small working multithreaded program: #include "stdafx.h" #include <stdio.h> #include <process.h> typedef void (*REQ_JUMP)(); typedef void (*REQ_RETURN)(); const char str[] = "\0\0\0\0\0\0\0"; bool loop = true; void return_func() { puts("Return Func"); loop = false; _endthread(); } void jump_func() { puts("Jump Func"); } void internal_func_2(void *param_jump,void *param_return) { REQ_JUMP f_jmp = (REQ_JUMP)param_jump; REQ_RETURN f_ret = (REQ_RETURN)param_return; puts("Internal Func 2"); // Uncomment memcpy to crash the program // Overwrite f_jmp and f_ret with NULL // memcpy(&f_ret, str, sizeof(str)); __asm { push f_ret; mov eax, f_jmp mov ebp, 0 // use ebp as a general purpose register jmp eax } } Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 158 PART 2: Professional Crash Dump Analysis void internal_func_1(void *param) { puts("Internal Func 1"); internal_func_2(param, &return_func); } void thread_request(void *param) { puts("Request"); internal_func_1(param); } int _tmain(int argc, _TCHAR* argv[]) { _beginthread(thread_request, 0, (void *)jump_func); while (loop); return 0; } I had to disable optimizations in Visual C++ compiler otherwise most of the code would have been eliminated because the program is very small and easy for code opti- mizer. If we run the program it displays the following output: Request Internal Func 1 Internal Func 2 Jump Func Return Func internal_func_2 gets two parameters: the function address to jump and the func- tion address to call upon the return. The latter sets loop variable to false in order to break infinite main thread loop and calls _endthread. Why we need this complexity in the small sample? Because I wanted to simulate FPO optimization in an inner function call and also gain control over a return address. This is why I set EBP to zero before jumping and pushed the custom return address which I can change any time. If I used the call instruction then the processor would have determined the return address as the next instruction address. The code also copies two internal_func_2 parameters into local variables f_jmp and f_ret because the commented memcpy call is crafted to overwrite them with zeroes and do not touch the saved EBP, return address and function arguments. This is all to make the stack trace incorrect but at the same time to make manual stack reconstruc- tion as easy as possible in this example. Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. Manual Stack Trace Reconstruction 159 Let’s suppose that memcpy call is a bug that overwrites local variables. Then we obviously have a crash because EAX is zero and jump to zero address will cause access violation. EBP is also 0 because we assigned 0 to it explicitly. Let’s pretend that we wanted to pass some constant via EBP and it is zero. What we have now: EBP is 0 EIP is 0 Return address is 0 When we load a crash dump WinDbg is utterly confused because it has no clue on how to reconstruct the stack trace: This dump file has an exception of interest stored in it. The stored exception information can be accessed via .ecxr. (bd0.ec8): Access violation – code c0000005 (first/second chance not available) eax=00000000 ebx=00595620 ecx=00000002 edx=00000000 esi=00000000 edi=00000000 eip=00000000 esp=0069ff54 ebp=00000000 iopl=0 nv up ei pl nz ac po nc cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010212 00000000 ?? ??? 0:001> kv ChildEBP RetAddr Args to Child WARNING: Frame IP not in any known module. Following frames may be wrong. 0069ff50 00000000 00000000 00000000 0069ff70 0×0 Fortunately ESP is not zero so we can look at raw stack: 0:001> dds esp 0069ff54 00000000 0069ff58 00000000 0069ff5c 00000000 0069ff60 0069ff70 0069ff64 0040187f WrongIP!internal_func_1+0x1f 0069ff68 00401830 WrongIP!jump_func 0069ff6c 00401840 WrongIP!return_func 0069ff70 0069ff7c 0069ff74 0040189c WrongIP!thread_request+0xc 0069ff78 00401830 WrongIP!jump_func 0069ff7c 0069ffb4 0069ff80 78132848 msvcr80!_endthread+0x4b 0069ff84 00401830 WrongIP!jump_func 0069ff88 aa75565b 0069ff8c 00000000 0069ff90 00000000 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 160 PART 2: Professional Crash Dump Analysis 0069ff94 00595620 0069ff98 c0000005 0069ff9c 0069ff88 0069ffa0 0069fb34 0069ffa4 0069ffdc 0069ffa8 78138cd9 msvcr80!_except_handler4 0069ffac d207e277 0069ffb0 00000000 0069ffb4 0069ffec 0069ffb8 781328c8 msvcr80!_endthread+0xcb 0069ffbc 7d4dfe21 kernel32!BaseThreadStart+0x34 0069ffc0 00595620 0069ffc4 00000000 0069ffc8 00000000 0069ffcc 00595620 0069ffd0 c0000005 Here we can start searching for the following pairs: EBP: PreviousEBP Function return address … … … PreviousEBP: PrePreviousEBP Function return address … … … for example: 0:001> dds esp 0069ff54 00000000 0069ff58 00000000 0069ff5c 00000000 0069ff60 0069ff70 0069ff64 0040187f WrongIP!internal_func_1+0×1f 0069ff68 00401830 WrongIP!jump_func 0069ff6c 00401840 WrongIP!return_func 0069ff70 0069ff7c 0069ff74 0040189c WrongIP!thread_request+0xc 0069ff78 00401830 WrongIP!jump_func 0069ff7c 0069ffb4 This is based on the fact that a function call saves its return address and the stan- dard function prolog saves the previous EBP value and sets ESP to point to it. Push ebp mov ebp, esp Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. [...]... Enabled Enabled Enabled Enabled Enabled Enabled Enabled Enabled Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 170 PART 2: Professional Crash Dump Analysis EXPORTED NTDLL AND KERNEL STRUCTURES During crash dump analysis or debugging session we can forget exact structure name when we want to use it in dt WinDbg command In this case wildcards can help us: dt module!*, for example:... 05e8f5d0 05e8f740 05e8f5d4 0633adfc "drwtsn32 -p %ld -e %ld -g" 05e8f5d8 00000000 Of course, we can apply these commands to any memory range, not only stack Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 168 PART 2: Professional Crash Dump Analysis TRACING WIN32 API WHILE DEBUGGING A PROCESS If we load an executable or attach WinDbg to an existing process then we can... command doesn’t work due to the lack of proper MS symbols but it reports TEB address and we can dump it: 0:001> !teb TEB at 7efda000 error InitTypeRead( TEB )… 0:001> dd 7efda000 l3 7efda000 0069ffa4 006a0000 0069e000 Usually the second double word is the stack limit and the third is the stack base address so we can dump the range and start reconstructing stack trace for our example from the bottom of the... ntdll!NtRaiseException+0×12 0069fc28 7d61ea51 ntdll!KiUserExceptionDispatcher+0×29 0069fc2c 0069fc38 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 166 PART 2: Professional Crash Dump Analysis … … … 0069ff38 0069ff3c 0069ff40 0069ff44 0069ff48 0069ff4c 0069ff50 0069ff54 0069ff58 0069ff5c 0069ff60 0069ff64 0069ff68 0069ff6c 0069ff70 0069ff74 0069ff78 0069ff7c 0069ff80 0069ff84 0069ff88... mov eax,dword ptr [ebp+8] 00401889 push eax 0040188a call WrongIP!internal_func_2 (004017e0) Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 162 PART 2: Professional Crash Dump Analysis 0:001> ub WrongIP!thread_request+0xc WrongIP!internal_func_1+0x2d: 0040189d int 3 0040189e int 3 0040189f int 3 WrongIP!thread_request: 004018a0 push ebp 004018a1 mov ebp,esp 004018a3 mov eax,dword... variable From the code we see that it comes from [EBP+8] which is the first function parameter: Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 164 PART 2: Professional Crash Dump Analysis EBP+C: EBP+8: EBP+4: EBP: EBP-4: EBP-8: second parameter first parameter return address previous EBP local variable local variable If we examine the first parameter we would see that it is... ntdll!_DISPATCHER_HEADER ntdll!_PF_SCENARIO_TYPE ntdll!_HANDLE_TRACE_DEBUG_INFO ntdll!_HANDLE_TABLE Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 172 PART 2: Professional Crash Dump Analysis ntdll!_KWAIT_BLOCK ntdll!_MMSUPPORT_FLAGS ntdll!_MMWSL ntdll!_MMSUPPORT ntdll!_EPROCESS_QUOTA_ENTRY ntdll!_EPROCESS_QUOTA_BLOCK ntdll!_UNICODE_STRING ntdll!_NT_TIB ntdll!_PS_JOB_TOKEN_FILTER... ntdll!_OBJECT _DUMP_ CONTROL ntdll!_OB_OPEN_REASON ntdll!_ACCESS_STATE ntdll!_SECURITY_OPERATION_CODE ntdll!_OBJECT_NAME_INFORMATION ntdll!_OBJECT_TYPE_INITIALIZER ntdll!_LARGE_INTEGER ntdll!_RTL_TRACE_BLOCK ntdll!_HEAP_UCR_SEGMENT ntdll!_KEXECUTE_OPTIONS ntdll!_OWNER_ENTRY ntdll!_ERESOURCE ntdll!_GENERIC_MAPPING ntdll!_SID_AND_ATTRIBUTES ntdll!_LUID_AND_ATTRIBUTES ntdll!_PS_JOB_TOKEN_FILTER ntdll! _MEMORY_ CACHING_TYPE_ORIG... ntdll!_KAPC_STATE ntdll!_MODE ntdll!_HEAP_PSEUDO_TAG_ENTRY ntdll!_RTL_CRITICAL_SECTION_DEBUG Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 174 PART 2: Professional Crash Dump Analysis ntdll!_RTL_CRITICAL_SECTION ntdll!_HEAP_SEGMENT ntdll!_KTRAP_FRAME ntdll!_KGDTENTRY ntdll!_KDEVICE_QUEUE_ENTRY ntdll!_IO_ALLOCATION_ACTION ntdll!_WAIT_CONTEXT_BLOCK ntdll!_KTIMER ntdll!_MDL... ntdll!_NAMED_PIPE_CREATE_PARAMETERS ntdll!_MAILSLOT_CREATE_PARAMETERS ntdll!_FILE_INFORMATION_CLASS Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 176 PART 2: Professional Crash Dump Analysis ntdll!_FSINFOCLASS ntdll!_SCSI_REQUEST_BLOCK ntdll!_FILE_GET_QUOTA_INFORMATION ntdll!_DEVICE_RELATION_TYPE ntdll!_GUID ntdll!_INTERFACE ntdll!_DEVICE_CAPABILITIES ntdll!_IO_RESOURCE_REQUIREMENTS_LIST . can avoid requesting complete memory dump of physi- cal memory and ask for kernel memory dump only together with a user dump. Note: do not set this option. watermark. 170 PART 2: Professional Crash Dump Analysis EXPORTED NTDLL AND KERNEL STRUCTURES During crash dump analysis or debugging session we can forget