High Contention 421 HIGH CONTENTION Some Windows synchronization objects like executive resources and critical sec- tions have a struct member called ContentionCount. This is the number of times a resource was accessed or, in another words, it is the accumulated number of threads waiting for an object: when a thread tries to acquire an object and is put into a wait state the count is incremented. Hence the name of this pattern: High Contention. Here is an example. In a kernel memory dump we have just one exclu- sively owned lock and it seems that no other threads were blocked by it at the time the dump was saved. However the high contention count reveals CPU spike: 3: kd> !locks **** DUMP OF ALL RESOURCE OBJECTS **** KD: Scanning for held locks . Resource @ 0x8abc11f0 Exclusively owned Contention Count = 19648535 Threads: 896395f8-01<*> KD: Scanning for held locks… Resource @ 0x896fab88 Shared 1 owning threads Threads: 88c78608-01<*> KD: Scanning for held locks . 15464 total locks, 2 locks currently held 3: kd> !thread 896395f8 THREAD 896395f8 Cid 04c0.0138 Teb: 7ffde000 Win32Thread: bc922d20 RUNNING on processor 1 Not impersonating DeviceMap e3d4c008 Owning Process 8a035020 Image: MyApp.exe Wait Start TickCount 36969283 Ticks: 0 Context Switch Count 1926423 LargeStack UserTime 00:00:53.843 KernelTime 00:13:10.703 Win32 Start Address 0×00401478 Start Address 0×77e617f8 Stack Init ba14b000 Current ba14abf8 Base ba14b000 Limit ba146000 Call 0 Priority 11 BasePriority 6 PriorityDecrement 5 ChildEBP RetAddr ba14ac94 bf8c6505 001544c8 bf995948 000c000a nt!_wcsicmp+0×3a ba14ace0 bf8c6682 00000000 00000000 00000000 win32k!_FindWindowEx+0xfb ba14ad48 8088978c 00000000 00000000 0012f8d4 win32k!NtUserFindWindowEx+0xef ba14ad48 7c8285ec 00000000 00000000 0012f8d4 nt!KiFastCallEntry+0xfc Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 422 PART 3: Crash Dump Analysis Patterns 3: kd> !process 8a035020 PROCESS 8a035020 SessionId: 9 Cid: 04c0 Peb: 7ffdf000 ParentCid: 10e8 DirBase: cffaf7a0 ObjectTable: e4ba30a0 HandleCount: 73. Image: MyApp.exe VadRoot 88bc1bf8 Vads 82 Clone 0 Private 264. Modified 0. Locked 0. DeviceMap e3d4c008 Token e5272028 ElapsedTime 00:14:19.360 UserTime 00:00:53.843 KernelTime 00:13:10.703 QuotaPoolUsage[PagedPool] 40660 QuotaPoolUsage[NonPagedPool] 3280 Working Set Sizes (now,min,max) (1139, 50, 345) (4556KB, 200KB, 1380KB) PeakWorkingSetSize 1141 VirtualSize 25 Mb PeakVirtualSize 27 Mb PageFaultCount 1186 MemoryPriority BACKGROUND BasePriority 6 CommitCharge 315 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. Accidental Lock 423 ACCIDENTAL LOCK When a system is unresponsive or sluggish we usually check _ERESOURCE locks in kernel or complete memory dumps to see Deadlock (page 323) or High Resource Contention (page 421) patterns. However there is some chance that reported locks are purely accidental and appear in a crash dump because they just happened at that time. We need to look at Contention Count, Ticks and KernelTime in both blocking and blocked threads to recognize an Accidental Lock. Also the current version of WinDbg doesn’t distinguish between prolonged and accidental locks when we use !analyze -v -hang command and merely reports some lock chain it finds among equal alternatives. Here is an example. The system was reported hang and kernel memory dump was saved. WinDbg analysis command reports one thread blocking 3 other threads and the driver on top of the blocking thread stack is AVDriver.sys. The algorithm WinDbg uses to point to specific image name is described in Minidump Analysis section (page 43) and in our case it chooses AVDriver: BLOCKED_THREAD: 8089d8c0 BLOCKING_THREAD: 8aab4700 LOCK_ADDRESS: 8859a570 -- (!locks 8859a570) Resource @ 0x8859a570 Exclusively owned Contention Count = 3 NumberOfExclusiveWaiters = 3 Threads: 8aab4700-01<*> Threads Waiting On Exclusive Access: 885d0020 88a7c020 8aafc7d8 1 total locks, 1 locks currently held BUGCHECK_STR: LOCK_HELD FAULTING_THREAD: 8aab4700 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 424 PART 3: Crash Dump Analysis Patterns STACK_TEXT: f592f698 80832f7a nt!KiSwapContext+0x26 f592f6c4 80828705 nt!KiSwapThread+0x284 f592f70c f720a394 nt!KeDelayExecutionThread+0x2ab WARNING: Stack unwind information not available. Following frames may be wrong. f592f734 f720ae35 AVDriver+0×1394 f592f750 f720b208 AVDriver+0×1e35 f592f794 f721945a AVDriver+0×2208 f592f7cc 8081dcdf AVDriver+0×1045a f592f7e0 f5b9f76a nt!IofCallDriver+0×45 f592f7f0 f5b9c621 Driver!FS_Dispatch+0xa4 f592f7fc 8081dcdf Driver!Kernel_dispatch+0×53 f592f810 f5eb2856 nt!IofCallDriver+0×45 f592f874 8081dcdf AVFilter!QueryFullName+0×5c10 f592f888 f5e9eae3 nt!IofCallDriver+0×45 f592f8b8 f5e9eca4 DrvFilter!PassThrough+0×115 f592f8d4 8081dcdf DrvFilter!Create+0xda f592f8e8 808f8275 nt!IofCallDriver+0×45 f592f9d0 808f86bc nt!IopParseDevice+0xa35 f592fa08 80936689 nt!IopParseFile+0×46 f592fa88 80932e04 nt!ObpLookupObjectName+0×11f f592fadc 808ea231 nt!ObOpenObjectByName+0xea f592fb58 808eb4cb nt!IopCreateFile+0×447 f592fbb4 f57c8efd nt!IoCreateFile+0xa3 f592fc24 f57c9f29 srv!SrvIoCreateFile+0×36d f592fcf0 f57ca5e4 srv!SrvNtCreateFile+0×5cc f592fd78 f57adbc6 srv!SrvSmbNtCreateAndX+0×15c f592fd84 f57c3451 srv!SrvProcessSmb+0xb7 f592fdac 80948bd0 srv!WorkerThread+0×138 f592fddc 8088d4e2 nt!PspSystemThreadStartup+0×2e 00000000 00000000 nt!KiThreadStartup+0×16 STACK_COMMAND: .thread 0xffffffff8aab4700 ; kb FOLLOWUP_IP: AVDriver+1394 f720a394 eb85 jmp AVDriver+0x131b (f720a31b) MODULE_NAME: AVDriver IMAGE_NAME: AVDriver.sys Motivated by this “discovery” we want to see all locks: 0: kd> !locks **** DUMP OF ALL RESOURCE OBJECTS **** KD: Scanning for held locks . Resource @ 0x895a62d8 Shared 1 owning threads Threads: 89570520-01<*> Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. Accidental Lock 425 Resource @ 0x897ceba8 Shared 1 owning threads Threads: 89584020-01<*> Resource @ 0x8958e020 Shared 1 owning threads Threads: 89555020-01<*> Resource @ 0x89590608 Shared 1 owning threads Threads: 89666020-01<*> Resource @ 0x89efc398 Shared 1 owning threads Threads: 89e277c0-01<*> Resource @ 0x88d70820 Shared 1 owning threads Threads: 88e43948-01<*> Resource @ 0x89f2fb00 Shared 1 owning threads Threads: 89674688-01<*> Resource @ 0x89c80370 Shared 1 owning threads Threads: 888496b8-01<*> Resource @ 0x89bfdf08 Shared 1 owning threads Threads: 88b62910-01<*> Resource @ 0x888b5488 Shared 1 owning threads Threads: 88536730-01<*> Resource @ 0x89f2e348 Shared 1 owning threads Threads: 89295930-01<*> Resource @ 0x891a0838 Shared 1 owning threads Threads: 88949020-01<*> Resource @ 0x8825bf08 Shared 1 owning threads Threads: 882b9a08-01<*> Resource @ 0x881a6510 Shared 1 owning threads Threads: 88a88338-01<*> Resource @ 0x885c5890 Shared 1 owning threads Threads: 881ab020-01<*> Resource @ 0x886633a8 Shared 1 owning threads Threads: 89b5f8b0-01<*> Resource @ 0x88216390 Shared 1 owning threads Threads: 88820020-01<*> Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 426 PART 3: Crash Dump Analysis Patterns Resource @ 0x88524490 Shared 1 owning threads Threads: 88073020-01<*> Resource @ 0x88f6a020 Shared 1 owning threads Threads: 88e547b0-01<*> Resource @ 0x88cf2020 Shared 1 owning threads Threads: 89af32d8-01<*> Resource @ 0x889cea80 Shared 1 owning threads Threads: 88d18b40-01<*> Resource @ 0x88486298 Shared 1 owning threads Threads: 88af7db0-01<*> Resource @ 0x88b22270 Exclusively owned Contention Count = 4 NumberOfExclusiveWaiters = 4 Threads: 8aad07d8-01<*> Threads Waiting On Exclusive Access: 8ad78020 887abdb0 88eb39a8 8aa1f668 Resource @ 0x88748c20 Exclusively owned Contention Count = 2 NumberOfExclusiveWaiters = 2 Threads: 8873c8d8-01<*> Threads Waiting On Exclusive Access: 88477478 88db6020 Resource @ 0x8859a570 Exclusively owned Contention Count = 3 NumberOfExclusiveWaiters = 3 Threads: 8aab4700-01<*> Threads Waiting On Exclusive Access: 885d0020 88a7c020 8aafc7d8 KD: Scanning for held locks . 18911 total locks, 25 locks currently held We can ignore shared locks and then concentrate on the last 3 exclusively owned resources. It looks suspicious that Contention Count has the same number as the num- ber of threads waiting on exclusive access (NumberOfExclusiveWaiters). This means that these resources had never been used before. If we dump locks verbosely we would see that blocked threads had been waiting no more than 2 seconds, for example, for resource 0×8859a570: Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. Accidental Lock 427 0: kd> !thread 885d0020; !thread 88a7c020; !thread 8aafc7d8 THREAD 885d0020 Cid 0004.1c34 Teb: 00000000 Win32Thread: 00000000 WAIT: (Unknown) KernelMode Non-Alertable 89908d50 SynchronizationEvent 885d0098 NotificationTimer Not impersonating DeviceMap e10022c8 Owning Process 8ad80648 Image: System Wait Start TickCount 7689055 Ticks: 127 (0:00:00:01.984) Context Switch Count 248 UserTime 00:00:00.000 KernelTime 00:00:00.000 Start Address srv!WorkerThread (0xf57c3394) Stack Init b4136000 Current b4135b74 Base b4136000 Limit b4133000 Call 0 Priority 9 BasePriority 9 PriorityDecrement 0 ChildEBP RetAddr b4135b8c 80832f7a nt!KiSwapContext+0×26 b4135bb8 8082925c nt!KiSwapThread+0×284 b4135c00 8087c1ad nt!KeWaitForSingleObject+0×346 b4135c3c 8087c3a1 nt!ExpWaitForResource+0xd5 b4135c5c f57c9e95 nt!ExAcquireResourceExclusiveLite+0×8d b4135cf0 f57ca5e4 srv!SrvNtCreateFile+0×510 b4135d78 f57adbc6 srv!SrvSmbNtCreateAndX+0×15c b4135d84 f57c3451 srv!SrvProcessSmb+0xb7 b4135dac 80948bd0 srv!WorkerThread+0×138 b4135ddc 8088d4e2 nt!PspSystemThreadStartup+0×2e 00000000 00000000 nt!KiThreadStartup+0×16 THREAD 88a7c020 Cid 0004.3448 Teb: 00000000 Win32Thread: 00000000 WAIT: (Unknown) KernelMode Non-Alertable 89908d50 SynchronizationEvent 88a7c098 NotificationTimer Not impersonating DeviceMap e10022c8 Owning Process 8ad80648 Image: System Wait Start TickCount 7689112 Ticks: 70 (0:00:00:01.093) Context Switch Count 210 UserTime 00:00:00.000 KernelTime 00:00:00.000 Start Address srv!WorkerThread (0xf57c3394) Stack Init b55dd000 Current b55dcb74 Base b55dd000 Limit b55da000 Call 0 Priority 9 BasePriority 9 PriorityDecrement 0 ChildEBP RetAddr b55dcb8c 80832f7a nt!KiSwapContext+0×26 b55dcbb8 8082925c nt!KiSwapThread+0×284 b55dcc00 8087c1ad nt!KeWaitForSingleObject+0×346 b55dcc3c 8087c3a1 nt!ExpWaitForResource+0xd5 b55dcc5c f57c9e95 nt!ExAcquireResourceExclusiveLite+0×8d b55dccf0 f57ca5e4 srv!SrvNtCreateFile+0×510 b55dcd78 f57adbc6 srv!SrvSmbNtCreateAndX+0×15c b55dcd84 f57c3451 srv!SrvProcessSmb+0xb7 b55dcdac 80948bd0 srv!WorkerThread+0×138 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 428 PART 3: Crash Dump Analysis Patterns b55dcddc 8088d4e2 nt!PspSystemThreadStartup+0×2e 00000000 00000000 nt!KiThreadStartup+0×16 THREAD 8aafc7d8 Cid 0004.058c Teb: 00000000 Win32Thread: 00000000 WAIT: (Unknown) KernelMode Non-Alertable 89908d50 SynchronizationEvent 8aafc850 NotificationTimer Not impersonating DeviceMap e10022c8 Owning Process 8ad80648 Image: System Wait Start TickCount 7689171 Ticks: 11 (0:00:00:00.171) Context Switch Count 310 UserTime 00:00:00.000 KernelTime 00:00:00.000 Start Address srv!WorkerThread (0xf57c3394) Stack Init f592c000 Current f592bb18 Base f592c000 Limit f5929000 Call 0 Priority 9 BasePriority 9 PriorityDecrement 0 ChildEBP RetAddr f592bb30 80832f7a nt!KiSwapContext+0×26 f592bb5c 8082925c nt!KiSwapThread+0×284 f592bba4 8087c1ad nt!KeWaitForSingleObject+0×346 f592bbe0 8087c3a1 nt!ExpWaitForResource+0xd5 f592bc00 f57c8267 nt!ExAcquireResourceExclusiveLite+0×8d f592bc18 f57ff0ed srv!UnlinkRfcbFromLfcb+0×33 f592bc34 f57ff2ea srv!SrvCompleteRfcbClose+0×1df f592bc54 f57b5e8f srv!CloseRfcbInternal+0xb6 f592bc78 f57ce8a9 srv!SrvCloseRfcbsOnSessionOrPid+0×74 f592bc94 f57e2b22 srv!SrvCloseSession+0xb0 f592bcb8 f57aeb12 srv!SrvCloseSessionsOnConnection+0xa9 f592bcd4 f57c79ed srv!SrvCloseConnection+0×143 f592bd04 f5808c50 srv!SrvCloseConnectionsFromClient+0×17f f592bdac 80948bd0 srv!WorkerThread+0×138 f592bddc 8088d4e2 nt!PspSystemThreadStartup+0×2e 00000000 00000000 nt!KiThreadStartup+0×16 Blocking threads themselves are not blocked and active: the number of ticks passed since their last wait or preemption is 0. This could be a sign of CPU spike pattern. However their accumulated KernelTime is less than a second: 0: kd> !thread 8aad07d8 THREAD 8aad07d8 Cid 0004.0580 Teb: 00000000 Win32Thread: 00000000 WAIT: (Unknown) KernelMode Non-Alertable 8aad0850 NotificationTimer IRP List: 8927ade0: (0006,0220) Flags: 00000884 Mdl: 00000000 Impersonation token: eafdc030 (Level Impersonation) DeviceMap e5d69340 Owning Process 8ad80648 Image: System Wait Start TickCount 7689182 Ticks: 0 Context Switch Count 915582 UserTime 00:00:00.000 KernelTime 00:00:00.125 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. Accidental Lock 429 Start Address srv!WorkerThread (0xf57c3394) Stack Init f59d8000 Current f59d7680 Base f59d8000 Limit f59d5000 Call 0 Priority 9 BasePriority 9 PriorityDecrement 0 0: kd> !thread 8873c8d8 THREAD 8873c8d8 Cid 0004.2898 Teb: 00000000 Win32Thread: 00000000 WAIT: (Unknown) KernelMode Non-Alertable 8873c950 NotificationTimer IRP List: 882a8de0: (0006,0220) Flags: 00000884 Mdl: 00000000 Impersonation token: eafdc030 (Level Impersonation) DeviceMap e5d69340 Owning Process 8ad80648 Image: System Wait Start TickCount 7689182 Ticks: 0 Context Switch Count 917832 UserTime 00:00:00.000 KernelTime 00:00:00.031 Start Address srv!WorkerThread (0xf57c3394) Stack Init ac320000 Current ac31f680 Base ac320000 Limit ac31d000 Call 0 Priority 9 BasePriority 9 PriorityDecrement 0 0: kd> !thread 8aab4700 THREAD 8aab4700 Cid 0004.0588 Teb: 00000000 Win32Thread: 00000000 WAIT: (Unknown) KernelMode Non-Alertable 8aab4778 NotificationTimer IRP List: 88453008: (0006,0220) Flags: 00000884 Mdl: 00000000 Impersonation token: e9a82728 (Level Impersonation) DeviceMap eb45f108 Owning Process 8ad80648 Image: System Wait Start TickCount 7689182 Ticks: 0 Context Switch Count 1028220 UserTime 00:00:00.000 KernelTime 00:00:00.765 Start Address srv!WorkerThread (0xf57c3394) Stack Init f5930000 Current f592f680 Base f5930000 Limit f592d000 Call 0 Priority 9 BasePriority 9 PriorityDecrement 0 Based on this observation we could say that locks were accidental and indeed, when the problem happened again, the new dump didn’t show them. Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 430 PART 3: Crash Dump Analysis Patterns PASSIVE THREAD (USER SPACE) When trying to understand why the particular application or service hangs we look at Stack Trace Collection pattern (page 409) and hope to find some suspicious threads that are waiting for a response. These are active blocked threads. Other threads may appear waiting but they are merely waiting for some notification or data that may or may not come during their lifetime and, therefore, are normal. In other words, they are passive and hence the name of the pattern Passive Thread. Typical examples from user space include The main service thread and dispatch threads (when idle). A thread waiting for file or registry notifications. A generic RPC/LPC/COM thread waiting for messages. Worker threads waiting for a data to appear in a queue. Window message loops (when idle). Socket and network protocol threads (when idle). A thread with function names on its stack trace suggesting that it is a notifica- tion or listener thread. Of course, sometimes these passive threads can be the reason for an application or service hang, but from my experience, most of the time they are not, unless there are other threads which they block. Let’s now look at example stack traces. NOTE: Generic threads spawned to service various requests and waiting for data to arrive can be filtered using !uniqstack WinDbg command. Conceptually these threads are part of the so called thread pool software design pattern. LPC/RPC/COM threads waiting for requests: 70 Id: 8f8.1100 Suspend: 1 Teb: 7ff80000 Unfrozen ChildEBP RetAddr 0d82fe18 7c82783b ntdll!KiFastSystemCallRet 0d82fe1c 77c885ac ntdll!NtReplyWaitReceivePortEx+0xc 0d82ff84 77c88792 rpcrt4!LRPC_ADDRESS::ReceiveLotsaCalls+0x198 0d82ff8c 77c8872d rpcrt4!RecvLotsaCallsWrapper+0xd 0d82ffac 77c7b110 rpcrt4!BaseCachedThreadRoutine+0x9d 0d82ffb8 77e64829 rpcrt4!ThreadStartRoutine+0x1b 0d82ffec 00000000 kernel32!BaseThreadStart+0x34 71 Id: 8f8.1e44 Suspend: 1 Teb: 7ffde000 Unfrozen ChildEBP RetAddr 0c01fe18 7c82783b ntdll!KiFastSystemCallRet Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. [...]... ntdll!_RtlUserThreadStart+0×23 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 440 PART 3: Crash Dump Analysis Patterns INSUFFICIENT MEMORY (KERNEL POOL) Although handle leaks may result in insufficient pool memory, many drivers allocate their own private memory and specify a 4-letter ASCII tag, for example, here is nonpaged pool from my x64 Vista workstation (shown in small... Crash Dump Analysis Patterns MAIN THREAD When we look at a thread and it is not in the Passive Thread pattern list (page 430) and it looks more like Blocked Thread (see Volume 2) we may ask whether it is Main Thread Every process has at least one thread of execution called main or primary thread Most GUI applications have window message processing loop inside their main process thread When a memory dump. .. object FCB_DATA , Binary: Here Microsoft article KB312362 might help Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 444 PART 3: Crash Dump Analysis Patterns Non-paged pool 0: kd> !vm *** Virtual Memory Usage *** Physical Memory: 851775 ( 3407100 Kb) Page File: \??\C:\pagefile.sys Current: 4190208 Kb Free Space: 4175708 Kb Minimum: 4190208 Kb Maximum: 4190208 Kb Available Pages:... = = = = =.= == .== = = = = = = = =.== .= = =.= = Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 446 PART 3: Crash Dump Analysis Patterns Here is another example: 0: kd> !vm *** Virtual Memory Usage *** Physical Memory: 786299 ( 3145196 Page File: \??\C:\pagefile.sys Current: 4193280Kb Free Space: Minimum: 4193280Kb Maximum: Available Pages: 200189 ( 800756 ResAvail... NumberOfExclusiveWaiters = 21 Shared 10 owning threads Therefore we see at least two problems in this memory dump: excessive paged pool usage and high thread contention around registry resource slowing down if not halting the system Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 448 PART 3: Crash Dump Analysis Patterns BUSY SYSTEM If there are no CPU-bound threads in a system then most... corresponding driver We can also use memory search in WinDbg to locate kernel space addresses and see what modules they correspond to WinDbg shows the number of failed pool allocations and also shows a message when pool usage is nearly its maximum Below I put some examples with possible troubleshooting hints Session pool 3: kd> !vm *** Virtual Memory Usage *** Physical Memory: 1572637 ( 6290548 Kb) Page... Used: NonPaged Tag Allocs Frees Ddk 9074558 3859522 (user‘s of ntddk.h) MmCm 43787 42677 Diff 5215036 1110 Default for driver allocated memory Calls made to MmAllocateContiguousMemory Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark Insufficient Memory (Kernel Pool) 445 , Binary: nt!mm LSwi 1 0 TCPt 3281838 3281808 1 30 initial work context TCP/IP network protocol , Binary: TCP... pool: 3: kd> !vm 4 Terminal Server Memory Usage By Session: Session Paged Pool Maximum is 32768K Session View Space Maximum is 20480K Session ID 0 @ f79a1000: Paged Pool Usage: 9824K Commit Usage: 10148K Session ID 2 @ f7989000: Paged Pool Usage: 1212K Commit Usage: 2180K Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark 442 PART 3: Crash Dump Analysis Patterns Session ID 9 @... non-paged pool it is not then most likely failed allocations were from paged pool: Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark Insufficient Memory (Kernel Pool) 443 0: kd> !vm *** Virtual Memory Usage *** Physical Memory: 4193696 ( 16774784 Kb) Page File: \??\C:\pagefile.sys Current: 4193280 Kb Free Space: 3313120 Kb Minimum: 4193280 Kb Maximum: 4193280 Kb Available Pages: 3210617... have a memory dump one of running threads would be the one that called KeBugCheck(Ex) function 3: kd> !running System Processors f (affinity mask) Idle Processors d 1 Prcb f7737120 Current 8a3da020 Next 3: kd> !thread 8a3da020 1f THREAD 8a3da020 Cid 0ebc.0dec Teb: 7ffdf000 Win32Thread: bc002328 RUNNING on processor 1 Not impersonating DeviceMap e3e3e080 Owning Process 8a0aea88 Image: SystemDump.exe . watermark. 440 PART 3: Crash Dump Analysis Patterns INSUFFICIENT MEMORY (KERNEL POOL) Although handle leaks may result in insufficient pool memory, many drivers. Here is an example. The system was reported hang and kernel memory dump was saved. WinDbg analysis command reports one thread blocking 3 other threads