title: Windows Server Crash Dump Analysis
body: <p>I'm not certain that this is the right venue for this question, but a programmer friend of mine said I should try this here.</p>
  
  <p>My company's main application is hosted on a terminal server running Windows Server 2008.  Since last Thursday we have seen this server crash and reboot 3 times, and we just went live with this server on the previous Tuesday.  I have used the the WinDbg program to analyze the crash dump file, but I'm a little outside by depth at this point and I'm hoping that someone out there can help me get this issue resolved.</p>
  
  <p>The application that appears to me to be at fault is winoac.exe which is the executable for SmartWare 4.5 (www.smartware4.com).  This is the platform that our application runs on.  If this application is at fault, is there anything that I can do about it, other than complaining to SmartWare?</p>
  
  <p>Thanks a million to anybody that can help.</p>
  
  <p>Here are the results of the analysis.</p>
  
  <pre><code>Microsoft (R) Windows Debugger Version 6.10.0003.233 X86
  Copyright (c) Microsoft Corporation. All rights reserved.
  
  
  Loading Dump File [C:\Users\esinnard\Desktop\Windows Dumps\1-29-09\MEMORY.DMP]
  Kernel Summary Dump File: Only kernel address space is available
  
  Symbol search path is: SRV*C:\ProgramData\Symbols*http://msdl.microsoft.com/download/symbols
  Executable search path is: 
  Windows Server 2008/Windows Vista SP1 Kernel Version 6001 (Service Pack 1) MP (8 procs) Free x86 compatible
  Product: Server, suite: TerminalServer
  Built by: 6001.18145.x86fre.vistasp1_gdr.080917-1612
  Machine Name:
  Kernel base = 0x81c41000 PsLoadedModuleList = 0x81d4e930
  Debug session time: Thu Jan 29 12:49:43.870 2009 (GMT-6)
  System Uptime: 0 days 11:18:08.929
  Loading Kernel Symbols
  ...............................................................
  ................................................................
  ..............
  Loading User Symbols
  PEB is paged out (Peb.Ldr = 7ffd400c).  Type ".hh dbgerr001" for details
  Loading unloaded module list
  .....
  *******************************************************************************
  *                                                                             *
  *                        Bugcheck Analysis                                    *
  *                                                                             *
  *******************************************************************************
  
  Use !analyze -v to get detailed debugging information.
  
  BugCheck 8E, {c0000005, 81c88043, 9cef0840, 0}
  
  Page bd1f2 not present in the dump file. Type ".hh dbgerr004" for details
  Page bc9c3 not present in the dump file. Type ".hh dbgerr004" for details
  PEB is paged out (Peb.Ldr = 7ffd400c).  Type ".hh dbgerr001" for details
  PEB is paged out (Peb.Ldr = 7ffd400c).  Type ".hh dbgerr001" for details
  Probably caused by : RDPDD.dll ( RDPDD!OE2_TableEncodeOrderFields+11e )
  
  Followup: MachineOwner
  ---------
  
  7: kd&gt; !analyze -v
  *******************************************************************************
  *                                                                             *
  *                        Bugcheck Analysis                                    *
  *                                                                             *
  *******************************************************************************
  
  KERNEL_MODE_EXCEPTION_NOT_HANDLED (8e)
  This is a very common bugcheck.  Usually the exception address pinpoints
  the driver/function that caused the problem.  Always note this address
  as well as the link date of the driver/image that contains this address.
  Some common problems are exception code 0x80000003.  This means a hard
  coded breakpoint or assertion was hit, but this system was booted
  /NODEBUG.  This is not supposed to happen as developers should never have
  hardcoded breakpoints in retail code, but ...
  If this happens, make sure a debugger gets connected, and the
  system is booted /DEBUG.  This will let us see why this breakpoint is
  happening.
  Arguments:
  Arg1: c0000005, The exception code that was not handled
  Arg2: 81c88043, The address that the exception occurred at
  Arg3: 9cef0840, Trap Frame
  Arg4: 00000000
  
  Debugging Details:
  ------------------
  
  PEB is paged out (Peb.Ldr = 7ffd400c).  Type ".hh dbgerr001" for details
  PEB is paged out (Peb.Ldr = 7ffd400c).  Type ".hh dbgerr001" for details
  
  EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.
  
  FAULTING_IP: 
  nt!RtlInitUnicodeString+1b
  81c88043 f266af          repne scas word ptr es:[edi]
  
  TRAP_FRAME:  9cef0840 -- (.trap 0xffffffff9cef0840)
  ErrCode = 00000000
  eax=00000000 ebx=fe414fd8 ecx=ffffffec edx=9cef0914 esi=fe40fcf0 edi=fe415000
  eip=81c88043 esp=9cef08b4 ebp=9cef0924 iopl=0         nv up ei pl zr na pe nc
  cs=0008  ss=0010  ds=0023  es=0023  fs=0030  gs=0000             efl=00010246
  nt!RtlInitUnicodeString+0x1b:
  81c88043 f266af          repne scas word ptr es:[edi]
  Resetting default scope
  
  DEFAULT_BUCKET_ID:  VISTA_DRIVER_FAULT
  
  BUGCHECK_STR:  0x8E
  
  PROCESS_NAME:  WINOAC.EXE
  
  CURRENT_IRQL:  0
  
  LAST_CONTROL_TRANSFER:  from 81c72fbe to 81cfc759
  
  STACK_TEXT:  
  9cef0400 81c72fbe 0000008e c0000005 81c88043 nt!KeBugCheckEx+0x1e
  9cef07d0 81c9953a 9cef07ec 00000000 9cef0840 nt!KiDispatchException+0x1a9
  9cef0838 81c994ee 9cef0924 81c88043 badb0d00 nt!CommonDispatchException+0x4a
  9cef085c 9976011a 99771680 997708e8 00000000 nt!Kei386EoiHelper+0x186
  9cef0924 9959efab 5d0102bb 00000006 00000002 RDPDD!OE2_TableEncodeOrderFields+0x11e
  9cef0a0c 995aeaf8 5d0102bb 00000006 00000002 win32k!xxxRealDrawMenuItem+0x80b
  9cef0abc 9958455b 5d0102bb 0110007e 9cef0b04 win32k!xxxDrawState+0x1c9
  9cef0b2c 995853e1 5d0102bb fe40fc78 00c8d0d4 win32k!xxxDrawMenuItem+0x3f8
  9cef0b98 9959f511 5d0102bb 00000000 fe414570 win32k!xxxMenuDraw+0x1f2
  9cef0bf0 994ed1d6 00000017 5d0102bb 00000004 win32k!xxxMenuBarDraw+0x1bf
  9cef0c38 9950c0f5 fe414570 5d0102bb 00000001 win32k!xxxDrawWindowFrame+0xf7
  9cef0cb4 9950d73d fe414570 00000085 090402df win32k!xxxRealDefWindowProc+0x88b
  9cef0ccc 994e673d fe414570 00000085 090402df win32k!xxxWrapRealDefWindowProc+0x2b
  9cef0ce8 9950d6f4 fe414570 00000085 090402df win32k!NtUserfnNCDESTROY+0x27
  9cef0d20 81c9897a 000200ba 00000085 090402df win32k!NtUserMessageCall+0xc6
  9cef0d20 77089a94 000200ba 00000085 090402df nt!KiFastCallEntry+0x12a
  WARNING: Frame IP not in any known module. Following frames may be wrong.
  0012d7cc 00000000 00000000 00000000 00000000 0x77089a94
  
  
  STACK_COMMAND:  kb
  
  FOLLOWUP_IP: 
  RDPDD!OE2_TableEncodeOrderFields+11e
  9976011a 8b4518          mov     eax,dword ptr [ebp+18h]
  
  SYMBOL_STACK_INDEX:  4
  
  SYMBOL_NAME:  RDPDD!OE2_TableEncodeOrderFields+11e
  
  FOLLOWUP_NAME:  MachineOwner
  
  MODULE_NAME: RDPDD
  
  IMAGE_NAME:  RDPDD.dll
  
  DEBUG_FLR_IMAGE_TIMESTAMP:  4791923e
  
  FAILURE_BUCKET_ID:  0x8E_RDPDD!OE2_TableEncodeOrderFields+11e
  
  BUCKET_ID:  0x8E_RDPDD!OE2_TableEncodeOrderFields+11e
  
  Followup: MachineOwner
  ---------
  
  ------------------------------------------------------------------------------------------
  
  
  Microsoft (R) Windows Debugger Version 6.10.0003.233 X86
  Copyright (c) Microsoft Corporation. All rights reserved.
  
  
  Loading Dump File [C:\Users\esinnard\Desktop\Windows Dumps\1-29-09\MEMORY.DMP]
  Kernel Summary Dump File: Only kernel address space is available
  
  Symbol search path is: SRV*C:\ProgramData\Symbols*http://msdl.microsoft.com/download/symbols
  Executable search path is: 
  Windows Server 2008/Windows Vista SP1 Kernel Version 6001 (Service Pack 1) MP (8 procs) Free x86 compatible
  Product: Server, suite: TerminalServer
  Built by: 6001.18145.x86fre.vistasp1_gdr.080917-1612
  Machine Name:
  Kernel base = 0x81c41000 PsLoadedModuleList = 0x81d4e930
  Debug session time: Thu Jan 29 12:49:43.870 2009 (GMT-6)
  System Uptime: 0 days 11:18:08.929
  Loading Kernel Symbols
  ...............................................................
  ................................................................
  ..............
  Loading User Symbols
  PEB is paged out (Peb.Ldr = 7ffd400c).  Type ".hh dbgerr001" for details
  Loading unloaded module list
  .....
  *******************************************************************************
  *                                                                             *
  *                        Bugcheck Analysis                                    *
  *                                                                             *
  *******************************************************************************
  
  Use !analyze -v to get detailed debugging information.
  
  BugCheck 8E, {c0000005, 81c88043, 9cef0840, 0}
  
  Page bd1f2 not present in the dump file. Type ".hh dbgerr004" for details
  Page bc9c3 not present in the dump file. Type ".hh dbgerr004" for details
  PEB is paged out (Peb.Ldr = 7ffd400c).  Type ".hh dbgerr001" for details
  PEB is paged out (Peb.Ldr = 7ffd400c).  Type ".hh dbgerr001" for details
  Probably caused by : RDPDD.dll ( RDPDD!OE2_TableEncodeOrderFields+11e )
  
  Followup: MachineOwner
  ---------
  
  7: kd&gt; !analyze -v
  *******************************************************************************
  *                                                                             *
  *                        Bugcheck Analysis                                    *
  *                                                                             *
  *******************************************************************************
  
  KERNEL_MODE_EXCEPTION_NOT_HANDLED (8e)
  This is a very common bugcheck.  Usually the exception address pinpoints
  the driver/function that caused the problem.  Always note this address
  as well as the link date of the driver/image that contains this address.
  Some common problems are exception code 0x80000003.  This means a hard
  coded breakpoint or assertion was hit, but this system was booted
  /NODEBUG.  This is not supposed to happen as developers should never have
  hardcoded breakpoints in retail code, but ...
  If this happens, make sure a debugger gets connected, and the
  system is booted /DEBUG.  This will let us see why this breakpoint is
  happening.
  Arguments:
  Arg1: c0000005, The exception code that was not handled
  Arg2: 81c88043, The address that the exception occurred at
  Arg3: 9cef0840, Trap Frame
  Arg4: 00000000
  
  Debugging Details:
  ------------------
  
  PEB is paged out (Peb.Ldr = 7ffd400c).  Type ".hh dbgerr001" for details
  PEB is paged out (Peb.Ldr = 7ffd400c).  Type ".hh dbgerr001" for details
  
  EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.
  
  FAULTING_IP: 
  nt!RtlInitUnicodeString+1b
  81c88043 f266af          repne scas word ptr es:[edi]
  
  TRAP_FRAME:  9cef0840 -- (.trap 0xffffffff9cef0840)
  ErrCode = 00000000
  eax=00000000 ebx=fe414fd8 ecx=ffffffec edx=9cef0914 esi=fe40fcf0 edi=fe415000
  eip=81c88043 esp=9cef08b4 ebp=9cef0924 iopl=0         nv up ei pl zr na pe nc
  cs=0008  ss=0010  ds=0023  es=0023  fs=0030  gs=0000             efl=00010246
  nt!RtlInitUnicodeString+0x1b:
  81c88043 f266af          repne scas word ptr es:[edi]
  Resetting default scope
  
  DEFAULT_BUCKET_ID:  VISTA_DRIVER_FAULT
  
  BUGCHECK_STR:  0x8E
  
  PROCESS_NAME:  WINOAC.EXE
  
  CURRENT_IRQL:  0
  
  LAST_CONTROL_TRANSFER:  from 81c72fbe to 81cfc759
  
  STACK_TEXT:  
  9cef0400 81c72fbe 0000008e c0000005 81c88043 nt!KeBugCheckEx+0x1e
  9cef07d0 81c9953a 9cef07ec 00000000 9cef0840 nt!KiDispatchException+0x1a9
  9cef0838 81c994ee 9cef0924 81c88043 badb0d00 nt!CommonDispatchException+0x4a
  9cef085c 9976011a 99771680 997708e8 00000000 nt!Kei386EoiHelper+0x186
  9cef0924 9959efab 5d0102bb 00000006 00000002 RDPDD!OE2_TableEncodeOrderFields+0x11e
  9cef0a0c 995aeaf8 5d0102bb 00000006 00000002 win32k!xxxRealDrawMenuItem+0x80b
  9cef0abc 9958455b 5d0102bb 0110007e 9cef0b04 win32k!xxxDrawState+0x1c9
  9cef0b2c 995853e1 5d0102bb fe40fc78 00c8d0d4 win32k!xxxDrawMenuItem+0x3f8
  9cef0b98 9959f511 5d0102bb 00000000 fe414570 win32k!xxxMenuDraw+0x1f2
  9cef0bf0 994ed1d6 00000017 5d0102bb 00000004 win32k!xxxMenuBarDraw+0x1bf
  9cef0c38 9950c0f5 fe414570 5d0102bb 00000001 win32k!xxxDrawWindowFrame+0xf7
  9cef0cb4 9950d73d fe414570 00000085 090402df win32k!xxxRealDefWindowProc+0x88b
  9cef0ccc 994e673d fe414570 00000085 090402df win32k!xxxWrapRealDefWindowProc+0x2b
  9cef0ce8 9950d6f4 fe414570 00000085 090402df win32k!NtUserfnNCDESTROY+0x27
  9cef0d20 81c9897a 000200ba 00000085 090402df win32k!NtUserMessageCall+0xc6
  9cef0d20 77089a94 000200ba 00000085 090402df nt!KiFastCallEntry+0x12a
  WARNING: Frame IP not in any known module. Following frames may be wrong.
  0012d7cc 00000000 00000000 00000000 00000000 0x77089a94
  
  
  STACK_COMMAND:  kb
  
  FOLLOWUP_IP: 
  RDPDD!OE2_TableEncodeOrderFields+11e
  9976011a 8b4518          mov     eax,dword ptr [ebp+18h]
  
  SYMBOL_STACK_INDEX:  4
  
  SYMBOL_NAME:  RDPDD!OE2_TableEncodeOrderFields+11e
  
  FOLLOWUP_NAME:  MachineOwner
  
  MODULE_NAME: RDPDD
  
  IMAGE_NAME:  RDPDD.dll
  
  DEBUG_FLR_IMAGE_TIMESTAMP:  4791923e
  
  FAILURE_BUCKET_ID:  0x8E_RDPDD!OE2_TableEncodeOrderFields+11e
  
  BUCKET_ID:  0x8E_RDPDD!OE2_TableEncodeOrderFields+11e
  
  Followup: MachineOwner
  ---------
  
  ------------------------------------------------------------------------------------------
  
  
  Microsoft (R) Windows Debugger Version 6.10.0003.233 X86
  Copyright (c) Microsoft Corporation. All rights reserved.
  
  
  Loading Dump File [C:\Users\esinnard\Desktop\Windows Dumps\2-3-09-2\MEMORY.DMP]
  Kernel Summary Dump File: Only kernel address space is available
  
  Symbol search path is: SRV*C:\ProgramData\Symbols*http://msdl.microsoft.com/download/symbols
  Executable search path is: 
  Windows Server 2008/Windows Vista SP1 Kernel Version 6001 (Service Pack 1) MP (8 procs) Free x86 compatible
  Product: Server, suite: TerminalServer
  Built by: 6001.18145.x86fre.vistasp1_gdr.080917-1612
  Machine Name:
  Kernel base = 0x81c13000 PsLoadedModuleList = 0x81d20930
  Debug session time: Tue Feb  3 14:20:03.117 2009 (GMT-6)
  System Uptime: 0 days 2:00:33.869
  Loading Kernel Symbols
  ...............................................................
  ................................................................
  .............
  Loading User Symbols
  PEB is paged out (Peb.Ldr = 7ffdc00c).  Type ".hh dbgerr001" for details
  Loading unloaded module list
  .....
  *******************************************************************************
  *                                                                             *
  *                        Bugcheck Analysis                                    *
  *                                                                             *
  *******************************************************************************
  
  Use !analyze -v to get detailed debugging information.
  
  BugCheck 8E, {c0000005, 81c5a043, d60a5840, 0}
  
  Page bce51 not present in the dump file. Type ".hh dbgerr004" for details
  Page bce22 not present in the dump file. Type ".hh dbgerr004" for details
  Page bb16b not present in the dump file. Type ".hh dbgerr004" for details
  Page bce5a not present in the dump file. Type ".hh dbgerr004" for details
  Page bce5a not present in the dump file. Type ".hh dbgerr004" for details
  Page bce5a not present in the dump file. Type ".hh dbgerr004" for details
  PEB is paged out (Peb.Ldr = 7ffdc00c).  Type ".hh dbgerr001" for details
  PEB is paged out (Peb.Ldr = 7ffdc00c).  Type ".hh dbgerr001" for details
  Probably caused by : win32k.sys ( win32k!OffBitBlt+97 )
  
  Followup: MachineOwner
  ---------
  
  0: kd&gt; !analyze -v
  *******************************************************************************
  *                                                                             *
  *                        Bugcheck Analysis                                    *
  *                                                                             *
  *******************************************************************************
  
  KERNEL_MODE_EXCEPTION_NOT_HANDLED (8e)
  This is a very common bugcheck.  Usually the exception address pinpoints
  the driver/function that caused the problem.  Always note this address
  as well as the link date of the driver/image that contains this address.
  Some common problems are exception code 0x80000003.  This means a hard
  coded breakpoint or assertion was hit, but this system was booted
  /NODEBUG.  This is not supposed to happen as developers should never have
  hardcoded breakpoints in retail code, but ...
  If this happens, make sure a debugger gets connected, and the
  system is booted /DEBUG.  This will let us see why this breakpoint is
  happening.
  Arguments:
  Arg1: c0000005, The exception code that was not handled
  Arg2: 81c5a043, The address that the exception occurred at
  Arg3: d60a5840, Trap Frame
  Arg4: 00000000
  
  Debugging Details:
  ------------------
  
  Page bb16b not present in the dump file. Type ".hh dbgerr004" for details
  Page bce5a not present in the dump file. Type ".hh dbgerr004" for details
  Page bce5a not present in the dump file. Type ".hh dbgerr004" for details
  Page bce5a not present in the dump file. Type ".hh dbgerr004" for details
  PEB is paged out (Peb.Ldr = 7ffdc00c).  Type ".hh dbgerr001" for details
  PEB is paged out (Peb.Ldr = 7ffdc00c).  Type ".hh dbgerr001" for details
  
  EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.
  
  FAULTING_IP: 
  nt!RtlInitUnicodeString+1b
  81c5a043 f266af          repne scas word ptr es:[edi]
  
  TRAP_FRAME:  d60a5840 -- (.trap 0xffffffffd60a5840)
  ErrCode = 00000000
  eax=00000000 ebx=fe41afd8 ecx=ffffffec edx=d60a5914 esi=fe40f5e0 edi=fe41b000
  eip=81c5a043 esp=d60a58b4 ebp=d60a5924 iopl=0         nv up ei pl zr na pe nc
  cs=0008  ss=0010  ds=0023  es=0023  fs=0030  gs=0000             efl=00010246
  nt!RtlInitUnicodeString+0x1b:
  81c5a043 f266af          repne scas word ptr es:[edi]
  Resetting default scope
  
  DEFAULT_BUCKET_ID:  VISTA_DRIVER_FAULT
  
  BUGCHECK_STR:  0x8E
  
  PROCESS_NAME:  WINOAC.EXE
  
  CURRENT_IRQL:  0
  
  LAST_CONTROL_TRANSFER:  from 81c44fbe to 81cce759
  
  STACK_TEXT:  
  d60a5400 81c44fbe 0000008e c0000005 81c5a043 nt!KeBugCheckEx+0x1e
  d60a57d0 81c6b53a d60a57ec 00000000 d60a5840 nt!KiDispatchException+0x1a9
  d60a5838 81c6b4ee d60a5924 81c5a043 badb0d00 nt!CommonDispatchException+0x4a
  d60a585c 999e2242 ff888010 00000000 00000000 nt!Kei386EoiHelper+0x186
  d60a5924 999befab 1401009b 00000006 00000002 win32k!OffBitBlt+0x97
  d60a5a0c 999ceaf8 1401009b 00000006 00000002 win32k!xxxRealDrawMenuItem+0x80b
  d60a5abc 999a455b 1401009b 0110007e d60a5b04 win32k!xxxDrawState+0x1c9
  d60a5b2c 999a53e1 1401009b fe40d168 00c8d0d4 win32k!xxxDrawMenuItem+0x3f8
  d60a5b98 999bf511 1401009b 00000000 fe418398 win32k!xxxMenuDraw+0x1f2
  d60a5bf0 9990d1d6 00000017 1401009b 00000004 win32k!xxxMenuBarDraw+0x1bf
  d60a5c38 9992c0f5 fe418398 1401009b 00000001 win32k!xxxDrawWindowFrame+0xf7
  d60a5cb4 9992d73d fe418398 00000085 0904035f win32k!xxxRealDefWindowProc+0x88b
  d60a5ccc 9990673d fe418398 00000085 0904035f win32k!xxxWrapRealDefWindowProc+0x2b
  d60a5ce8 9992d6f4 fe418398 00000085 0904035f win32k!NtUserfnNCDESTROY+0x27
  d60a5d20 81c6a97a 0003001c 00000085 0904035f win32k!NtUserMessageCall+0xc6
  d60a5d20 77049a94 0003001c 00000085 0904035f nt!KiFastCallEntry+0x12a
  WARNING: Frame IP not in any known module. Following frames may be wrong.
  0012d7cc 00000000 00000000 00000000 00000000 0x77049a94
  
  
  STACK_COMMAND:  kb
  
  FOLLOWUP_IP: 
  win32k!OffBitBlt+97
  999e2242 8b4d20          mov     ecx,dword ptr [ebp+20h]
  
  SYMBOL_STACK_INDEX:  4
  
  SYMBOL_NAME:  win32k!OffBitBlt+97
  
  FOLLOWUP_NAME:  MachineOwner
  
  MODULE_NAME: win32k
  
  IMAGE_NAME:  win32k.sys
  
  DEBUG_FLR_IMAGE_TIMESTAMP:  48d1b9ef
  
  FAILURE_BUCKET_ID:  0x8E_win32k!OffBitBlt+97
  
  BUCKET_ID:  0x8E_win32k!OffBitBlt+97
  
  Followup: MachineOwner
  ---------
  </code></pre>