WinDBG w3wp.exe x64 hang dump WRONG_SYMBOLS

Tags:
ajax android angularjs arrays asp.net bash c# css database date dictionary eclipse eloquent function gcc git hibernate html java javascript jquery laravel laravel-4 laravel-5 linux linux-kernel mysql nginx php python r reactjs regex ruby-on-rails select shell sql sql-server sql-server-2008 string tsql twitter-bootstrap ubuntu version-control xml

WinDBG w3wp.exe x64 hang dump WRONG_SYMBOLS

I am analyzing a hanging ASP.NET MVC website problem running under a 64 bit ASP.NET v4.0 AppPool in IIS 7.5, Windows 2008 R2 64. I have taken a dump through taskmgr, and am analysing in WinDBG x64 on Windows 7 64bit.

When running !analyze -v I see the error WRONG_SYMBOLS and BUGCHECK_STR: APPLICATION_FAULT_WRONG_SYMBOLS. This is hindering my debugging as I cannot check some threads that I need to investigate for deadlocking.

Details

My symbol server is SRV*D:DOWNLOADEDSYMBOLS*http://msdl.microsoft.com/download/symbols
I have deleted this local folder and allowed all symbols to be download from m$.

I have loaded sos with .loadby sos clr

Output from !analyze -v:

*******************************************************************************
*                                                                             *
*                        Exception Analysis                                   *
*                                                                             *
*******************************************************************************

FAULTING_IP: 
+0
00000000`00000000 ??              ???

EXCEPTION_RECORD:  ffffffffffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 0000000000000000
   ExceptionCode: 80000003 (Break instruction exception)
  ExceptionFlags: 00000000
NumberParameters: 0

FAULTING_THREAD:  0000000000000544

DEFAULT_BUCKET_ID:  WRONG_SYMBOLS

PROCESS_NAME:  w3wp.exe

ERROR_CODE: (NTSTATUS) 0x80000003 - {EXCEPTION}  Breakpoint  A breakpoint has been reached.

EXCEPTION_CODE: (HRESULT) 0x80000003 (2147483651) - One or more arguments are invalid

NTGLOBALFLAG:  0

APPLICATION_VERIFIER_FLAGS:  0

APP:  w3wp.exe

MANAGED_STACK: !dumpstack -EE
OS Thread Id: 0x944 (15)
Current frame: 
Child-SP         RetAddr          Caller, Callee

PRIMARY_PROBLEM_CLASS:  WRONG_SYMBOLS

BUGCHECK_STR:  APPLICATION_FAULT_WRONG_SYMBOLS

LAST_CONTROL_TRANSFER:  from 000007fefd8910dc to 000000007712135a

STACK_TEXT:  
00000000`000afac8 000007fe`fd8910dc : 00000000`00000000 00000000`00000000 000007fe`f9921630 000007fe`fadb7f66 : ntdll!NtWaitForSingleObject+0xa
00000000`000afad0 000007fe`f99241bc : 00000000`00000000 00000000`ffaf6de0 00000000`00000000 00000000`00000128 : KERNELBASE!WaitForSingleObjectEx+0x79
00000000`000afb70 00000000`ffaf3c60 : 00000000`fffffffe 00000000`00415f90 00000000`ffaf4588 000007fe`f9920000 : w3wphost!AppHostInitialize+0x278
00000000`000afbd0 00000000`ffaf11f1 : 00000000`00000000 00000000`ffaf1351 00000000`00000000 000003fd`deed0e35 : w3wp!wmain+0x470
00000000`000afd60 00000000`76e6652d : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : w3wp!PerfStopProvider+0x19b
00000000`000afda0 00000000`770fc521 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0xd
00000000`000afdd0 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x1d

STACK_COMMAND:  ~0s; .ecxr ; kb

FOLLOWUP_IP: 
w3wphost!AppHostInitialize+278
000007fe`f99241bc f605e1b4000003  test    byte ptr [w3wphost!g_dwDebugFlags (000007fe`f992f6a4)],3

SYMBOL_STACK_INDEX:  2

SYMBOL_NAME:  w3wphost!AppHostInitialize+278

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: w3wphost

IMAGE_NAME:  w3wphost.dll

DEBUG_FLR_IMAGE_TIMESTAMP:  4ce7c9ef

FAILURE_BUCKET_ID:  WRONG_SYMBOLS_80000003_w3wphost.dll!AppHostInitialize

BUCKET_ID:  X64_APPLICATION_FAULT_WRONG_SYMBOLS_w3wphost!AppHostInitialize+278

WATSON_STAGEONE_URL:  http://watson.microsoft.com/StageOne/w3wp_exe/7_5_7601_17514/4ce7afa2/unknown/0_0_0_0/bbbbbbb4/80000003/00000000.htm?Retriage=1

Followup: MachineOwner
---------

Output from .chain:

Extension DLL search Path:
    C:Program Files (x86)Windows Kits8.0Debuggersx64WINXP;C:Program Files (x86)Windows Kits8.0Debuggersx64winext;C:Program Files (x86)Windows Kits8.0Debuggersx64winextarcade;C:Program Files (x86)Windows Kits8.0Debuggersx64pri;C:Program Files (x86)Windows Kits8.0Debuggersx64;C:Program Files (x86)Windows Kits8.0Debuggersx64winextarcade;C:Windowssystem32;C:Windows;C:WindowsSystem32Wbem;C:WindowsSystem32WindowsPowerShellv1.0;C:Program Files (x86)Microsoft SQL Server100ToolsBinn;C:Program FilesMicrosoft SQL Server100ToolsBinn;C:Program FilesMicrosoft SQL Server100DTSBinn;C:Program Files (x86)Microsoft SQL Server100ToolsBinnVSShellCommon7IDE;C:Program Files (x86)Microsoft SQL Server100DTSBinn;C:Program Files (x86)IntelServicesIPT;C:Program FilesIntelWiFibin;C:Program FilesCommon FilesIntelWirelessCommon;C:Program Files (x86)Microsoft Team Foundation Server 2012 Power Tools;C:Program Files (x86)Microsoft Team Foundation Server 2012 Power ToolsBest Practices Analyzer;C:Program Files (x86)Microsoft SQL Server110ToolsBinn;C:Program FilesMicrosoft SQL Server110ToolsBinn;C:Program FilesMicrosoft SQL Server110DTSBinn;C:Program Files (x86)Microsoft SQL Server110ToolsBinnManagementStudio;C:Program Files (x86)Microsoft SQL Server110DTSBinn;C:Program FilesMicrosoftWeb Platform Installer;C:Program Files (x86)Microsoft ASP.NETASP.NET Web Pagesv1.0;C:Program Files (x86)Windows Kits8.0Windows Performance Toolkit
Extension DLL chain:
    D:DOWNLOADEDSYMBOLSsos_AMD64_AMD64_4.0.30319.18034.dll50B5A78395e000sos_AMD64_AMD64_4.0.30319.18034.dll: image 4.0.30319.18034, API 1.0.0, built Wed Nov 28 18:45:59 2012
        [path: D:DOWNLOADEDSYMBOLSsos_AMD64_AMD64_4.0.30319.18034.dll50B5A78395e000sos_AMD64_AMD64_4.0.30319.18034.dll]
    sosex: image 4.5.0.0, API 1.0.0, built Thu Oct 04 03:57:55 2012
        [path: C:Program Files (x86)Windows Kits8.0Debuggersx64sosex.dll]
    D:DOWNLOADEDSYMBOLSsos_AMD64_AMD64_4.0.30319.18034.dll50B5A78395e000sos_AMD64_AMD64_4.0.30319.18034.dll: image 4.0.30319.18034, API 1.0.0, built Wed Nov 28 18:45:59 2012
        [path: D:DOWNLOADEDSYMBOLSsos_AMD64_AMD64_4.0.30319.18034.dll50B5A78395e000sos_AMD64_AMD64_4.0.30319.18034.dll]
    C:WindowsMicrosoft.NETFramework64v4.0.30319sos: image 4.0.30319.18034, API 1.0.0, built Wed Nov 28 18:45:59 2012
        [path: C:WindowsMicrosoft.NETFramework64v4.0.30319sos.dll]
    dbghelp: image 6.2.9200.20512, API 6.2.6, built Fri Sep 07 17:45:49 2012
        [path: C:Program Files (x86)Windows Kits8.0Debuggersx64dbghelp.dll]
    ext: image 6.2.9200.20522, API 1.0.0, built Fri Sep 21 20:17:05 2012
        [path: C:Program Files (x86)Windows Kits8.0Debuggersx64winextext.dll]
    exts: image 6.2.9200.16384, API 1.0.0, built Thu Jul 26 14:15:20 2012
        [path: C:Program Files (x86)Windows Kits8.0Debuggersx64WINXPexts.dll]
    uext: image 6.2.9200.16384, API 1.0.0, built Thu Jul 26 14:15:09 2012
        [path: C:Program Files (x86)Windows Kits8.0Debuggersx64winextuext.dll]
    ntsdexts: image 6.2.9200.16384, API 1.0.0, built Thu Jul 26 14:16:01 2012
        [path: C:Program Files (x86)Windows Kits8.0Debuggersx64WINXPntsdexts.dll]

Output from .reload with !sym noisy

DBGHELP: ntdll - public symbols  
      d:downloadedsymbolsntdll.pdb15EB43E23B12409C84E3CC7635BAF5A32ntdll.pdb
..............................................................
DBGHELP: KERNELBASE - public symbols 
      d:downloadedsymbolskernelbase.pdb91C72371DD43448192B7B46F5ED10AA02kernelbase.pdb

Deadlock Analysis

Thread List

Output from !threads:

    ID  OSID    ThreadOBJ           State   GC Mode     GC Alloc Context                    Domain              Lock Count  Apt Exception
7   1   7a8     0000000001f92a50    28220   Preemptive  0000000000000000:0000000000000000   000000000119ee70    0           Ukn 
13  2   1764    0000000001fa9fb0    2b220   Preemptive  0000000102D5DCC8:0000000102D5F7E8   000000000119ee70    0           MTA (Finalizer) 
15  3   944     0000000001ffa010    102a220 Preemptive  0000000000000000:0000000000000000   000000000119ee70    0           MTA (Threadpool Worker) 
16  4   bb0     0000000002006ec0    21220   Preemptive  0000000000000000:0000000000000000   000000000119ee70    0           Ukn
 6  15  1018    00000000086704d0    20220 Preemptive  0000000000000000:0000000000000000 000000000119ee70 0     Ukn 
35  16  12e4    00000000098fc820  202b220 Preemptive  000000010400E5A0:00000001040102B0 0000000002005e90 1     MTA

….etc

Output from !dlk:

*DEADLOCK DETECTED*
CLR thread 0x15 holds the lock on SyncBlock 0000000009913cf8 OBJ:00000000ffe31898
  [System.Collections.Generic.Dictionary``2[[System.String, mscorlib],[System.Resources.ResourceLocator, mscorlib]]]
...and is waiting for the lock on SyncBlock 0000000009913ca8 OBJ:00000000ffe318e8[System.Resources.ResourceReader]
CLR thread 0x16 holds the lock on SyncBlock 0000000009913ca8 OBJ:00000000ffe318e8[System.Resources.ResourceReader]
...and is waiting for the lock on SyncBlock 0000000009913cf8 OBJ:00000000ffe31898
  [System.Collections.Generic.Dictionary``2[[System.String, mscorlib],[System.Resources.ResourceLocator, mscorlib]]]
CLR Thread 0x15 is waiting at System.Resources.ResourceReader.AllocateStringForNameIndex(Int32, Int32 ByRef)(+0x17 IL,+0xc9 Native)
CLR Thread 0x16 is waiting at System.Resources.RuntimeResourceSet.GetObject(System.String, Boolean, Boolean)(+0xee IL,+0x375 Native)

Analysis of thread 15

Output from ~6s:

ntdll!ZwDelayExecution+0xa:
00000000`771213aa c3              ret

.. and then output from !clrstack:

OS Thread Id: 0x1018 (6)
        Child SP               IP Call Site
GetFrameContext failed: 1
0000000000000000 0000000000000000 <unknown>

Analysis of thread 16

Output from ~35s:

ntdll!NtWaitForMultipleObjects+0xa:
00000000`771218ca c3              ret

Output from !clrstack:

OS Thread Id: 0x12e4 (35)
    Child SP               IP Call Site
000000000cc9e188 00000000771218ca [HelperMethodFrame_1OBJ: 000000000cc9e188]     System.Threading.WaitHandle.WaitMultiple(System.Threading.WaitHandle[], Int32, Boolean, Boolean)
000000000cc9e2c0 000007fef65e562c     System.Threading.WaitHandle.WaitAny(System.Threading.WaitHandle[], Int32, Boolean)
000000000cc9e320 000007fef5670d06 System.Net.TimerThread.ThreadProc()
000000000cc9e3f0 000007fef65a0a05     System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext,     System.Threading.ContextCallback, System.Object, Boolean)
000000000cc9e550 000007fef65a0769     System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext,     System.Threading.ContextCallback, System.Object, Boolean)
000000000cc9e580 000007fef65a0727     System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext,     System.Threading.ContextCallback, System.Object)
000000000cc9e5d0 000007fef65b3e81 System.Threading.ThreadHelper.ThreadStart()
000000000cc9e8e8 000007fef9ec07f3 [GCFrame: 000000000cc9e8e8] 
000000000cc9ec18 000007fef9ec07f3 [DebuggerU2MCatchHandlerFrame: 000000000cc9ec18] 
000000000cc9edf8 000007fef9ec07f3 [ContextTransitionFrame: 000000000cc9edf8] 
000000000cc9efe8 000007fef9ec07f3 [DebuggerU2MCatchHandlerFrame: 000000000cc9efe8]

Ignore that error. !analyze -v is only useful for crash dumps (well there are some parameters I believe you can pass to it for things other than crash dumps). What it is reporting is that the reason for the “crash” is that a debug point was hit. Which is true, because when you dump a process, it injects a thread in to the process and causes a break point to be hit, and then dumps the memory for the process. So what the analyze is telling you is that the reason for the “crash” is that a break point was hit. It can be safely ignored. You should be able to see all your threads when you go to look at them. Everything else should work normally.

Recent Activity

Related Posts

.
.
.
.