One of application pools we have in our environment would frequently crash with following entry being written to Application EventLog (event id 1026).

Based on stack trace it appears our Company custom assembly was trying to write to EventLog but was unable to find a source registered and crashed during enumerating sources for “Security” eventlog since it does not have permission to enumerate those under ApplicationPool identity.

This is not a mister on it’s own and expected, what is not expected or known which source it was looking for which was missing.

Application: w3wp.exe

Framework Version: v4.0.30319

Description: The process was terminated due to an unhandled exception.

Exception Info: System.Security.SecurityException

Stack:

   at System.Diagnostics.EventLog.FindSourceRegistration(System.String, System.String, Boolean, Boolean)

   at System.Diagnostics.EventLog.SourceExists(System.String, System.String, Boolean)

   at System.Diagnostics.EventLogInternal.VerifyAndCreateSource(System.String, System.String)

   at System.Diagnostics.EventLogInternal.WriteEntry(System.String, System.Diagnostics.EventLogEntryType, Int32, Int16, Byte[])

   at System.Diagnostics.EventLog.WriteEntry(System.String, System.Diagnostics.EventLogEntryType)

   at EventLogger.WriteEventLog(System.String, System.String)

   at Company.Logging.Client.CompanyLogWriter.GetLogValuesAndAddIntoDb()

   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)

   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)

   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)

   at System.Threading.ThreadHelper.ThreadStart()

Steps to troubleshoot this and similar issues are below

  1. You need to generate memory dump to analyse for this you would have to create certain registry entries and reboot machine to activate those. Entries are documented in this article https://msdn.microsoft.com/en-us/library/windows/desktop/bb787181%28v=vs.85%29.aspx?f=255&MSPPError=-2147217396

In our company we push those through Group Policy Preferences to all machines, so if you set it up right it’s essentially 0 configuration turnkey solution

  1. After those entries created reboot machine and wait for issue to occur. Once it’s happened your crashdump folder will be populated with full memory dump of the process.
  2. Open crashdump in WinDbg and load sos and issue !clrstack to command which will shall display the same stack you had in EventLog entry, just this time with memory addresses.

0:040> .loadby sos clr

0:040> !clrstack

OS Thread Id: 0xa40 (40)

        Child SP               IP Call Site

000000000efbbcf8 00000000778bc3ca [HelperMethodFrame: 000000000efbbcf8]

000000000efbbde0 000007fef747889a *** WARNING: Unable to verify checksum for System.ni.dll

System.Diagnostics.EventLog.FindSourceRegistration(System.String, System.String, Boolean, Boolean)

000000000efbbec0 000007fef74775fb System.Diagnostics.EventLog.SourceExists(System.String, System.String, Boolean)

000000000efbbf40 000007fef747b5ac System.Diagnostics.EventLogInternal.VerifyAndCreateSource(System.String, System.String)

000000000efbbfd0 000007fef747dc04 System.Diagnostics.EventLogInternal.WriteEntry(System.String, System.Diagnostics.EventLogEntryType, Int32, Int16, Byte[])

000000000efbc080 000007fef7477ebe System.Diagnostics.EventLog.WriteEntry(System.String, System.Diagnostics.EventLogEntryType)

000000000efbc0c0 000007fe99bdd523 *** ERROR: Module load completed but symbols could not be loaded for Company.Logging.dll

EventLogger.WriteEventLog(System.String, System.String)

000000000efbc130 000007fe99bd943a Company.Logging.Client.CompanyLogWriter.GetLogValuesAndAddIntoDb()

000000000efbe398 000007fef90c7b15 [HelperMethodFrame: 000000000efbe398]

000000000efbe480 000007fef8ac2005 *** WARNING: Unable to verify checksum for mscorlib.ni.dll

System.Threading.Tasks.Task.Wait(Int32, System.Threading.CancellationToken)

000000000efbe4c0 000007fef7ce05a1 System.Threading.Tasks.Task.Wait()

000000000efbe4f0 000007fe99bd92a6 Company.Logging.Client.CompanyLogWriter.GetLogValuesAndAddIntoDb()

000000000efbe5e0 000007fef7d8d0b5 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)

000000000efbe740 000007fef7d8ce19 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)

000000000efbe770 000007fef7d8cdd7 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)

000000000efbe7c0 000007fef7d00301 System.Threading.ThreadHelper.ThreadStart()

000000000efbead8 000007fef8f69e03 [GCFrame: 000000000efbead8]

000000000efbee08 000007fef8f69e03 [DebuggerU2MCatchHandlerFrame: 000000000efbee08]

000000000efbefe8 000007fef8f69e03 [ContextTransitionFrame: 000000000efbefe8]

000000000efbf1d8 000007fef8f69e03 [DebuggerU2MCatchHandlerFrame: 000000000efbf1d8]

  1. Issue !clrstack -p which will list parameters passed between methods

0:040> !clrstack -p

The version of SOS does not match the version of CLR you are debugging.  Please

load the matching version of SOS for the version of CLR you are debugging.

CLR Version: 4.0.30319.18444

SOS Version: 4.6.96.0

OS Thread Id: 0xa40 (40)

        Child SP               IP Call Site

000000000efbbcf8 00000000778bc3ca [HelperMethodFrame: 000000000efbbcf8]

000000000efbbde0 000007fef747889a System.Diagnostics.EventLog.FindSourceRegistration(System.String, System.String, Boolean, Boolean)

    PARAMETERS:

        source = <no data>

        machineName = <no data>

        readOnly = <no data>

        wantToCreate = <no data>

000000000efbbec0 000007fef74775fb System.Diagnostics.EventLog.SourceExists(System.String, System.String, Boolean)

    PARAMETERS:

        source = <no data>

        machineName = <no data>

        wantToCreate = <no data>

000000000efbbf40 000007fef747b5ac System.Diagnostics.EventLogInternal.VerifyAndCreateSource(System.String, System.String)

    PARAMETERS:

        this (0x000000000efbbfd0) = 0x00000001ff89bdd0

        sourceName = <no data>

        currentMachineName = <no data>

000000000efbbfd0 000007fef747dc04 System.Diagnostics.EventLogInternal.WriteEntry(System.String, System.Diagnostics.EventLogEntryType, Int32, Int16, Byte[])

    PARAMETERS:

        this = <no data>

        message = <no data>

        type = <no data>

        eventID = <no data>

        category (0x000000000efbc0a0) = 0x0000000000000000

        rawData (0x000000000efbc0a8) = 0x0000000000000000

000000000efbc080 000007fef7477ebe System.Diagnostics.EventLog.WriteEntry(System.String, System.Diagnostics.EventLogEntryType)

    PARAMETERS:

        this = <no data>

        message = <no data>

        type = <no data>

000000000efbc0c0 000007fe99bdd523 EventLogger.WriteEventLog(System.String, System.String)

    PARAMETERS:

        message (0x000000000efbc130) = 0x00000001ff89a608

        source (0x000000000efbc138) = 0x00000001ff848f38

000000000efbc130 000007fe99bd943a Company.Logging.Client.CompanyLogWriter.GetLogValuesAndAddIntoDb()

000000000efbe398 000007fef90c7b15 [HelperMethodFrame: 000000000efbe398]

000000000efbe480 000007fef8ac2005 System.Threading.Tasks.Task.Wait(Int32, System.Threading.CancellationToken)

        5. Frame we are interested in is colored in red above which has pointers to parameters which will identify which source it was looking for. We can dump object from that address with command below and find what the source is.

0:040> !DumpObj /d 00000001ff848f38

The version of SOS does not match the version of CLR you are debugging.  Please

load the matching version of SOS for the version of CLR you are debugging.

CLR Version: 4.0.30319.18444

SOS Version: 4.6.96.0

Name:        System.String

MethodTable: 000007fef7f06508

EEClass:     000007fef7823750

Size:        64(0x40) bytes

File:        C:\Windows\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll

String:      Company Cloud Logger

Fields:

              MT    Field   Offset                 Type VT     Attr            Value Name

000007fef7f092b8  40000aa        8         System.Int32  1 instance               19 m_stringLength

000007fef7f077f0  40000ab        c          System.Char  1 instance               4d m_firstChar

000007fef7f06508  40000ac       18        System.String  0   shared           static Empty

                                 >> Domain:Value  0000000001e1be20:NotInit  0000000001ec95a0:NotInit  <<

Mystery solved. During rollout source with the Name “Company Cloud Logger” was not created and hence the crash in worker process.