title: IIS Application Pool randomly recycles
body: <p>Running multiple .netcore applications on our servers.
  We see random recycles on the application pools with the message</p>
  <blockquote>
  <p>Sent shutdown HTTP message to process 'PID_NUMBER_HERE' and received
  http status '202'.</p>
  </blockquote>
  <p>Sometimes there is an accompanying</p>
  <blockquote>
  <p>&quot;App_offline file 'app_offline.htm' was detected.&quot;</p>
  </blockquote>
  <p>message just before the shutdown message but not always.</p>
  <p>There are both .netcore 2.1 and 5.0 applications in the server. The 5.0 apps have their own application pools each and the rest were in a common pool.</p>
  <ol>
  <li>Disabled every &quot;recycling&quot; option in IIS (memory/time/fixed
  interval)</li>
  <li>Made sure there is no app_offline.htm anywhere in the
  server. Also set up a file monitor to check and no app_offline.htm
  is ever created anywhere.</li>
  <li>All applications were moved to seperate application pools one-to-one.</li>
  <li>There is no memory or cpu bottleneck. The recycles happen at 50% memory 10% cpu or 30% memory 20% cpu randomly</li>
  </ol>
  <p>Each application pool still randomly is recycled automatically.</p>
  <p>Any ideas as to why IIS recyles the application pools randomly or any pointers to where to look for a clue is welcome.</p>
  <p>EDIT:</p>
  <ol>
  <li>There arent any unusaly HTTPERR logs at the time of restarts</li>
  <li>applicationhost.config + web.config manually checked. no recycle
  option set</li>
  <li>There arent any correspondonce between website traffic
  and the restarts. They seem to happen randomly</li>
  <li>The restarts are sporadic. No set obvious frequency</li>
  <li>Enabled core application debugLevel logging: FILE,TRACE</li>
  </ol>
  <p>Below are the messages during recycles</p>
  <pre><code>[aspnetcorev2.dll] ASPNET_CORE_GLOBAL_MODULE::OnGlobalConfigurationChange 'MACHINE/WEBROOT/APPHOST/XXX.com/XXXCoreApplication.WEB'
  [aspnetcorev2.dll] Stopping application '/LM/W3SVC/2/ROOT/XXXCoreApplication.Web'
  [aspnetcorev2_inprocess.dll] Stopping CLR
  [aspnetcorev2_inprocess.dll] Drained all requests, notifying managed.
  [aspnetcorev2_inprocess.dll] Waiting for 0 requests to drain
  [aspnetcorev2_inprocess.dll] Starting shutdown sequence 0
  [aspnetcorev2_inprocess.dll] Removing Request 0
  [aspnetcorev2_inprocess.dll] Adding request. Total Request Count 1
  [aspnetcorev2.dll] Creating handler application
  [aspnetcorev2_inprocess.dll] Adding request. Total Request Count 2
  [aspnetcorev2.dll] ASPNET_CORE_GLOBAL_MODULE::OnGlobalConfigurationChange 'MACHINE/WEBROOT/APPHOST/XXX.com/XXXCoreApplication.WEB'
  [aspnetcorev2_inprocess.dll] Removing Request 1
  [aspnetcorev2.dll] Stopping application '/LM/W3SVC/2/ROOT/XXXCoreApplication.Web'
  [aspnetcorev2_inprocess.dll] Stopping CLR
  [aspnetcorev2_inprocess.dll] Starting shutdown sequence 0
  [aspnetcorev2_inprocess.dll] Waiting for 1 requests to drain
  [aspnetcorev2_inprocess.dll] Removing Request 0
  [aspnetcorev2_inprocess.dll] Drained all requests, notifying managed.
  [aspnetcorev2.dll] Creating handler application
  [aspnetcorev2_inprocess.dll] Managed application exited with code 0
  [aspnetcorev2_inprocess.dll] Managed application exited with code 0
  [aspnetcorev2_inprocess.dll] Clr thread wait ended: clrThreadExited: 1
  [aspnetcorev2_inprocess.dll] Clr thread wait ended: clrThreadExited: 1
  [aspnetcorev2_inprocess.dll] Event Log: 'Application 'MACHINE/WEBROOT/APPHOST/XXX.com/XXXCoreApplication.WEB' has shutdown.' 
  End Event Log Message.
  [aspnetcorev2_inprocess.dll] Event Log: 'Application 'MACHINE/WEBROOT/APPHOST/XXX.com/XXXCoreApplication.WEB' has shutdown.' 
  End Event Log Message.
  [aspnetcorev2_inprocess.dll] Canceling standard stream pipe reader
  [aspnetcorev2_inprocess.dll] Canceling standard stream pipe reader
  [aspnetcorev2_inprocess.dll] Stopping in-process worker thread
  [aspnetcorev2_inprocess.dll] Stopping in-process worker thread
  [aspnetcorev2_inprocess.dll] Stopping file watcher thread
  [aspnetcorev2_inprocess.dll] Stopping file watcher thread
  [aspnetcorev2.dll] 
  ASPNET_CORE_GLOBAL_MODULE::OnGlobalConfigurationChange 'MACHINE/WEBROOT/APPHOST/XXX.com/XXXCoreApplication.WEB'
  [aspnetcorev2.dll] ASPNET_CORE_GLOBAL_MODULE::OnGlobalConfigurationChange 'MACHINE/WEBROOT/APPHOST/XXX.com/XXXCoreApplication.WEB'
  [aspnetcorev2.dll] Stopping application '/LM/W3SVC/2/ROOT/XXXCoreApplication.Web'
  [aspnetcorev2.dll] Stopping application '/LM/W3SVC/2/ROOT/XXXCoreApplication.Web'
  [aspnetcorev2.dll] ASPNET_CORE_GLOBAL_MODULE::OnGlobalConfigurationChange 'MACHINE/WEBROOT/APPHOST/XXX.com/XXXCoreApplication.WEB'
  [aspnetcorev2.dll] ASPNET_CORE_GLOBAL_MODULE::OnGlobalConfigurationChange 'MACHINE/WEBROOT/APPHOST/XXX.com/XXXCoreApplication.WEB'
  [aspnetcorev2.dll] ASPNET_CORE_GLOBAL_MODULE::OnGlobalConfigurationChange 'MACHINE/WEBROOT/APPHOST/XXX.com/XXXCoreApplication.WEB'
  [aspnetcorev2.dll] ASPNET_CORE_GLOBAL_MODULE::OnGlobalStopListening
  [aspnetcorev2.dll] ASPNET_CORE_GLOBAL_MODULE::OnGlobalStopListening
  [aspnetcorev2.dll] ASPNET_CORE_GLOBAL_MODULE::Terminate
  [aspnetcorev2.dll] ASPNET_CORE_GLOBAL_MODULE::Terminate
  </code></pre>
  <p>It seems like a global configuration change is taking place. No idea why though</p>
  <p>EDIT:
  Another bit of info. We have multiple servers that are acessing the IIS Config as shared on a network location. So the application files are also accessed as shared on a network device.
  XXCoreApp is served from \SharedNetworkLocation\XXCoreApp
  Although we see recyles on all servers they dont happen at the same time.
  Webserver1 recyles XXCoreApp at some time Webserver2 recyles it at another time</p>
  <p>EDIT:
  Moved the application files to local folders. That prevents the recycles but is not a real solution as there are quite a few servers and managing them locally is not acceptable.</p>
  <p>That suggests there is something going on when using a shared network folder.</p>
  <p>We tried changing the application file locations from</p>
  <blockquote>
  <p>\fully-qualified-domain-name\app_files</p>
  </blockquote>
  <p>to</p>
  <blockquote>
  <p>\serverName\d$\app_files</p>
  </blockquote>
  <p>That reduced the number of recycles. There are still recycles but infrequently.</p>
  <p>Still trying to figure out what is causing the application pools to recycle on network shares.</p>