title: Django background thread exiting unexpectedly when running in Azure web service
body: <p>First post to Stack Overflow and not (yet) that experienced with Python/Django so be kind :)</p>
  
  <p>I am working on a Django web app which is exhibiting some odd behavior when deployed to an Azure web service container.  The application includes a page from which you can upload a CSV file containing records to be entered into the database which backs the app.</p>
  
  <p>The view which handles the file submission calls a function which in turn launches a background worker thread (a threading.Thread object) to process the CSV file and returns a simple True/False status to the calling view to indicate whether the file was accepted and the thread was launched successfully or not.</p>
  
  <p>The same application page from which you upload the CSV file also has a JQuery mechanism with timer to retrieve the thread status every 2 seconds.  This is done with a simple AJAX query that is handled by a Django view which returns the text status in JSON format.  The status is contained in a Python dictionary which contains key-value pairs where the key is the background thread name and the value is the status as a text string.  The status is updated by the worker thread and simply read and returned by the AJAX handler.</p>
  
  <p>The background thread does a few things:</p>
  
  <ul>
  <li><p>Writes a running log to the local file system which contains one entry per CSV record processed plus any additional errors encountered (exceptions etc.)</p></li>
  <li><p>Copies image files from an Azure storage account to local application storage, one per CSV record</p></li>
  <li><p>Makes several accesses to the Azure SQL database backing the application.  These access are via the Django models and forms that are implemented in the application.  Those classes are used to obtain the primary keys of various entities and ultimately create a new object for each CSV record which is then saved to the database using the usual Django form.save() call.</p></li>
  </ul>
  
  <p>The Problems:</p>
  
  <ol>
  <li><p>The background worker thread is arbitrarily exiting or being killed with absolutely no indication of any errors up to the point that it stops.  I've added a ton of logging (to the same log file mentioned above) to show the execution path to almost line-by-line detail.  The log stops at different spots every time I submit a CSV file (using the same file every time).  It also seems to have no relation to how many records have been processed or how long the thread has been running.  I've seen the thread survive for anything from perhaps 30 seconds to over 5 minutes but always FAR short of the ~2 hours it should take to fully process the CSV file.</p></li>
  <li><p>The thread status retrieved every 2 seconds by the AJAX mechanism described above is seeing the dictionary holding the status as empty about half the time (??!!)  Normally the dictionary will be empty when no background thread is present and will have 1 entry per thread when threads are present.  Even before the thread inexplicably terminates, the main application appears to intermittently see the dictionary (of which there should be only one global instance) as empty and then containing data (and then empty and then .....)</p></li>
  <li><p>Likely directly related to #2, the main application actually seems to be completely "losing sight" of the background thread.  I have a simple view attached to the URL .../threads which returns a simple text list of all active threads, as returned by the threading.enumerate() function.  Repeatedly hitting that URL will show the background thread present and then not present and then present again.  The process ID of the background thread, when it appears in the list, is constant but surprisingly to me the main thread process ID which also shows up in the list is NOT constant.  The main thread seems to be changing ID (or re-spawning or something) every few seconds.  I do not know if that is just part of Django doing its thing but seems odd.</p></li>
  </ol>
  
  <p>Note that running locally within Visual Studio, using the built-in Django server and connecting to a local SQL Server instance, I see absolutely none of these issues.  They only appear when running in the Azure container.</p>
  
  <p>Unfortunately I cannot really post any code because there are some intellectual property concerns attached to this application, but I can try to answer any questions anyone may have.</p>
  
  <p>Tech Details:</p>
  
  <ul>
  <li><p>Python V2.7</p></li>
  <li><p>Django V1.8.5</p></li>
  <li><p>Azure App Service, Standard: 4 Large (not shared)</p></li>
  <li><p>Developed using Visual Studio Enterprise 2015</p></li>
  </ul>
  
  <p>Any insights into this issue are greatly appreciated.</p>
  
  <p>UPDATE:
  Through some additional reading and some investigation by Microsoft Azure support, we've determined that the observed behavior with my long running background thread is almost certainly due to how the underlying IIS web server is handling threads and thread pools.  Short story is that I guess I was doing something that just doesn't play nice with the web hosting container I'm using.  I've switched gears and carved off my background processing task into a standalone entity which will be run as an Azure Web Job.  Thanks to all that responded.</p>