title: How do I detect unexpected worker role failures and reprocess data in those cases?
body: <p>I want to create a web service hosted in Windows Azure. The clients will upload files for processing, the cloud will process those files, produce resulting files, the client will download them.</p>
  
  <p>I guess I'll use web roles for handling HTTP requests and worker roles for actual processing and something like Azure Queue or Azure Table Storage for tracking requests. Let's pretend it'll be Azure Table Storage - one "request" record per user uploaded file.</p>
  
  <p>A major design problem is processing a single file can take anywhere from one second to say ten hours.</p>
  
  <p>So I expect the following case: a worker role is started, gets to Azure Table Storage, finds a request marked "ready for processing", marks it "is being processed", starts actual processing. Normally it would process the file and mark the request "processed", but what if it dies unexpectedly?</p>
  
  <p>Unless I take care of it the request will remain in "is being processed" state forever.</p>
  
  <p>How do I track requests that are marked "is being processed" but abandoned? What mechanism in Windows Azure would be most convenient for that?</p>