title: Best Practice for watching an upload folder?
body: <p>Ok I've got a bit of an interesting problem on my hands. Here's a bit of background:</p>
  
  <p>I'm writing a media library service implementation that will serve URLs to a front end flash player. The client wants to be able to push content into the service by uploading files with some metadata into an FTP folder - I have control of the metadata schema. The service watching this folder will pick up any new files, copy them to a "content" folder with, then push the meta data and the url's to the content down the content service and into the database.</p>
  
  <p>The content service isn't a problem, done. Watching an FTP folder is.</p>
  
  <p>My current implementation uses the FileSystemWatcher object with a filter for xml files. </p>
  
  <p>There may be more than one file for each content item e.g. high, med, low quality videos. </p>
  
  <p>I plan to enforce either by process or by a tool that the content is organised into it's own folders just for brevity but this isn't really an issue.</p>
  
  <p>The xml file will look a bit like this:</p>
  
  <pre><code>&lt;media&gt;
    &lt;meta type="video"&gt;
      &lt;name&gt;The Name Displayed&lt;/name&gt;
      &lt;heading&gt;The title of the video&lt;/heading&gt;
      &lt;description&gt;
          A lengthy description about the video..
      &lt;/description&gt;
      &lt;length&gt;00:00:19&lt;/length&gt;
  &lt;/meta&gt;
  &lt;files&gt;
      &lt;video file="somevideo.flv" quality="low"/&gt;
      &lt;video file="somevideo.flv" quality="medium"/&gt;
      &lt;video file="somevideo.flv" quality="high"/&gt;
  &lt;/files&gt;
   &lt;/media&gt;
  </code></pre>
  
  <p>So when a new file is created the FileSystemWatcher.Created event fires. I've got a seperate thread running to process the content which shares a queue with the main service process (don't worry it's using the producer consumer pattern as detailed here: <a href="http://msdn.microsoft.com/en-us/library/yy12yx1f.aspx" rel="noreferrer">http://msdn.microsoft.com/en-us/library/yy12yx1f.aspx</a>). </p>
  
  <p>This all works fine, but now I'm running into edge cases left right and centre!</p>
  
  <p>I've taken into account that the videos will take longer to upload so the processor will try to get an exclusive lock, if that fails it will move the item to the back of the queue and move onto the next item.</p>
  
  <ul>
  <li>What happens if the service crashes or there's already files there? so I wrote a method to queue up an existing files.</li>
  <li>What if files are getting uploaded while it's queueing up the existing files? Hopefully it will handle this? Should I periodically rescan the directory to see if I missed anything?</li>
  </ul>
  
  <p>Can anyone recommend a best practice for this scenario? Is the filesystemwatcher a good idea or should the service just periodically scan the folder?</p>
  
  <p>Edit: Just to give you some idea of scale. We're talking 10s of 1000s of items in total. Probably uploaded in large chunks.</p>