title: processing large number of files - any library or utility to help?
body: <p>I'd like to process a large number of files - think video transcoding, like youtube, where you have to transform everything into FLV format or image processing where you create thumbs from large RAW files.</p>
  
  <p>Is there any service or library that can help on such processing? I don't mean actual transcoding, but organizing, launching the tasks, monitoring them and handling errors somehow.</p>
  
  <p>Ideally on windows but linux could also be fine.</p>
  
  <p>Update: I'd like utility/lib to handle such tasks at higher level. Dispatch multiple processes, handle and report any failures somehow, etc. </p>
  
  <p>So, in case of transcoding, I'd like such lib/utility to get a list of files to transcode and then it would handle starting the trancoding in multiple threads/processes. You would be able to define what to do on successful completition of the task. And you'd also be able to do this in case of a failure.</p>