title: Recommendation for batch processing on aws
body: <p>I'm new to using AWS, so any pointers would be appreciated.
  I have a need to process large files using our in-house software.</p>
  
  <p>It takes about 2GB of input and generates 5GB of output, running for 2 hours on a c3.8xlarge.</p>
  
  <p>For now I do it manually, start an instance (either on-demand or spot-request), but now I want to reliably automate and scale this processing - what are good frameworks or platform or amazon services to do that?
  Especially regarding the possibility that a spot-instance will be terminated half-way through (and I'll need to detect that and restart the job).</p>
  
  <p>I heard about Python Celery, but does it work well with amazon and spot-instances?
  Or are there other recommended mechanisms?</p>
  
  <p>Thank you!</p>