title: Amazon MapReduce best practices for logs analysis
body: <p>I'm parsing access logs generated by Apache, Nginx, Darwin (video streaming server) and aggregating statistics for each delivered file by date / referrer / useragent. </p>
  
  <p>Tons of logs generated every hour and that number likely to be increased dramatically in near future - so processing that kind of data in distributed manner via Amazon Elastic MapReduce sounds reasonable.</p>
  
  <p>Right now I'm ready with mappers and reducers to process my data and tested the whole process with the following flow:</p>
  
  <ul>
  <li>uploaded mappers, reducers and data to Amazon S3</li>
  <li>configured appropriate job and processed it successfully</li>
  <li>downloaded aggregated results from Amazon S3 to my server and inserted them into MySQL database by running CLI script</li>
  </ul>
  
  <p>I've done that manually according to thousands of tutorials that are googlable on the Internet about Amazon ERM. </p>
  
  <p>What should I do next? What is a best approach to automate this process?</p>
  
  <ul>
  <li>Should I control Amazon EMR jobTracker via API?</li>
  <li>How can I make sure my logs will not be processed twice?</li>
  <li>What is the best way to move processed files to archive?</li>
  <li>What is the best approach to insert results into PostgreSQL/MySQL?</li>
  <li>How data for the jobs should be laid out in input/output directories?</li>
  <li>Should I create a new EMR job each time using the API?</li>
  <li>What is the best approach to upload raw logs to Amazon S3?</li>
  <li>Can anyone share their setup of the data processing flow?</li>
  <li>How to control file uploads and jobs completions?</li>
  </ul>
  
  <p>I think that this topic can be useful for many people who try to process access logs with Amazon Elastic MapReduce but were not able to find good materials and/or best practices.</p>
  
  <p><strong>UPD:</strong> Just to clarify here is the single final question:</p>
  
  <p><strong>What are best practices for logs processing powered by Amazon Elastic MapReduce?</strong></p>
  
  <p>Related posts:</p>
  
  <p><a href="https://stackoverflow.com/questions/7701678/getting-data-in-and-out-of-elastic-mapreduce-hdfs">Getting data in and out of Elastic MapReduce HDFS</a></p>