title: How can I read a huge CSV file in Rails on Heroku?
body: <p>I am working on a Rails app hosted on Heroku.</p>
  
  <p>Everyday I need to download a huge ZIP file (> 200MB) which contains a CSV with a list of products, product attributes and prices, which I must then unzip and go through to update the prices of those products on my website.</p>
  
  <p>What I was doing initially was :</p>
  
  <ul>
  <li>Job A to download the ZIP in the <code>tmp</code> folder on Heroku, unzip the CSV in the <code>tmp</code> folder</li>
  <li>Job B to read through the CSV, find the products and update the prices in the DB</li>
  </ul>
  
  <p>This was fine until I added multiple workers and dynos to my app, because Heroku's temporary files are only available to one dyno so the CSV file would only be available to the dyno that was responsible for Job A. I added multiple dynos and workers to split Job B into smaller chunks and run this whole process faster overall.</p>
  
  <p>As an alternative I tried downloading the file to an S3 bucket instead - but parsing an 200MB CSV file from S3 is extremely long (approximately 15 minutes). So I thought of parsing it and converting it to an array of hashes which I would then store somewhere to be able to access it easily but where could I store this?</p>
  
  <p>I would appreciate any guidance regarding this issue. There probably is an obvious solution which I haven't thought of because I've been working on this for too long now.</p>