title: MySQL: Retrieve millions of rows from multiple shards at the same time
body: <p>We have 2 MySQL DBs(both tables in 16 shards) with 100M+ rows and growing/updating. For reporting purposes, </p>
  
  <ol>
  <li>I need to fetch all the rows(50M+) on a daily basis, from 1st DB</li>
  <li>I need to fetch all the rows(50M+) created on that day, from 2nd DB</li>
  </ol>
  
  <p>These rows might update/insertion can happen anytime.</p>
  
  <p>Requirements:</p>
  
  <ol>
  <li>Minimum memory utilization(the retrieved data should not be kept in memory, either client or server, before writing into a file)</li>
  <li>Fetch the data from a midnight DB snapshot.</li>
  <li>DB snapshot because if the query took 10 mins and data changed in b/w, the data retrieved should be of midnight snapshot(should not fetch the changed/updated data after midnight).</li>
  <li>There are 16 shards. The dump should happen in all shards at the same time. Else data would be inconsistent.</li>
  </ol>
  
  <p>What is the best way to query the 2 tables with above requirements in mind?</p>
  
  <p>Note: I tried to write a python script with MySQLDB and <a href="http://mysql-python.sourceforge.net/MySQLdb-1.2.2/public/MySQLdb.cursors-module.html" rel="nofollow noreferrer">this doc</a> mentions that the retrieved data would be kept in either client mem or server mem, which I don't want.</p>
  
  <p>Any help would be greatly appreciated.</p>