title: Which is most efficient and failproof way to fetch a table which has almost 50 million records using python?
body: <p>I have a requirement where I need to fetch a huge table that is presently stored in MySQL database using python.
  I have tried with Limit and offset, but that keeps on getting slowed down with the number of iterations. <br>
  I am using <code>mysqlclient</code> for making connections. <br>
  My current requirement is to write it into a file after each iteration.
  For testing, I am using CSV but I will switch to <code>parquet</code> file format once I find a good solution</p>
  <pre><code>select * from table_name  limit {} ,{} 
  </code></pre>
  <p>Please suggest me some of the methods that you are using and it works flawlessly.</p>
  <p>code snippet that i am using :</p>
  <pre><code>i = 1
  limit = 1000000
  curs.execute(&quot;select * from table_name limit 1&quot;)
  col = [col_[0] for col_ in curs.description]
  print(col)
  
  while True:
      start_ = time.time()
      query = f'select * from table_name    limit {i} ,{limit} '
      print(query)
      curs.execute(query)
      rows = curs.fetchall()
      if not len(rows):
          print(&quot;done&quot;)
          break
      print(&quot;###########&quot;)
      print(i)
      print(&quot;###########&quot;)
      end_ = time.time()
      print(f&quot;Runtime of the program is {end_ - start_}&quot;)
  
      df_temp = pd.DataFrame(rows, columns=col)
      df_temp.to_csv(f&quot;test_{i}.csv&quot;, header=True, index=False)
      # df_total.append(df_temp)
      i = i + limit
  </code></pre>
  <p>Or should I consider using different languages like scala?</p>