title: Reading and writing large volume of data in Python
body: <p>I am trying to retrieve a large amount of data(more than 7 million) from database and trying to save a s flat file. The data is being retrieved using python code(python calls stored procedure). But I am having a problem here. The process is eating up so much of memory hence killing the process automatically by unix machine. I am using read_sql_query to read the data and to_csv to write into flat file. So, I wanted to ask if there is a way to solve this problem. May be reading only a few thousand rows at a time and saving them and go to next line.
  I even used chunksize parameter as well. But it does not seem to resolve the issue.</p>
  
  <p>Any help or suggestion will be greatly appreciated.</p>