How to deal with memory exceptions while processing large amount of data? Am trying to develop a python script which reads a large CSV file (approx 1.2 GB historical data) in chunks and performs following steps:


Take backup of the file
Extract new records of previous day transactions,append to original/base CSV file and store the data in dataframe.
Perform mathematical operations on the big dataframe
Convert big dataframe to CSV and store in same location for nex day processing.


Similar process runs next day as well and so on...

I'm getting memory exception error while processing the step 3 above (probably step 1 &amp; 2 would have consumed most of the memory max-3GB limited space). Even if I extend the space in the server, I foresee a problem as my input file size is going to be increase daily.

I need to dip through historical data daily for mathematical operations along with daily transactions so can't avoid storing/accessing of base CSV file which contains historical data and can't afford to buy space in cloud as well.

I have used pd.read_csv for reading records from CSV file in chunks, for mathematical operations pandas and numpy has been used.

The script runs without any issue on my local machine, however I have the memory issue while processing on the server. So, as far as the code goes, it seems to be good.

I believe that if I can change the processing in steps 1 to 4 to be more efficient, the memory exception can be solved and with limited space my script can be executed.

Can someone suggest the best way to handle the above steps?
I had also faced similar issues when my python script was processing my 12GB and 35GB files. It used take weeks for processing and many times failed.  

You need to check your mathematical operation, if it can work on subset of data. In my case, I finally was able to split the file in multiple pieces and run the mathematical operation on individual pieces. It helped improving the speed as well as better exceptional handling.  

If you cant split your file, then check if your code can work in batches. Then you can read your file in chunks (in pd.read_csv you can supply chunk size) and then feed to your formula piece by piece.