memory problem due to large file
I guess you are trying to use pandas, if so. Don't because you can't do that for what you want. I guess your operation needs all data to be loaded to the memory which is not possible. Try to use dask or a kind of sql. I'm not sure whether you want to do operations such as group by which needs all data to be loaded simultaneously or not. If you can do extra coding and your operations don't imply to load all data, such as finding the minimum of a column, you can use generators and specify the chunk_size of the read_csv method in pandas. But you have to do extra coding. You can also take a look at here. You can also take a look at here to figure out why sql and dask are more better for large operations.