Opening a 20GB file for analysis with pandas I am currently trying to open a file with pandas and python for machine learning purposes it would be ideal for me to have them all in a DataFrame. Now The file is 18GB large and my RAM is 32 GB but I keep getting memory errors.

From your experience is it possible? If not do you know of a better way to go around this? (hive table? increase the size of my RAM to 64? create a database and access it from python)
If it's a csv file and you do not need to access all of the data at once when training your algorithm, you can read it in chunks. The pandas.read_csv method allows you to read a file in chunks like this: 

import pandas as pd
for chunk in pd.read_csv(&lt;filepath&gt;, chunksize=&lt;your_chunksize_here&gt;)
    do_processing()
    train_algorithm()


Here is the method's documentation