How to load a csv file into [Pandas] dataframe if computer runs out of RAM?
As noted many times by the writers of pandas, the ideal size of memory for analyzing with pandas is around 5-10 times the load of data you are giving. 

That being said, if you can afford to load data is chunks for pre-processing and dump only the columns that are needed, I recommended pandas.read_table option to load data in pieces (check option chunks).

pyspark as you mentioned might be good to go with. But there is dask which is built around python stack of pandas and numpy for distributed work loads which has support for pandas.DataFrame and numpy.array . But I never followed anyone who was successful in using it in production (or atleast mentioned that they had used it). May be some of the people here can vouch for it. There is another library called sframe to load the data way out of memory as the author suggested in one of his presentations. It keeps on serializing data onto disk. 

Hope I summarized it well.