Predictive modeling on big data set that can't fit into memory I am trying to build a Decision-Tree model on top of a dataset that is about 10G in size on my local computer. However, I only have 8G memory. What I am doing now is just random sampling certain subset of the data, trying different model parameters and checking the prediction performance with the scikit-learn package. Still I need a final model based on the model parameters from those random sampling results, but obviously I cannot build a model on top of the full dataset. What can I do with this problem? I am a newbie of machine learning, any suggestions are welcome. Thanks!
Whether you're using this for a hobby or for your job, I would recommend EMR on Amazon Web Services. The name is anachronistic (it's no longer just for map reduce), but EMR enables you to quickly stand up an Apache Spark cluster of several machines. Spark comes with machine learning libraries for building trees, and can scale to however many machines you need (to the point that you no longer face memory constraints). EMR is cheap because you're charged hourly and don't have to buy your own hardware. Apache Spark is great for your resume, and it's also a practical skill because it's generally the first tool most data scientists turn to when working with data sets that are too large for a single machine.