tools and techniques we ran into memory error while loading single file--still to go luckilywe have few tricks up our sleeve let' try these techniques over the course of the case studyuse sparse representation of data feed the algorithm compressed data instead of raw data use an online algorithm to make predictions we'll go deeper into each "trickwhen we get to use it now that we have our data locallylet' access it step of the data science processdata preparation and cleansingisn' necessary in this case because the urls come pre-cleaned we'll need form of exploration before unleashing our learning algorithmthough step data exploration to see if we can even apply our first trick (sparse representation)we need to find out whether the data does indeed contain lots of zeros we can check this with the following piece of codeprint "number of non-zero entries % ffloat( shape[ ]))float(( nnz)/(float( shape[ ]this outputs the followingnumber of non-zero entries data that contains little information compared to zeros is called sparse data this can be saved more compactly if you store the data as [( , , ),( , , )instead of [[ , , , , ],[ , , , , ],[ , , , , ],[ , , , , ],[ , , , , ]one of the file formats that implements this is svmlightand that' exactly why we downloaded the data in this format we're not finished yetthoughbecause we need to get feel of the dimensions within the data to get this information we already need to keep the data compressed while checking for the maximum number of observations and variables we also need to read in data file by file this way you consume even less memory second trick is to feed the cpu compressed files in our exampleit' already packed in the tar gz format you unpack file only when you need itwithout writing it to the hard disk (the slowest part of your computerfor our exampleshown in listing we'll only work on the first filesbut feel free to use all of them