handling large data on single computer listing checking data size we don' know how many features we haveso let' initialize it at we don' know how many observations we haveso let' initialize it at import tarfile from sklearn linear_model import sgdclassifier from sklearn metrics import classification_report from sklearn datasets import load_svmlight_file import numpy as np the uri variable holds the location in which you saved the downloaded files you'll need to fill out this uri variable yourself for the code to run on your computer all files together take up uri ' :\python book\\url_svmlight tar gzaround gb the tar tarfile open(uri," :gz"trick here is to leave the max_obs stop at the th file data compressed in max_vars (instead of all of themfor main memory and only demonstration purposesunpack what you need split initialize for tarinfo in tarfile print extracting % , size % (tarinfo nametarinfo sizecounter if tarinfo isfile()at tar extractfile(tarinfo nameuse helper functionx, load_svmlight_file(fload_svmlight_file(we unpack to load specific file max_vars np maximum(max_varsx shape[ ]the files one max_obs np maximum(max_obsx shape[ ]by one to reduce the memory needed if splitbreak + stop when we reach files adjust maximum number of observations and variables when necessary (big fileprint "max %smax dimension % (max_obsmax_vars print results part of the code needs some extra explanation in this code we loop through the svm files inside the tar archive we unpack the files one by one to reduce the memory needed as these files are in the svm formatwe use helperfunctionload_svmlight _file(to load specific file then we can see how many observations and variables the file has by checking the shape of the resulting data set armed with this information we can move on to model building step model building now that we're aware of the dimensions of our datawe can apply the same two tricks (sparse representation of compressed fileand add the third (using an online algorithm)in the following listing let' find those harmful websites