handling large data on single computer step defining the research goal the goal of our project is to detect whether certain urls can be trusted or not because the data is so large we aim to do this in memory-friendly way in the next step we'll first look at what happens if we don' concern ourselves with memory (ramissues step acquiring the url data start by downloading the data from and place it in folder choose the data in svmlight format svmlight is text-based format with one observation per row to save spaceit leaves out the zeros figure memory error when trying to take large data set into memory the following listing and figure show what happens when you try to read in file out of the and create the normal matrix as most algorithms expect the todense(method changes the data from special file format to normal matrix where every entry contains value listing generating an out-of-memory error import glob from sklearn datasets import load_svmlight_file files glob glob(' :\users\gebruiker\downloadsurl_svmlight tar\url_svmlight\svm'files glob glob(' :\users\gebruiker\downloadsurl_svmlight\url_svmlight\svm'print "there are % fileslen(filesx, load_svmlight_file(files[ ],n_features= todense(the data is bigbut sparsematrix by turning it into dense matrix (every is represented in the file)we create an out-of-memory error points to files (linuxpoints to files (windowstar file needs to be untarred firstindication of number of files loads files surprisesurprisewe get an out-of-memory error that isunless you run this code on huge machine after few tricks you'll no longer run into these memory problems and will detect of the malicious sites