lower-level language such as or fortran and integrate this with your codebase if you make the step to lower-level languages (languages that are closer to the universal computer bytecode)learn to work with computational libraries such as lapackblastintel mkland atlas these are highly optimizedand it' difficult to achieve similar performance to them avoid pulling data into memory when you work with data that doesn' fit in your memoryavoid pulling everything into memory simple way of doing this is by reading data in chunks and parsing the data on the fly this won' work on every algorithm but enables calculations on extremely large data sets use generators to avoid intermediate data storage generators help you return data per observation instead of in batches this way you avoid storing intermediate results use as little data as possible if no large-scale algorithm is available and you aren' willing to implement such technique yourselfthen you can still train your data on only sample of the original data use your math skills to simplify calculations as much as possible take the following equationfor example( ) ab the left side will be computed much faster than the right side of the equationeven for this trivial exampleit could make difference when talking about big chunks of data case study predicting malicious urls the internet is probably one of the greatest inventions of modern times it has boosted humanity' developmentbut not everyone uses this great invention with honorable intentions many companies (googlefor onetry to protect us from fraud by detecting malicious websites for us doing so is no easy taskbecause the internet has billions of web pages to scan in this case study we'll show how to work with data set that no longer fits in memory what we'll use data--the data in this case study was made available as part of research project the project contains data from daysand each observation has approximately , , features the target variable contains if it' malicious website and - otherwise for more informationplease see "beyond blacklistslearning to detect malicious web sites from suspicious urls " the scikit-learn library--you should have this library installed in your python environment at this pointbecause we used it in the previous as you can seewe won' be needing much for this caseso let' dive into it justin malawrence saulstefan savageand geoffrey voelker"beyond blacklistslearning to detect malicious web sites from suspicious urls,proceedings of the acm sigkdd conferenceparis (june ) -