general programming tips for dealing with large data sets the tricks that work in general programming context still apply for data science several might be worded slightly differentlybut the principles are essentially the same for all programmers this section recapitulates those tricks that are important in data science context you can divide the general tricks into three partsas shown in the figure mind mapdon' reinvent the wheel use tools and libraries developed by others get the most out of your hardware your machine is never used to its full potentialwith simple adaptions you can make it work harder reduce the computing need slim down your memory and processing needs as much as possible problems solutions handling large data don' reinvent the wheel general tips get the most out of your hardware reduce the computing needs figure overview of general programming best practices when working with large data "don' reinvent the wheelis easier said than done when confronted with specific problembut your first thought should always be'somebody else must have encountered this same problem before me don' reinvent the wheel "don' repeat anyoneis probably even better than "don' repeat yourself add value with your actionsmake sure that they matter solving problem that has already been solved is waste of time as data scientistyou have two large rules that can help you deal with large data and make you much more productiveto bootexploit the power of databases the first reaction most data scientists have when working with large data sets is to prepare their analytical base tables inside database this method works well when the features you want to prepare are fairly simple when this preparation involves advanced modelingfind out if it' possible to employ user-defined functions and procedures the last example of this is on integrating database into your workflow use optimized libraries creating libraries like mahoutwekaand other machinelearning algorithms requires time and knowledge they are highly optimized