handling large data on single computer mission accomplished our happy movie addict can now indulge himself with new movietailored to his preferences in the next we'll look at even bigger data and see how we can handle that using the horton sandbox we downloaded in summary this discussed the following topicsthe main problems you can run into when working with large data sets are thesenot enough memory long-running programs resources that form bottlenecks and cause speed problems there are three main types of solutions to these problemsadapt your algorithms use different data structures rely on tools and libraries three main techniques can be used to adapt an algorithmpresent algorithm data one observation at time instead of loading the full data set at once divide matrices into smaller matrices and use these to make your calculations implement the mapreduce algorithm (using python libraries such as hadoopyoctopydiscoor dumbothree main data structures are used in data science the first is type of matrix that contains relatively little informationthe sparse matrix the second and third are data structures that enable you to retrieve information quickly in large data setthe hash function and tree structure python has many tools that can help you deal with large data sets several tools will help you with the size of the volumeothers will help you parallelize the computationsand still others overcome the relatively slow speed of python itself it' also easy to use python as tool to control other data science tools because python is often chosen as language in which to implement an api the best practices from computer science are also valid in data science contextso applying them can help you overcome the problems you face in big data context