them in the last example of this when you build recommender system within database hash tables are used extensively in databases as indices for fast information retrieval selecting the right tools with the right class of algorithms and data structures in placeit' time to choose the right tool for the job the right tool can be python library or at least tool that' controlled from pythonas shown figure the number of helpful tools available is enormousso we'll look at only handful of them problems choose the right algorithms choose the right data structures numexpr bcolz numba solutions python tools blaze handling large data choose the right tools theano cython general tips use python as master to control other tools figure overview of tools that can be used when working with large data python tools python has number of libraries that can help you deal with large data they range from smarter data structures over code optimizers to just-in-time compilers the following is list of libraries we like to use when confronted with large datacython--the closer you get to the actual hardware of computerthe more vital it is for the computer to know what types of data it has to process for computeradding is different from adding the first example consists of integers and the second consists of floatsand these calculations are performed by different parts of the cpu in python you don' have to specify what data types you're usingso the python compiler has to infer them but inferring data types is slow operation and is partially why python isn' one of the fastest languages available cythona superset of pythonsolves this problem by forcing the programmer to specify the data type while developing the program once the compiler has this informationit runs programs much faster see numexpr --numexpr is at the core of many of the big data packagesas is numpy for in-memory packages numexpr is numerical expression evaluator for numpy but can be many times faster than the original numpy to achieve