handling large data on single computer thisit rewrites your expression and uses an internal (just-in-timecompiler see numba --numba helps you to achieve greater speed by compiling your code right before you execute italso known as just-in-time compiling this gives you the advantage of writing high-level code but achieving speeds similar to those of code using numba is straightforwardsee bcolz--bcolz helps you overcome the out-of-memory problem that can occur when using numpy it can store and work with arrays in an optimal compressed form it not only slims down your data need but also uses numexpr in the background to reduce the calculations needed when performing calculations with bcolz arrays see blaze --blaze is ideal if you want to use the power of database backend but like the "pythonic wayof working with data blaze will translate your python code into sql but can handle many more data stores than relational databases such as csvsparkand others blaze delivers unified way of working with many databases and data libraries blaze is still in developmentthoughso many features aren' implemented yet see index html theano--theano enables you to work directly with the graphical processing unit (gpuand do symbolical simplifications whenever possibleand it comes with an excellent just-in-time compiler on top of that it' great library for dealing with an advanced but useful mathematical concepttensors see deeplearning net/software/theanodask--dask enables you to optimize your flow of calculations and execute them efficiently it also enables you to distribute calculations see dask pydata org/en/latestthese libraries are mostly about using python itself for data processing (apart from blazewhich also connects to databasesto achieve high-end performanceyou can use python to communicate with all sorts of databases or other software use python as master to control other tools most software and tool producers support python interface to their software this enables you to tap into specialized pieces of software with the ease and productivity that comes with python this way python sets itself apart from other popular data science languages such as and sas you should take advantage of this luxury and exploit the power of specialized tools to the fullest extent possible features case study using python to connect to nosql databaseas does with graph data let' now have look at more general helpful tips when dealing with large data