handling large data on single computer (continuedthe formula in figure shows that there' no difference between adding matrices and together in one step or first adding the upper half of the matrices and then adding the lower half all the common matrix and vector operationssuch as multiplicationinversionand singular value decomposition ( variable reduction technique like pca)can be written in terms of block matrices block matrix operations save memory by splitting the problem into smaller blocks and are easy to parallelize although most numerical packages have highly optimized codethey work only with matrices that can fit into memory and will use block matrices in memory when advantageous with out-of-memory matricesthey don' optimize this for you and it' up to you to partition the matrix into smaller matrices and to implement the block matrix version linear regression is way to predict continuous variables with linear combination of its predictorsone of the most basic ways to perform the calculations is with technique called ordinary least squares the formula in matrix form is xtx- xty where is the coefficients you want to retrievex is the predictorsand is the target variable the python tools we have at our disposal to accomplish our task are the following: bcolz is python library that can store data arrays compactly and uses the hard drive when the array no longer fits into the main memory dask is library that enables you to optimize the flow of calculations and makes performing calculations in parallel easier it doesn' come packaged with the default anaconda setup so make sure to use conda install dask on your virtual environment before running the code below notesome errors have been reported on importing dask when using bit python dask is dependent on few other libraries (such as toolz)but the dependencies should be taken care of automatically by pip or conda the following listing demonstrates block matrix calculations with these libraries for those who want to give it trygiven transformations are easier to achieve than householder transformations when calculating singular value decompositions