listing block matrix calculations with bcolz and dask libraries number of observations (scientific notation feel free to change this import dask array as da import bcolz as bc import numpy as np import dask creates fake datanp arange(nreshape( / , creates matrix of by (because we set to bc carray numpy is an array extension that can swap to disc this is also stored in compressed way rootdir 'ar bcolz--creates file on disc in case out of ram you can check this on your file system next to this ipython file or whatever location you ran this code from mode ' --is the write mode dtype 'float --is the storage type of the data (which is float numbersn ar bc carray(np arange(nreshape( / , dtype='float 'rootdir 'ar bcolz'mode ' ' bc carray(np arange( / )dtype='float 'rootdir 'yy bcolz'mode ' 'dax da from_array(archunks=( , )dy da from_array( ,chunks=( , )the xtx is defined (defining it as "lazy"as the matrix multiplied with its transposed version this is building block of the formula to do linear regression using matrix calculation xtx dax dot(daxxy dax dot(dyblock matrices are created for the predictor variables (arand target (ya block matrix is matrix cut in pieces (blocksda from_array(reads data from disc or ram (wherever it resides currentlychunks=( , )every block is matrix (unless observations or variables are leftxy is the vector multiplied with the transposed matrix again the matrix is only definednot calculated yet this is also building block of the formula to do linear regression using matrix calculation (see formulacoefficients np linalg inv(xtx compute()dot(xy compute()coef da from_array(coefficients,chunks=( , )ar flush( flush(flush memory data it' no longer needed to have large matrices in memory predictions dax dot(coefcompute(print predictions the coefficients are also put into block matrix we got numpy array back from the last step so we need to explicitly convert it back to "da array score the model (make predictionsthe coefficients are calculated using the matrix linear regression function np linalg inv(is the ^(- in this functionor "inversionof the matrix dot( --multiplies the matrix with another matrix note that you don' need to use block matrix inversion because xtx is square matrix with size nr of predictors nr of predictors this is fortunate because dask