Correlation matrix with huge dataset - Python
Question: In python, when I have a dataset X, whose rows are the different elements of
the sample and the columns are different feature of the sample, I usually
calculate the correlation matrix as follows (assuming zero mean):

    
    
    import numpy as np
    np.transpose(X).dot(X)/row
    

Now I have a dataset X that has 10 milion rows and 1 milion of columns, and if
I try to calculate that correlation matrix, the computer remains blocked. The
alternative of performing a for-loop seems to me impracticable, because is
very slow also with smaller datasets.

How shoul I manage such amount of data?


Answer: Maybe you could use a generator? I don't know if it would be acceptably fast
but it would be better memory-wise so that the computer won't block. Something
like this <https://gist.github.com/thatdatabaseguy/2577076>

Paraphrasing the example:

    
    
    import numpy as np
    
    def my_gen(some_tuple):
        for each in some_tuple:
            yield each
    
    X = np.fromiter(my_gen(database.get_data()))  # get_data() would return a cursor
    

**Edit** : This is in the case you cannot find another way of reducing/
transforming/ splitting your data. Either way it is going to be difficult to
handle on a single computer. Good luck!

