Looking for efficient python program for this following python script
Question: I am looking for a memory efficient python script for the following script.
The following script works well for smaller dimension but the dimension of the
matrix is 5000X5000 in my actual calculation. Therefore, it takes very long
time to finish it. Can anyone help me how can I do that?

    
    
    def check(v1,v2):
        if len(v1)!=len(v2):
            raise ValueError,"the lenght of both arrays must be the same"
        pass
    def d0(v1, v2):
        check(v1, v2)
        return dot(v1, v2)
    import numpy as np
    from pylab import *
    vector=[[0.1, .32, .2, 0.4, 0.8], [.23, .18, .56, .61, .12], [.9, .3, .6, .5, .3], [.34, .75, .91, .19, .21]]
    rav= np.mean(vector,axis=0)
    #print rav
    #print vector
    m= vector-rav
    corr_matrix=[]
    for i in range(0,len(vector)):
        tmp=[]
        x=sqrt(d0(m[i],m[i]))
        for j in range(0,len(vector)):
            y=sqrt(d0(m[j],m[j]))
            z=d0(m[i],m[j])
            w=z/(x*y)
            tmp.append(w)
        corr_matrix.append(tmp)
    print corr_matrix
    


Answer: Make your `matrix` (and your `vector`) into numpy `array`s instead of Python
`list`s. That will make it take much less memory (and also run faster).

To understand why:

A Python `list` is a list of Python object instances. Each one of these has
type information, pointers, and all kinds of other stuff to keep around beyond
just the 8-byte number. Let's say each one ends up being 64 bytes instead of
8. So, that's 64 bytes per element, times 25M elements, equals 1600M bytes!

By contrast, a numpy `array` is a list of just the raw values, together with a
single copy of all that extra information (in the `dtype`). So, instead of 64
* 25M bytes, you've got 8 * 25M + 64 bytes, which is only 1/8th the size.

As for the speed increase: If you iterate over a 5000x5000 matrix, you're
calling some code in the inner loop 25M times. If you're doing a numpy
expression like `m + m`, the code inside the loop is a few lines of C code
that get compiled down to a few dozen machine-code operations, which is
blazingly fast. If you're doing the loop explicitly in Python, the inside of
that loop has to drive the Python interpreter every time through the loop,
which is much, much slower. (On top of that, the C compiler will optimize the
code, and numpy may have some explicit optimizations too.) Depending on how
trivial the work inside the loop is, the speedup can be anywhere from 2x to
10000x. So, even if you have to make things a bit convoluted, try to find a
way to express each step as an array broadcast rather than a loop, and it will
be much faster.

* * *

So, how do you do that? Simple. Instead of this:

    
    
    corr_matrix=[]
    for i in range(len(vector)):
        tmp=[]
        # …
        for j in range(len(vector)):
            # …
            tmp.append(w)
        corr_matrix.append(tmp)
    

Do this:

    
    
    corr_matrix=np.zeros((len(vector), len(vector))
    for i in range(len(vector)):
        # …
        for j in range(len(vector)):
            # …
            corr_matrix[i, j] = w
    

That immediately eliminates all the memory problems caused by the overhead of
keeping around 25M Python `float` objects, and will give you a significant
speed boost too. You can't reduce the memory any further except by not keeping
the whole `array` in memory at once, but you should be fine already. (You can
boost the speed even more by using broadcast operations in place of loops, but
if the memory is your problem, and the performance is fine, it may not be
necessary.)

