Speeding up summation for loop in python
Question: I have the following bottleneck and am wondering if anyone can suggest ways to
speed it up.

I have three lists `x,y,z` of length `N`. and I apply the following
[summation](http://en.wikipedia.org/wiki/Correlation_sum).

    
    
    def abs_val_diff(x1, x2, x3, y1, y2, y3):
        """ Find the absolute value of the difference between x and y """
        return py.sqrt((x1 - y1) ** 2.0 + (x2 - y2) ** 2.0 + (x3 - y3) ** 2.0)
    
    R = 0.1  
    sumV = 0.0
    for i in xrange(N):
        for j in xrange(i + 1, N):
            if R > abs_val_diff(x[i], y[i], z[i],
                                x[j], y[j], z[j]):
                    sumV += 1.0
    

I have tried using numpy arrays, but either I am doing something wrong or
there is a reduction in speed of about a factor of 2.

Any ideas would be highly appreciated.


Answer: I believe you can utilize numpy a little more efficiently by doing something
like the following. Make a small modification to your function to use the
numpy.sqrt:

    
    
    import numpy as np
    
    def abs_val_diff(x1, x2, x3, y1, y2, y3):
        """ Find the absolute value of the difference between x and y """
        return np.sqrt((x1 - y1) ** 2.0 + (x2 - y2) ** 2.0 + (x3 - y3) ** 2.0)
    

Then call with the full arrays:

    
    
    res = abs_val_diff(x[:-1],y[:-1],z[:-1],x[1:],y[1:],z[1:])
    

Then, because you're adding 1 for each match, you can simply take the length
of the array resulting from a query against the result:

    
    
    sumV = len(res[R>res])
    

This lets numpy handle the iteration. Hopefully that works for you

