Optimizing a python function with numpy arrays
Question: I have been trying to optimize a python script I wrote for the last two days.
Using several profiling tools (cProfile, line_profiler etc.) I narrowed down
the issue to the following function below.

`df` is a numpy array with 3 columns and +1,000,000 rows (data type is float).
Using line_profiler, I found out that the function spends most of the time
whenever it needs to access the numpy array.

`full_length += head + df[rnd_truck, 2]`

and

`full_weight += df[rnd_truck,1]`

take most of the time, followed by

`full_length = df[rnd_truck,2]`

`full_weight = df[rnd_truck,1]`

lines.

As far as I see the bottleneck is caused by the access time the function tries
to grab a number from the numpy array.

When I run the function as `MonteCarlo(df, 15., 1000.)` it takes 37 seconds to
call the function for 1,000,000 times in a i7 3.40GhZ 64bit Windows machine
with 8GB RAM. In my application, I need to run it for 1,000,000,000 to ensure
convergence, which brings the execution time to more than an hour. I tried
using the `operator.add` method for the summation lines, but it did not help
me at all. It looks like I have to figure out a faster way to access this
numpy array.

Any ideas would be welcome!

    
    
    def MonteCarlo(df,head,span):
        # Pick initial truck
        rnd_truck = np.random.randint(0,len(df))
        full_length = df[rnd_truck,2]
        full_weight = df[rnd_truck,1]
    
        # Loop using other random truck until the bridge is full
        while 1:
            rnd_truck = np.random.randint(0,len(df))
            full_length += head + df[rnd_truck, 2]
            if full_length > span:
                break
            else:
                full_weight += df[rnd_truck,1]
    
        # Return average weight per feet on the bridge
        return(full_weight/span)
    

Below is a portion of the `df` numpy array I am using:

    
    
    In [31] df
    Out[31]: 
    array([[  12. ,  220.4,  108.4],
           [  11. ,  220.4,  106.2],
           [  11. ,  220.3,  113.6],
           ..., 
           [   4. ,   13.9,   36.8],
           [   3. ,   13.7,   33.9],
           [   3. ,   13.7,   10.7]])
    


Answer: As noted by other people, this isn't vectorized at all, so your slowness is
really due to slowness of the Python interpreter. [Cython](http://cython.org)
can help you a lot here with minimal changes:

    
    
    >>> %timeit MonteCarlo(df, 5, 1000)
    10000 loops, best of 3: 48 us per loop
    
    >>> %timeit MonteCarlo_cy(df, 5, 1000)
    100000 loops, best of 3: 3.67 us per loop
    

where `MonteCarlo_cy` is just (in the IPython notebook, after `%load_ext
cythonmagic`):

    
    
    %%cython
    import numpy as np
    cimport numpy as np
    
    def MonteCarlo_cy(double[:, ::1] df, double head, double span):
        # Pick initial truck
        cdef long n = df.shape[0]
        cdef long rnd_truck = np.random.randint(0, n)
        cdef double full_weight = df[rnd_truck, 1]
        cdef double full_length = df[rnd_truck, 2]
    
        # Loop using other random truck until the bridge is full
        while True:
            rnd_truck = np.random.randint(0, n)
            full_length += head + df[rnd_truck, 2]
            if full_length > span:
                break
            else:
                full_weight += df[rnd_truck, 1]
    
        # Return average weight per feet on the bridge
        return full_weight / span
    

