Outputting scatter plots I have written a python function that outputs scatter plots using Matplotlib after processing the data a little. It works but it's painfully slow. I was wondering if anybody had any suggestions as to how I can improve the efficiency of this function or whether its slow just because it's processing a lot of data (a 30x43 element Pandas data frame).

def scatter_x_yi(self,filtered_data,x_parameter):
    '''
   takes some prefiltered data (len(x) by len(Y)) and the name of a
    data column (x_parameter), plots it against every other column (yi)
    '''
    PE=Parameter_Estimation_Tools() #custom class 
    number_of_full_subplots=len(filtered_data)/16   
    remainder=len(filtered_data)-(number_of_full_subplots*16)-1
    os.path.dirname(str(filtered_data))     
    '''
    some code for isolating the x parameter column
    and all the y's (yi)
    '''
    yi=[]
    for i in range(len(filtered_data)):
        if filtered_data[i].name!=x_parameter:
            yi.append(filtered_data[i])      
    '''
    code for plotting the scatter plots (43 in total)
    using matplotlib in 'units' of 4 by 4, then plotting 
    the remainder in a last figure

    '''
    try:
        for i in range(len(filtered_data)):
            for j in range(number_of_full_subplots+1):
                fig=plt.figure(j)
                txt = fig.suptitle('{} , {} of {}'.format(str(x_parameter),str(j),str(number_of_full_subplots)),fontsize='18')
                txt.set_text('{} , {} of {}'.format(str(x_parameter),str(j),str(number_of_full_subplots)))                    
                for k in range(16):
                    plt.subplot(4,4,k)
                    PE.scatter_x_y(filtered_data,x_parameter,yi[16*j+k].name)
                    plt.subplots_adjust(wspace=0.2, hspace=0.25,top=0.9)
            mng=plt.get_current_fig_manager()
            mng.window.showMaximized()
            plt.ioff()
            fig.savefig(str(j)+'.png',bbox_inches='tight',orientation='landscape')
            plt.close(fig)
            plt.ion()
    except IndexError:
        for i in range(len(filtered_data)-remainder,len(filtered_data)-1):
            PE.scatter_x_y(filtered_data,x_parameter,yi[i].name)
            plt.subplots_adjust(wspace=0.2, hspace=0.25,top=0.9)
            mng=plt.get_current_fig_manager()
            mng.window.showMaximized()
            plt.ioff()
            fig.savefig('2.png',bbox_inches='tight',orientation='landscape')
            plt.close(fig)
            plt.ion()
        print '{} has been plotted against all other parameters'.format(x_parameter)
Based on the code provided, here are a few suggestions for where decreasing the time it takes to run it:

Too many ranges, iterators

The first thing that pops out as an area for improvement is the number of basic iterators being used, like for x in range()... and when you are using the Pandas library, it's likely there's a speed improvement there.

The problem with using for x in range() is that the script has to go through every single item in the range, one at a time. Pandas has a number of more efficient approaches to handling this, namely df.groupby(), df[colname].apply(), and in the worst case scenario for index, row in df.iterrows() that will very likely cut down your time.

Here's one example:

yi = []
for i in range(len(filtered_data)):
    if filtered_data[i].name != x_parameter:
        yi.append(filtered_data[i])


It appears you are simply trying to make a subset of filtered_data where the column name doesn't match the x_parameter. Assuming that filtered_data is a Pandas dataframe object, you can replace the code block above with this and it should be faster:

yi = filtered_data.loc[filtered_data.name != x_parameter]


Parallelize the chart generation

Instead of having the code blocked from processing data while it waits for matplotlib to create and save the next chart, making this into a separate function and process that can be run concurrently may save you some time. Breaking up the different steps in your pipeline allows you to run the slow loops separately, and also makes testing easier. For a specific example of how to use multiprocessing to run your charting code concurrently, check out this answer on Stackoverflow.

Use ipython magic functions

It is very helpful to use ipython and the magic functions it contains to debug time issues in code. For instance, if you have broken up your one big script into smaller functions, you can use %timeit and to find out what is taking too long.

Lose your self

This isn't really a speed optimization, but definitely helps with readability.

self is the first argument to this function, however it doesn't seem this variable is used elsewhere. My guess is that the function being asked about is actually a method within a class. One quick and easy optimization would be to remove self from the argument list, since it doesn't appear that you need it, and then add @staticmethod to the just before your function. So this will look like:

@staticmethod
def scatter_x_yi(filtered_data, x_parameter):
    [... rest of the method...]


More Possibilities


There may be some additional ways to optimize this code hidden in the Parameter_Estimation_Tools(). What is that process doing? Take a look if that is taking too much time.
It appears this isn't a standalone function but a method contained in a class. What if this function were broken out to run in isolation?