Excluding data from a pandas dataframe based on percentiles
Question: I am having difficulties trying to use Python to remove some data outliers
prior to producing a scatterplot. I have a n by 43 dataframe imported using
pandas. I have figured out how to determine the thresholds for the outliers
and applied this to the dataframe such that I now have some boolean values
corresponding to whether the data should be included in the scatter plot or
not. I am however stuck on how to use this information to exclude the
appropriate data points.

My code so far:

    
    
    def identify_outliers(self,parameters_file):
        data=pandas.read_csv(parameters_file) #import data
        header=data.keys()                    #get header
        quantiles = data.quantile([0.25,0.75],1) #determine thresholds for all data 
        for i in range(len(header)):
            qnt_i = quantiles[header[i]].as_matrix() #get handle to quantiles for specific column of data
            boolean_data=data[header[i]].between(qnt_i[0],qnt_i[1])  #identify data points that fall outside this range   
            for j in range(len(boolean_data)):      #attempt to use boolean values to filter data in 'data' to only include 'True' (doesn't work)
                if boolean_data[j]:
                    print data[header[i]]
    

Here is a snippet of data that is imported using pandas.read_csv

    
    
    (v1).Kcat   (v1).km (v11).k1
    1.22E-02    1.20E-02    1.72E-06
    0.0122441   1.42E-02    1.61E-06
    1.04E-02    1.01E-02    1.00E-06
    0.0136581   0.0185623   5.01158
    0.0113221   0.0221445   0.0785929
    0.506949    0.01    1.35E-06
    1.16567 0.0141031   168.078
    0.01    0.0100055   1.25E-06
    0.0351003   153.682 163.082
    0.0129821   0.0164996   0.0560866
    0.01    1.61671 1166.5
    0.0112294   0.0100472   1.17E-06
    0.0104352   0.0124419   1.63E-06
    0.0173491   0.01    0.000110292
    0.01    0.0409099   1.00E-06
    490.531 557.418 5.85845
    199.639 0.79314 0.00155387
    0.0104078   25.2456 0.0212165
    0.920923    84.1231 1.04E-05
    1.00E-02    3.07E+02    1.01E-06
    0.01    0.0113395   1.23E-06
    0.0799303   1.14812 1.00E-06
    0.403507    0.76484 436.664
    0.0118404   0.38389 1.06E-06
    

Does anybody have a suggestion as to how I can filter 'data' to remove all the
values which do not fall within the specified range

Thanks


Answer: **Edit:** Prior version used apply/lambda which is not really necessary. This
is a simpler version.

Here's a smaller dataframe based on just your first 10 rows.

    
    
    df
    
             v1          v2          v3
    0  0.012200    0.012000    0.000002
    1  0.012244    0.014200    0.000002
    2  0.010400    0.010100    0.000001
    3  0.013658    0.018562    5.011580
    4  0.011322    0.022145    0.078593
    5  0.506949    0.010000    0.000001
    6  1.165670    0.014103  168.078000
    7  0.010000    0.010006    0.000001
    8  0.035100  153.682000  163.082000
    9  0.012982    0.016500    0.056087
    

And here's mask that selects only values between the 25th and 75th
percentiles. Note that the syntax for this is somewhat precise so be careful
with the parentheses and such.

    
    
     ( df > df.quantile(.25) ) & ( df < df.quantile(.75) )
    
          v1     v2     v3
    0   True   True   True
    1   True   True   True
    2  False  False  False
    3   True  False  False
    4  False  False   True
    5  False  False  False
    6  False   True  False
    7  False  False  False
    8  False  False  False
    9   True   True   True
    

This is column-based, btw. I just glanced quickly at your code and couldn't
easily tell if the percentile measures were intended per-column of for the
combination of the 3 columns. For the whole dataframe you can do:

    
    
    ( df > df.stack().quantile(.25) ) & ( df < df.stack().quantile(.75) )
    

