What am I getting wrong in my pandas lambda map?
Question: I'm trying to find the percentile of a dataframe which observations in a
second dataframe would belong to, and I thought a lambda function would do the
trick here like so:

    
    
    df1.var1.map(lambda x: np.percentile(df2.var1, x))
    

which I read as for each `x` in the series `df1.var1`, apply the function
`np.percentile(df2.var1, x)`, which finds the percentile of `x` in the series
`df2.var1`. For some reason, I'm getting the error

    
    
    kth(=-9223372036854775599) out of bounds (209)
    

where 209 is the length of `df2`, but I have no idea what the `kth` part
refers to. Any ideas what I'm doing wrong here?

FULL ERROR:

    
    
    ValueError                                Traceback (most recent call last)
    <ipython-input-82-02d5cacfecd4> in <module>()
    ----> 1 df1.var1.map(lambda x: np.percentile(df2.var1, x))
       C:\Users\ngudat\AppData\Local\Continuum\Anaconda\lib\site-packages\pandas\core\series.pyc in map(self, arg, na_action)
       2043                                      index=self.index).__finalize__ (self)
       2044         else:
    -> 2045             mapped = map_f(values, arg)
       2046             return self._constructor(mapped,
       2047                                      index=self.index).__finalize__(self)
    
    pandas\src\inference.pyx in pandas.lib.map_infer (pandas\lib.c:62187)()
    
    <ipython-input-82-02d5cacfecd4> in <lambda>(x)
    ----> 1 df.qof.map(lambda x: np.percentile(prac_prof.qof, x))
    
    C:\Users\ngudat\AppData\Local\Continuum\Anaconda\lib\site-packages\numpy\lib\function_base.pyc in percentile(a, q, axis, out, overwrite_input, interpolation, keepdims)
       3266     r, k = _ureduce(a, func=_percentile, q=q, axis=axis, out=out,
       3267                     overwrite_input=overwrite_input,
    -> 3268                     interpolation=interpolation)
       3269     if keepdims:
       3270         if q.ndim == 0:
    
    C:\Users\ngudat\AppData\Local\Continuum\Anaconda\lib\site-packages\numpy\lib\function_base.pyc in _ureduce(a, func, **kwargs)
       2995         keepdim = [1] * a.ndim
       2996 
    -> 2997     r = func(a, **kwargs)
       2998     return r, keepdim
       2999 
    
    C:\Users\ngudat\AppData\Local\Continuum\Anaconda\lib\site-packages\numpy\lib\function_base.pyc in _percentile(a, q, axis, out, overwrite_input, interpolation, keepdims)
       3370         weights_above.shape = weights_shape
       3371 
    -> 3372         ap.partition(concatenate((indices_below, indices_above)),axis=axis)
       3373 
       3374         # ensure axis with qth is first
    
    ValueError: kth(=-9223372036854775599) out of bounds (209)
    


Answer: Percentile will not give you what you need here, it takes a percentile and
gives you the value. You need the opposite. You should rank the entries in the
column and calculate the percentiles from that:

    
    
    import pandas as pd
    aa = [1,3,2,4,11,8,9]
    dd = pd.DataFrame(data=aa,columns=['xx'])
    dd['rank']=dd['xx'].rank()
    dd['percentile'] = dd['rank']/len(dd)
    

This gives you the percentile corresponding to each entry:

    
    
       xx  rank  percentile
    0   1     1    0.142857
    1   3     3    0.428571
    2   2     2    0.285714
    3   4     4    0.571429
    4  11     7    1.000000
    5   8     5    0.714286
    6   9     6    0.857143
    

