<p dir="auto">Hello,</p>
<p dir="auto">It would be nice to have the possibility to do parallel processing using <code class="notranslate">joblib</code> when applying something to a groupby. I'm dreaming about a <code class="notranslate">parallel</code> optional argument which defaults to <code class="notranslate">False</code>.</p>
<p dir="auto">I think it's best to have that parallel argument while grouping</p>
<p dir="auto">The only option I have now is to make list on the groupby results, apply a function on that list using joblib, and concatenate the results. It leads to unefficient and difficult to read code.</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="
############
# Examples #
############
data.groupby(...).fillna(..., parallel=True)
data.groupby(...).apply(..., parallel=True)
data.groupby(...).whatever(..., parallel=True)
data.groupby(...).whatever(...) # not parallel"><pre class="notranslate"><span class="pl-c">############</span>
<span class="pl-c"># Examples #</span>
<span class="pl-c">############</span>
<span class="pl-s1">data</span>.<span class="pl-en">groupby</span>(...).<span class="pl-en">fillna</span>(..., <span class="pl-s1">parallel</span><span class="pl-c1">=</span><span class="pl-c1">True</span>)
<span class="pl-s1">data</span>.<span class="pl-en">groupby</span>(...).<span class="pl-en">apply</span>(..., <span class="pl-s1">parallel</span><span class="pl-c1">=</span><span class="pl-c1">True</span>)
<span class="pl-s1">data</span>.<span class="pl-en">groupby</span>(...).<span class="pl-en">whatever</span>(..., <span class="pl-s1">parallel</span><span class="pl-c1">=</span><span class="pl-c1">True</span>)
<span class="pl-s1">data</span>.<span class="pl-en">groupby</span>(...).<span class="pl-en">whatever</span>(...) <span class="pl-c"># not parallel</span></pre></div>
<p dir="auto">xref <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="24587639" data-permission-text="Title is private" data-url="https://github.com/pandas-dev/pandas/issues/5751" data-hovercard-type="issue" data-hovercard-url="/pandas-dev/pandas/issues/5751/hovercard" href="https://github.com/pandas-dev/pandas/issues/5751">#5751</a></p>
<p dir="auto"><a href="http://stackoverflow.com/questions/37078880/status-of-parallelization-of-pandas-apply?noredirect=1#comment61713215_37078880" rel="nofollow">questions from SO</a>.<br>
<a href="http://stackoverflow.com/questions/31361721/python-dask-dataframe-support-for-trivially-parallelizable-row-apply" rel="nofollow">mrocklins nice example of using .apply</a></p>
<p dir="auto">So here is an example of how to do a parallel apply using <a href="https://dask.readthedocs.io/en/latest/" rel="nofollow">dask</a>. This could be baked into <code class="notranslate">.apply()</code> in pandas by the following signature enhancement:</p>
<p dir="auto">current:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="DataFrame.apply(self, func, axis=0, broadcast=False, raw=False, reduce=None, 
                args=(), **kwds)"><pre class="notranslate"><code class="notranslate">DataFrame.apply(self, func, axis=0, broadcast=False, raw=False, reduce=None, 
                args=(), **kwds)
</code></pre></div>
<p dir="auto">proposed:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="DataFrame.apply(self, func, axis=0, broadcast=False, raw=False, reduce=None, 
                engine=None, chunksize=None, args=(), **kwds)"><pre class="notranslate"><code class="notranslate">DataFrame.apply(self, func, axis=0, broadcast=False, raw=False, reduce=None, 
                engine=None, chunksize=None, args=(), **kwds)
</code></pre></div>
<p dir="auto">where <code class="notranslate">engine='dask'</code> (or <code class="notranslate">numba</code> at some point) are possibilities<br>
<code class="notranslate">chunksize</code> would map directly to <code class="notranslate">npartitions</code> and default to the number of cores if not specified.<br>
further would allow <code class="notranslate">engine</code> to be a meta object like <code class="notranslate">Dask(scheduler='multiprocessing')</code> to support other options one would commonly pass (could also move <code class="notranslate">chunksize</code> inside that instead of as a separate object).</p>
<p dir="auto">impl and timings:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="from functools import partial
import pandas as pd
import dask
import dask.dataframe as dd
from dask import threaded, multiprocessing
from time import sleep

pd.__version__
dask.__version__

def make_frame(N):
    return pd.DataFrame({'A' : range(N)})
def slow_func(x):
    sleep(0.5)
    return x
df = make_frame(40)

# reg apply
def f1(df):
    return df.apply(slow_func, axis=1)
# dask apply
def f2(df, get):
    ddf = dd.from_pandas(df, npartitions=8, sort=False)
    return ddf.apply(slow_func, columns=df.columns, axis=1).compute(get=get)

f1 = partial(f1, df)
f2_threaded = partial(f2, df, threaded.get)
f2_multi = partial(f2, df, multiprocessing.get)

result1 = f1()
result2 = f2_threaded()
result3 = f2_multi()"><pre class="notranslate"><code class="notranslate">from functools import partial
import pandas as pd
import dask
import dask.dataframe as dd
from dask import threaded, multiprocessing
from time import sleep

pd.__version__
dask.__version__

def make_frame(N):
    return pd.DataFrame({'A' : range(N)})
def slow_func(x):
    sleep(0.5)
    return x
df = make_frame(40)

# reg apply
def f1(df):
    return df.apply(slow_func, axis=1)
# dask apply
def f2(df, get):
    ddf = dd.from_pandas(df, npartitions=8, sort=False)
    return ddf.apply(slow_func, columns=df.columns, axis=1).compute(get=get)

f1 = partial(f1, df)
f2_threaded = partial(f2, df, threaded.get)
f2_multi = partial(f2, df, multiprocessing.get)

result1 = f1()
result2 = f2_threaded()
result3 = f2_multi()
</code></pre></div>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="In [18]: result1.equals(result2)
Out[18]: True

In [19]: result1.equals(result3)
Out[19]: True

In [20]: %timeit -n 1 -r 1 f1()
1 loop, best of 1: 20.6 s per loop

In [21]: %timeit -n 1 -r 1 f2_threaded()
1 loop, best of 1: 3.03 s per loop

In [22]: %timeit -n 1 -r 1 f2_multi()
1 loop, best of 1: 3.07 s per loop"><pre class="notranslate"><code class="notranslate">In [18]: result1.equals(result2)
Out[18]: True

In [19]: result1.equals(result3)
Out[19]: True

In [20]: %timeit -n 1 -r 1 f1()
1 loop, best of 1: 20.6 s per loop

In [21]: %timeit -n 1 -r 1 f2_threaded()
1 loop, best of 1: 3.03 s per loop

In [22]: %timeit -n 1 -r 1 f2_multi()
1 loop, best of 1: 3.07 s per loop
</code></pre></div>
<p dir="auto">Now for some caveats.</p>
<p dir="auto">People want to parallelize a poor implementation. Generally you proceed thru the following steps first:</p>
<ul dir="auto">
<li>get your problem correct; optimizing incorrect results is useless</li>
<li>profile profile profile. This is always the first thing to do</li>
<li>use built-in pandas / numpy vectorized routines</li>
<li>use <code class="notranslate">cython</code> or <code class="notranslate">numba</code> on the user defined function</li>
<li><code class="notranslate">.apply</code> is always the last choice</li>
<li>if its still not enough, parallelizaton.</li>
</ul>
<p dir="auto">You always want to make code simpler, not more complex. Its hard to know a-priori where bottlenecks are. People think <code class="notranslate">.apply</code> is some magical thing, its NOT, its JUST A FOR LOOP. The problem is people tend to throw in the kitchen sink, and just everything, which is just a terrible idea.</p>
<p dir="auto">Ok my 2c about optimizing things.</p>
<p dir="auto">In order for parallelization to actually matter the function you are computing should take some non-trivial amount of time to things like:</p>
<ul dir="auto">
<li>iteration costs of the loop</li>
<li>serialization time (esp if using multi-processing / distributed computing)</li>
<li>does the function release the GIL if not, then threading will probably not help much</li>
<li>development resources (your time)</li>
</ul>
<p dir="auto">If these criteria are met, then sure give it a try.</p>
<p dir="auto">I think providing pandas a first class way to parallelize things, even tough people will just naively use it is probably not a bad thing.</p>
<p dir="auto">Further extensions to this are: <code class="notranslate">to_dask()</code> (return a <code class="notranslate">dask.dataframe</code> to the user directly), and <code class="notranslate">engine='dask'</code> syntax for <code class="notranslate">.groupby()</code></p>