<p dir="auto">stats.mode seems surprisingly slow, especially since it's very much the same as <code class="notranslate">find_repeats</code>. Despite only needing to return the first value and being pretty much the same function, stats.mode is much slower than find_repeats (which appears to be a very thin wrapper around a fortran function), here's some benchmarking:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import numpy as np
import scipy.stats as stats
arr1 = np.random.randint(-1000, 1000, 10000)
%timeit stats.mode(arr1)
%timeit stats.find_repeats(arr1)"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">numpy</span> <span class="pl-k">as</span> <span class="pl-s1">np</span>
<span class="pl-k">import</span> <span class="pl-s1">scipy</span>.<span class="pl-s1">stats</span> <span class="pl-k">as</span> <span class="pl-s1">stats</span>
<span class="pl-s1">arr1</span> <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-s1">random</span>.<span class="pl-en">randint</span>(<span class="pl-c1">-</span><span class="pl-c1">1000</span>, <span class="pl-c1">1000</span>, <span class="pl-c1">10000</span>)
<span class="pl-c1">%</span><span class="pl-s1">timeit</span> <span class="pl-s1">stats</span>.<span class="pl-en">mode</span>(<span class="pl-s1">arr1</span>)
<span class="pl-c1">%</span><span class="pl-s1">timeit</span> <span class="pl-s1">stats</span>.<span class="pl-en">find_repeats</span>(<span class="pl-s1">arr1</span>)</pre></div>
<p dir="auto">which yields</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="10 loops, best of 3: 142 ms per loop
1000 loops, best of 3: 832 µs per loop"><pre class="notranslate"><code class="notranslate">10 loops, best of 3: 142 ms per loop
1000 loops, best of 3: 832 µs per loop
</code></pre></div>
<p dir="auto">The problem is that the implementation is very slow when the number of unique labels is much greater than the number of unique values, because it iterates over every count:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="arr1 = np.random.randint(-100, 100, 10000)
ser = pd.Series(arr1)
%timeit stats.mode(arr1)
%timeit stats.find_repeats(arr1)"><pre class="notranslate"><span class="pl-s1">arr1</span> <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-s1">random</span>.<span class="pl-en">randint</span>(<span class="pl-c1">-</span><span class="pl-c1">100</span>, <span class="pl-c1">100</span>, <span class="pl-c1">10000</span>)
<span class="pl-s1">ser</span> <span class="pl-c1">=</span> <span class="pl-s1">pd</span>.<span class="pl-v">Series</span>(<span class="pl-s1">arr1</span>)
<span class="pl-c1">%</span><span class="pl-s1">timeit</span> <span class="pl-s1">stats</span>.<span class="pl-en">mode</span>(<span class="pl-s1">arr1</span>)
<span class="pl-c1">%</span><span class="pl-s1">timeit</span> <span class="pl-s1">stats</span>.<span class="pl-en">find_repeats</span>(<span class="pl-s1">arr1</span>)</pre></div>
<p dir="auto">yields</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="100 loops, best of 3: 15.2 ms per loop
1000 loops, best of 3: 681 µs per loop"><pre class="notranslate"><code class="notranslate">100 loops, best of 3: 15.2 ms per loop
1000 loops, best of 3: 681 µs per loop
</code></pre></div>
<p dir="auto">And is still twice as slow with 20 uniques vs. 10000 entries:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="arr1 = np.random.randint(-10, 10, 10000)
%timeit stats.mode(arr1)
%timeit stats.find_repeats(arr1)"><pre class="notranslate"><code class="notranslate">arr1 = np.random.randint(-10, 10, 10000)
%timeit stats.mode(arr1)
%timeit stats.find_repeats(arr1)
</code></pre></div>
<p dir="auto">yielding</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="1000 loops, best of 3: 1.89 ms per loop
1000 loops, best of 3: 508 µs per loop"><pre class="notranslate"><code class="notranslate">1000 loops, best of 3: 1.89 ms per loop
1000 loops, best of 3: 508 µs per loop
</code></pre></div>
<p dir="auto">And still not great with 2 dimensions:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="arr1 = np.random.randint(-100, 100, (500, 100))
%timeit stats.mode(arr1)
%timeit stats.find_repeats(arr1)
# approximate finding repeats on each axis:
%timeit [stats.find_repeats(arr1[i]) for i in range(len(arr1))]"><pre class="notranslate"><span class="pl-s1">arr1</span> <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-s1">random</span>.<span class="pl-en">randint</span>(<span class="pl-c1">-</span><span class="pl-c1">100</span>, <span class="pl-c1">100</span>, (<span class="pl-c1">500</span>, <span class="pl-c1">100</span>))
<span class="pl-c1">%</span><span class="pl-s1">timeit</span> <span class="pl-s1">stats</span>.<span class="pl-en">mode</span>(<span class="pl-s1">arr1</span>)
<span class="pl-c1">%</span><span class="pl-s1">timeit</span> <span class="pl-s1">stats</span>.<span class="pl-en">find_repeats</span>(<span class="pl-s1">arr1</span>)
<span class="pl-c"># approximate finding repeats on each axis:</span>
<span class="pl-c1">%</span><span class="pl-s1">timeit</span> [<span class="pl-s1">stats</span>.<span class="pl-en">find_repeats</span>(<span class="pl-s1">arr1</span>[<span class="pl-s1">i</span>]) <span class="pl-s1">for</span> <span class="pl-s1">i</span> <span class="pl-c1">in</span> <span class="pl-en">range</span>(<span class="pl-en">len</span>(<span class="pl-s1">arr1</span>))]</pre></div>
<p dir="auto">which yields:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="10 loops, best of 3: 50.6 ms per loop
100 loops, best of 3: 3.6 ms per loop
100 loops, best of 3: 4.61 ms per loop"><pre class="notranslate"><code class="notranslate">10 loops, best of 3: 50.6 ms per loop
100 loops, best of 3: 3.6 ms per loop
100 loops, best of 3: 4.61 ms per loop
</code></pre></div>
<p dir="auto">At least for 1D, a better implementation would be to take advantage of find_repeats (or reuse its internals) rather than iterating over uniques, e.g. something like this for 1D:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="def faster_mode1D(a):
    arr = np.asarray(a) # would be _chk_array
    v, c = stats.find_repeats(arr)
    if len(c) == 0:
        arr.sort() # mimic first value behavior
        return arr[0], 1.
    else:
        pos = c.argmax()
        return v[pos], c[pos]"><pre class="notranslate"><span class="pl-k">def</span> <span class="pl-en">faster_mode1D</span>(<span class="pl-s1">a</span>):
    <span class="pl-s1">arr</span> <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-en">asarray</span>(<span class="pl-s1">a</span>) <span class="pl-c"># would be _chk_array</span>
    <span class="pl-s1">v</span>, <span class="pl-s1">c</span> <span class="pl-c1">=</span> <span class="pl-s1">stats</span>.<span class="pl-en">find_repeats</span>(<span class="pl-s1">arr</span>)
    <span class="pl-k">if</span> <span class="pl-en">len</span>(<span class="pl-s1">c</span>) <span class="pl-c1">==</span> <span class="pl-c1">0</span>:
        <span class="pl-s1">arr</span>.<span class="pl-en">sort</span>() <span class="pl-c"># mimic first value behavior</span>
        <span class="pl-k">return</span> <span class="pl-s1">arr</span>[<span class="pl-c1">0</span>], <span class="pl-c1">1.</span>
    <span class="pl-k">else</span>:
        <span class="pl-s1">pos</span> <span class="pl-c1">=</span> <span class="pl-s1">c</span>.<span class="pl-en">argmax</span>()
        <span class="pl-k">return</span> <span class="pl-s1">v</span>[<span class="pl-s1">pos</span>], <span class="pl-s1">c</span>[<span class="pl-s1">pos</span>]</pre></div>
<p dir="auto">Not sure whether this makes as much sense for multiple dimensions, but maybe it would be worth special-casing 1D ndarrays?  The performance difference is pretty big.</p>
<p dir="auto"><em>Original ticket <a href="http://projects.scipy.org/scipy/ticket/905" rel="nofollow">http://projects.scipy.org/scipy/ticket/905</a> on 2009-03-31 by trac user parejkoj, assigned to unknown.</em></p>
<p dir="auto">I'm trying to run scipy.stats.mode (in scipy.stats.stats) on a 2-d ndarray, but it takes an extremely long time.  The data is an image (astronomical data), with:</p>
<p dir="auto">data.shape = (227,3104)<br>
data.dtype = ('float32')</p>
<p dir="auto">A quick-and-dirty mode function like this:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="def mode(data):
    counts = {}
    for x in data.flatten():
    counts[x] = counts.get(x,0) + 1
    maxcount = max(counts.values())
    modelist = []
    for x in counts:
    if counts[x] == maxcount:
        modelist.append(x)
    return modelist,maxcount
#..."><pre class="notranslate"><code class="notranslate">def mode(data):
    counts = {}
    for x in data.flatten():
    counts[x] = counts.get(x,0) + 1
    maxcount = max(counts.values())
    modelist = []
    for x in counts:
    if counts[x] == maxcount:
        modelist.append(x)
    return modelist,maxcount
#...
</code></pre></div>
<p dir="auto">takes only a second or so to run, while stats.mode(data) or stats.mode(data.flatten()) takes ~20 minutes, no matter how I set the axis parameter.  Am I missing something, or shouldn't it be significantly faster?</p>
<p dir="auto">I'm using scipy 0.6.0 from Fink on Mac OS X Leopard with python2.5, but there appear to be no changes to the mode function between 0.6.0 and the current svn.</p>