<p dir="auto">Hi everyone,</p>
<p dir="auto">I've found a nasty memory leak while using <code class="notranslate">np.random.shuffle</code>:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import numpy as np

@profile
def shuffle_leaktest():
    x = np.random.randn(1000, 1000)
    for ii in xrange(1000):
        np.random.shuffle(x)
    del x

if __name__ == '__main__':
    print &quot;numpy version: &quot; + np.__version__
    shuffle_leaktest()"><pre class="notranslate"><code class="notranslate">import numpy as np

@profile
def shuffle_leaktest():
    x = np.random.randn(1000, 1000)
    for ii in xrange(1000):
        np.random.shuffle(x)
    del x

if __name__ == '__main__':
    print "numpy version: " + np.__version__
    shuffle_leaktest()
</code></pre></div>
<p dir="auto"><code class="notranslate">$ python -m memory_profiler shuffle_leaktest.py</code></p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="numpy version: 1.9.0.dev-fae89b0
Filename: shuffle_leaktest.py

Line #    Mem usage    Increment   Line Contents
================================================
     3  110.535 MiB    0.000 MiB   @profile
     4                             def shuffle_leaktest():
     5  118.195 MiB    7.660 MiB       x = np.random.randn(1000, 1000)
     6  337.770 MiB  219.574 MiB       for ii in xrange(1000):
     7  337.770 MiB    0.000 MiB           np.random.shuffle(x)
     8  337.770 MiB    0.000 MiB       del x"><pre class="notranslate"><code class="notranslate">numpy version: 1.9.0.dev-fae89b0
Filename: shuffle_leaktest.py

Line #    Mem usage    Increment   Line Contents
================================================
     3  110.535 MiB    0.000 MiB   @profile
     4                             def shuffle_leaktest():
     5  118.195 MiB    7.660 MiB       x = np.random.randn(1000, 1000)
     6  337.770 MiB  219.574 MiB       for ii in xrange(1000):
     7  337.770 MiB    0.000 MiB           np.random.shuffle(x)
     8  337.770 MiB    0.000 MiB       del x
</code></pre></div>
<p dir="auto">Using <code class="notranslate">git bisect</code> I've tracked down the first instance of memory leakage to commit <a class="commit-link" data-hovercard-type="commit" data-hovercard-url="https://github.com/numpy/numpy/commit/607863d7387e80dab4b064856b598a8c86e9bee4/hovercard" href="https://github.com/numpy/numpy/commit/607863d7387e80dab4b064856b598a8c86e9bee4"><tt>607863d</tt></a>.</p>
<p dir="auto">No leak:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content=" numpy version: 1.9.0.dev-c7a30d5
 Filename: shuffle_leaktest.py

 Line #    Mem usage    Increment   Line Contents
 ================================================
      3  103.633 MiB    0.000 MiB   @profile
      4                             def shuffle_leaktest():
      5  111.285 MiB    7.652 MiB       x = np.random.randn(1000, 1000)
      6  111.305 MiB    0.020 MiB       for ii in xrange(1000):
      7  111.305 MiB    0.000 MiB           np.random.shuffle(x)
      8  103.672 MiB   -7.633 MiB       del x"><pre class="notranslate"><code class="notranslate"> numpy version: 1.9.0.dev-c7a30d5
 Filename: shuffle_leaktest.py

 Line #    Mem usage    Increment   Line Contents
 ================================================
      3  103.633 MiB    0.000 MiB   @profile
      4                             def shuffle_leaktest():
      5  111.285 MiB    7.652 MiB       x = np.random.randn(1000, 1000)
      6  111.305 MiB    0.020 MiB       for ii in xrange(1000):
      7  111.305 MiB    0.000 MiB           np.random.shuffle(x)
      8  103.672 MiB   -7.633 MiB       del x
</code></pre></div>
<p dir="auto">Leak:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content=" numpy version: 1.9.0.dev-607863d
 Filename: shuffle_leaktest.py

 Line #    Mem usage    Increment   Line Contents
 ================================================
      3  112.664 MiB    0.000 MiB   @profile
      4                             def shuffle_leaktest():
      5  120.316 MiB    7.652 MiB       x = np.random.randn(1000, 1000)
      6  183.031 MiB   62.715 MiB       for ii in xrange(1000):
      7  183.031 MiB    0.000 MiB           np.random.shuffle(x)
      8  175.398 MiB   -7.633 MiB       del x"><pre class="notranslate"><code class="notranslate"> numpy version: 1.9.0.dev-607863d
 Filename: shuffle_leaktest.py

 Line #    Mem usage    Increment   Line Contents
 ================================================
      3  112.664 MiB    0.000 MiB   @profile
      4                             def shuffle_leaktest():
      5  120.316 MiB    7.652 MiB       x = np.random.randn(1000, 1000)
      6  183.031 MiB   62.715 MiB       for ii in xrange(1000):
      7  183.031 MiB    0.000 MiB           np.random.shuffle(x)
      8  175.398 MiB   -7.633 MiB       del x
</code></pre></div>
<p dir="auto">However, this leak seems to be much less pronounced than the one I'm currently seeing in master. It seems that things suddenly got a lot worse with commit <a class="commit-link" data-hovercard-type="commit" data-hovercard-url="https://github.com/numpy/numpy/commit/9f8771accdc11a83dc928a99bd0ba48fe7bcca89/hovercard" href="https://github.com/numpy/numpy/commit/9f8771accdc11a83dc928a99bd0ba48fe7bcca89"><tt>9f8771a</tt></a>:</p>
<p dir="auto">Small leak:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content=" numpy version: 1.9.0.dev-f57c77b
 Filename: shuffle_leaktest.py

 Line #    Mem usage    Increment   Line Contents
 ================================================
      3   87.219 MiB    0.000 MiB   @profile
      4                             def shuffle_leaktest():
      5   94.871 MiB    7.652 MiB       x = np.random.randn(1000, 1000)
      6  157.789 MiB   62.918 MiB       for ii in xrange(1000):
      7  157.789 MiB    0.000 MiB           np.random.shuffle(x)
      8  150.156 MiB   -7.633 MiB       del x"><pre class="notranslate"><code class="notranslate"> numpy version: 1.9.0.dev-f57c77b
 Filename: shuffle_leaktest.py

 Line #    Mem usage    Increment   Line Contents
 ================================================
      3   87.219 MiB    0.000 MiB   @profile
      4                             def shuffle_leaktest():
      5   94.871 MiB    7.652 MiB       x = np.random.randn(1000, 1000)
      6  157.789 MiB   62.918 MiB       for ii in xrange(1000):
      7  157.789 MiB    0.000 MiB           np.random.shuffle(x)
      8  150.156 MiB   -7.633 MiB       del x
</code></pre></div>
<p dir="auto">Big leak:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content=" numpy version: 1.9.0.dev-9f8771a
 Filename: shuffle_leaktest.py

 Line #    Mem usage    Increment   Line Contents
 ================================================
      3  118.914 MiB    0.000 MiB   @profile
      4                             def shuffle_leaktest():
      5  126.566 MiB    7.652 MiB       x = np.random.randn(1000, 1000)
      6  346.133 MiB  219.566 MiB       for ii in xrange(1000):
      7  346.133 MiB    0.000 MiB           np.random.shuffle(x)
      8  346.133 MiB    0.000 MiB       del x"><pre class="notranslate"><code class="notranslate"> numpy version: 1.9.0.dev-9f8771a
 Filename: shuffle_leaktest.py

 Line #    Mem usage    Increment   Line Contents
 ================================================
      3  118.914 MiB    0.000 MiB   @profile
      4                             def shuffle_leaktest():
      5  126.566 MiB    7.652 MiB       x = np.random.randn(1000, 1000)
      6  346.133 MiB  219.566 MiB       for ii in xrange(1000):
      7  346.133 MiB    0.000 MiB           np.random.shuffle(x)
      8  346.133 MiB    0.000 MiB       del x
</code></pre></div>
<p dir="auto">Hi everyone,</p>
<p dir="auto">I found a memory leak writing code using the <a href="http://deeplearning.net/software/theano/" rel="nofollow">theano</a> module and I was able to trace it back to numpy.</p>
<p dir="auto"><a href="https://groups.google.com/forum/#!topic/theano-dev/3gI_S9ns9oE" rel="nofollow">I already posted on the theano mailing list about it</a></p>
<p dir="auto">Sample code to reproduce the leak:</p>
<p dir="auto">Required modules:</p>
<ul dir="auto">
<li>Numpy (<a href="https://github.com/numpy/numpy/commit/0c9f285f38ea4d143c5e79badd0d36cb808242a6">commit</a>: <code class="notranslate">0c9f285f38ea4d143c5e79badd0d36cb808242a6</code>)</li>
<li>Scipy (<a href="https://github.com/scipy/scipy/commit/d9a8c214c4cea75f7d240957a646b562cf081f8b">commit</a>: <code class="notranslate">d9a8c214c4cea75f7d240957a646b562cf081f8b</code>)</li>
<li>Theano (<a href="https://github.com/Theano/Theano/commit/26d913091e56b3e80e7c562175b856a90dd782e4">commit</a>: <code class="notranslate">26d913091e56b3e80e7c562175b856a90dd782e4</code>)</li>
<li>for "<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/Profile/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/Profile">@Profile</a>" decorator: memory_profiler and psutils</li>
<li>I also use OpenBlas</li>
</ul>
<div class="highlight highlight-source-ruby notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import theano
from theano import tensor as T
import numpy as np

k = T.iscalar(&quot;k&quot;)
A = T.vector(&quot;A&quot;)

# Symbolic description of the result
result, updates = theano.scan(fn=lambda prior_result, A: prior_result * A,
                              outputs_info=T.ones_like(A),
                              non_sequences=A,
                              n_steps=k)

print &quot;Compiling function&quot;
power = theano.function(inputs=[A,k], outputs=result, updates=updates)
print &quot;Compiling function - done&quot;

ip = np.arange(10000)
x = None
print &quot;Starting loop&quot;
@profile
def test():
    for i in xrange(10000):
        x = power(ip,6)
test()
print &quot;Done.&quot;"><pre class="notranslate"><span class="pl-en">import</span> <span class="pl-en">theano</span>
<span class="pl-en">from</span> <span class="pl-en">theano</span> <span class="pl-en">import</span> <span class="pl-en">tensor</span> <span class="pl-en">as</span> <span class="pl-c1">T</span>
<span class="pl-en">import</span> <span class="pl-en">numpy</span> <span class="pl-en">as</span> <span class="pl-en">np</span>

<span class="pl-s1">k</span> <span class="pl-c1">=</span> <span class="pl-c1">T</span><span class="pl-kos">.</span><span class="pl-en">iscalar</span><span class="pl-kos">(</span><span class="pl-s">"k"</span><span class="pl-kos">)</span>
<span class="pl-c1">A</span> <span class="pl-c1">=</span> <span class="pl-c1">T</span><span class="pl-kos">.</span><span class="pl-en">vector</span><span class="pl-kos">(</span><span class="pl-s">"A"</span><span class="pl-kos">)</span>

<span class="pl-c"># Symbolic description of the result</span>
<span class="pl-s1">result</span><span class="pl-kos">,</span> <span class="pl-s1">updates</span> <span class="pl-c1">=</span> <span class="pl-en">theano</span><span class="pl-kos">.</span><span class="pl-en">scan</span><span class="pl-kos">(</span><span class="pl-s1">fn</span><span class="pl-c1">=</span><span class="pl-en">lambda</span> <span class="pl-en">prior_result</span><span class="pl-kos">,</span> <span class="pl-pds">A</span>: <span class="pl-en">prior_result</span> * <span class="pl-c1">A</span><span class="pl-kos">,</span>
                              <span class="pl-s1">outputs_info</span><span class="pl-c1">=</span><span class="pl-c1">T</span><span class="pl-kos">.</span><span class="pl-en">ones_like</span><span class="pl-kos">(</span><span class="pl-c1">A</span><span class="pl-kos">)</span><span class="pl-kos">,</span>
                              <span class="pl-s1">non_sequences</span><span class="pl-c1">=</span><span class="pl-c1">A</span><span class="pl-kos">,</span>
                              <span class="pl-s1">n_steps</span><span class="pl-c1">=</span><span class="pl-s1">k</span><span class="pl-kos">)</span>

<span class="pl-en">print</span> <span class="pl-s">"Compiling function"</span>
<span class="pl-s1">power</span> <span class="pl-c1">=</span> <span class="pl-en">theano</span><span class="pl-kos">.</span><span class="pl-en">function</span><span class="pl-kos">(</span><span class="pl-s1">inputs</span><span class="pl-c1">=</span><span class="pl-kos">[</span><span class="pl-c1">A</span><span class="pl-kos">,</span><span class="pl-s1">k</span><span class="pl-kos">]</span><span class="pl-kos">,</span> <span class="pl-s1">outputs</span><span class="pl-c1">=</span><span class="pl-s1">result</span><span class="pl-kos">,</span> <span class="pl-s1">updates</span><span class="pl-c1">=</span><span class="pl-s1">updates</span><span class="pl-kos">)</span>
<span class="pl-en">print</span> <span class="pl-s">"Compiling function - done"</span>

<span class="pl-s1">ip</span> <span class="pl-c1">=</span> <span class="pl-en">np</span><span class="pl-kos">.</span><span class="pl-en">arange</span><span class="pl-kos">(</span><span class="pl-c1">10000</span><span class="pl-kos">)</span>
<span class="pl-s1">x</span> <span class="pl-c1">=</span> <span class="pl-v">None</span>
<span class="pl-en">print</span> <span class="pl-s">"Starting loop"</span>
<span class="pl-c1">@profile</span>
<span class="pl-k">def</span> <span class="pl-en">test</span><span class="pl-kos">(</span><span class="pl-kos">)</span>:
    <span class="pl-k">for</span> <span class="pl-en">i</span> <span class="pl-k">in</span> <span class="pl-en">xrange</span><span class="pl-kos">(</span><span class="pl-c1">10000</span><span class="pl-kos">)</span>:
        <span class="pl-s1">x</span> <span class="pl-c1">=</span> <span class="pl-s1">power</span><span class="pl-kos">(</span><span class="pl-s1">ip</span><span class="pl-kos">,</span><span class="pl-c1">6</span><span class="pl-kos">)</span>
<span class="pl-en">test</span><span class="pl-kos">(</span><span class="pl-kos">)</span>
<span class="pl-en">print</span> <span class="pl-s">"Done."</span></pre></div>
<p dir="auto">For more logs:<br>
<a href="https://gist.github.com/ogh/b6694c8eb7793b454dee">https://gist.github.com/ogh/b6694c8eb7793b454dee</a></p>
<p dir="auto">With the same configuration except for numpy switched out with a version from last September everything works fine without any leak.<br>
(tried the November version since I was using it on a server where I did not have the memory leak)</p>
<p dir="auto">The version of numpy that works without a leak: <a href="https://github.com/numpy/numpy/tree/135443768a24ab4fbfd4fa5c8fc40f27d2e25c96">commit</a><code class="notranslate">135443768a24ab4fbfd4fa5c8fc40f27d2e25c96</code></p>
<p dir="auto">Can you give me any advice on how to narrow down the problem further?</p>