<p dir="auto">permutation and shuffle functions in numpy.random leak memory. On different test machines it either caused MemoryError, script termination with SIGKILL from OS, or produced no errors but the script consumed huge amount of  memory (~ 5Gb).</p>
<h3 dir="auto">Reproducing code example:</h3>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import numpy as np

for k in range(1000):
    ind = np.random.permutation(44100 * 1000)"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">numpy</span> <span class="pl-k">as</span> <span class="pl-s1">np</span>

<span class="pl-k">for</span> <span class="pl-s1">k</span> <span class="pl-c1">in</span> <span class="pl-en">range</span>(<span class="pl-c1">1000</span>):
    <span class="pl-s1">ind</span> <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-s1">random</span>.<span class="pl-en">permutation</span>(<span class="pl-c1">44100</span> <span class="pl-c1">*</span> <span class="pl-c1">1000</span>)</pre></div>
<p dir="auto">or</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import numpy as np

for k in range(1000):
    x = np.empty(44100 * 1000)
    np.random.shuffle(x)"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">numpy</span> <span class="pl-k">as</span> <span class="pl-s1">np</span>

<span class="pl-k">for</span> <span class="pl-s1">k</span> <span class="pl-c1">in</span> <span class="pl-en">range</span>(<span class="pl-c1">1000</span>):
    <span class="pl-s1">x</span> <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-en">empty</span>(<span class="pl-c1">44100</span> <span class="pl-c1">*</span> <span class="pl-c1">1000</span>)
    <span class="pl-s1">np</span>.<span class="pl-s1">random</span>.<span class="pl-en">shuffle</span>(<span class="pl-s1">x</span>)</pre></div>
<p dir="auto">Start <code class="notranslate">top</code> and watch the memory grow.</p>
<h3 dir="auto">Error message:</h3>
<p dir="auto">Traceback (most recent call last):<br>
File "leak_memory.py", line 5, in <br>
ind = np.random.permutation(44100 * 1000)<br>
File "mtrand.pyx", line 4917, in mtrand.RandomState.permutation<br>
MemoryError</p>
<h3 dir="auto">Numpy/Python version information:</h3>
<p dir="auto">1.16.3 3.7.1 (default, Oct 22 2018, 11:21:55)<br>
[GCC 8.2.0]</p>
<p dir="auto">OS: Ubuntu 18.04 (Error message and SIGKILL)<br>
Windows 10 (huge memory consumption)</p>
<p dir="auto">Reading attribute "ctypes.data" from an ndarray causes a leak.<br>
version &lt;= 1.14.5 doesn't leak<br>
version &gt;= 1.16.2 leaks<br>
latest 1.16.4 leaks</p>
<h3 dir="auto">Reproducing code example:</h3>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import numpy as np
import sys

print(np.version.full_version)

data = np.empty(8192)

before = sys.getrefcount(data)
data.ctypes.data
after = sys.getrefcount(data)
print(before, after)

assert before==after"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">numpy</span> <span class="pl-k">as</span> <span class="pl-s1">np</span>
<span class="pl-k">import</span> <span class="pl-s1">sys</span>

<span class="pl-en">print</span>(<span class="pl-s1">np</span>.<span class="pl-s1">version</span>.<span class="pl-s1">full_version</span>)

<span class="pl-s1">data</span> <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-en">empty</span>(<span class="pl-c1">8192</span>)

<span class="pl-s1">before</span> <span class="pl-c1">=</span> <span class="pl-s1">sys</span>.<span class="pl-en">getrefcount</span>(<span class="pl-s1">data</span>)
<span class="pl-s1">data</span>.<span class="pl-s1">ctypes</span>.<span class="pl-s1">data</span>
<span class="pl-s1">after</span> <span class="pl-c1">=</span> <span class="pl-s1">sys</span>.<span class="pl-en">getrefcount</span>(<span class="pl-s1">data</span>)
<span class="pl-en">print</span>(<span class="pl-s1">before</span>, <span class="pl-s1">after</span>)

<span class="pl-k">assert</span> <span class="pl-s1">before</span><span class="pl-c1">==</span><span class="pl-s1">after</span></pre></div>
<h3 dir="auto">Error message:</h3>
<p dir="auto">File "/mnt/d/code/python/test_numpy_leak.py", line 17, in <br>
assert before==after<br>
AssertionError</p>
<h3 dir="auto">Numpy/Python version information:</h3>
<p dir="auto">1.16.4 3.6.8 (default, Jan 14 2019, 11:02:34)<br>
[GCC 8.0.1 20180414 (experimental) [trunk revision 259383]</p>
<hr>
<p dir="auto">Summary (<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/seberg/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/seberg">@seberg</a>) 2019-12-03</p>
<p dir="auto">This has been partially fixed by <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="491558213" data-permission-text="Title is private" data-url="https://github.com/numpy/numpy/issues/14469" data-hovercard-type="pull_request" data-hovercard-url="/numpy/numpy/pull/14469/hovercard" href="https://github.com/numpy/numpy/pull/14469">gh-14469</a>; There are still issues here, although they are mainly issues with upstream ctypes creating deep reference cycles.</p>