<p dir="auto">Impossible to do parallel numpy computations using python subprocesses,<br>
because just simple import numpy requires ~1Gb paging file</p>
<p dir="auto">I have 32Gb ram and 4Gb paging file<br>
this code causes an error</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import multiprocessing
import time

def proc():
    import numpy
    time.sleep(1000)

if __name__ == '__main__':
    for i in range(50):
        multiprocessing.Process(target=proc).start()

    time.sleep(1000)"><pre class="notranslate"><code class="notranslate">import multiprocessing
import time

def proc():
    import numpy
    time.sleep(1000)

if __name__ == '__main__':
    for i in range(50):
        multiprocessing.Process(target=proc).start()

    time.sleep(1000)
</code></pre></div>
<h3 dir="auto">Error message:</h3>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="  File &quot;D:\DevelopPPP\projects\DeepFaceLive\_internal\python\lib\site-packages\numpy\__init__.py&quot;, line 140, in &lt;module&gt;
    from . import core
  File &quot;D:\DevelopPPP\projects\DeepFaceLive\_internal\python\lib\site-packages\numpy\core\__init__.py&quot;, line 72, in &lt;module&gt;
    from . import numeric
  File &quot;&lt;frozen importlib._bootstrap&gt;&quot;, line 971, in _find_and_load
  File &quot;&lt;frozen importlib._bootstrap&gt;&quot;, line 955, in _find_and_load_unlocked
  File &quot;&lt;frozen importlib._bootstrap&gt;&quot;, line 665, in _load_unlocked
  File &quot;&lt;frozen importlib._bootstrap_external&gt;&quot;, line 674, in exec_module
  File &quot;&lt;frozen importlib._bootstrap_external&gt;&quot;, line 764, in get_code
  File &quot;&lt;frozen importlib._bootstrap_external&gt;&quot;, line 833, in get_data
MemoryError"><pre class="notranslate"><code class="notranslate">  File "D:\DevelopPPP\projects\DeepFaceLive\_internal\python\lib\site-packages\numpy\__init__.py", line 140, in &lt;module&gt;
    from . import core
  File "D:\DevelopPPP\projects\DeepFaceLive\_internal\python\lib\site-packages\numpy\core\__init__.py", line 72, in &lt;module&gt;
    from . import numeric
  File "&lt;frozen importlib._bootstrap&gt;", line 971, in _find_and_load
  File "&lt;frozen importlib._bootstrap&gt;", line 955, in _find_and_load_unlocked
  File "&lt;frozen importlib._bootstrap&gt;", line 665, in _load_unlocked
  File "&lt;frozen importlib._bootstrap_external&gt;", line 674, in exec_module
  File "&lt;frozen importlib._bootstrap_external&gt;", line 764, in get_code
  File "&lt;frozen importlib._bootstrap_external&gt;", line 833, in get_data
MemoryError
</code></pre></div>
<p dir="auto">OS : Windows 10 64-bit</p>
<h3 dir="auto">NumPy/Python version information:</h3>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="1.19.5 3.6.8 (tags/v3.6.8:3c6b436a57, Dec 24 2018, 00:16:47) [MSC v.1916 64 bit (AMD64)]"><pre class="notranslate"><code class="notranslate">1.19.5 3.6.8 (tags/v3.6.8:3c6b436a57, Dec 24 2018, 00:16:47) [MSC v.1916 64 bit (AMD64)]
</code></pre></div>
<h3 dir="auto">Background</h3>
<p dir="auto">We run our compile runs with daemontools to lock virtual memory usage to around 1GB per process (important when running very very parallel builds).  During update from 1.19.4 -&gt; 1.19.5 our nightlies trip over code that uses numpy.</p>
<p dir="auto">Fine with numpy==1.19.4 (1GB):</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="softlimit -a $(expr 1024 \* 1024 \* 1024) make"><pre class="notranslate"><code class="notranslate">softlimit -a $(expr 1024 \* 1024 \* 1024) make
</code></pre></div>
<p dir="auto">Needed to build with numpy==1.19.5 (1.7GB):</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="softlimit -a $(expr 1740 \* 1024 \* 1024) make"><pre class="notranslate"><code class="notranslate">softlimit -a $(expr 1740 \* 1024 \* 1024) make
</code></pre></div>
<p dir="auto">I've whittled it down to the smallest possible reproduction which is simply importing <code class="notranslate">numpy</code> version 1.19.5 with 1GB virtual memory limit.  See "Reproducing code example"</p>
<p dir="auto">To examine further I ratcheted the limit up/down with 1.19.4 and 1.9.5 to find the point at which <code class="notranslate">import</code> succeeds.</p>
<table role="table">
<thead>
<tr>
<th>version</th>
<th>approx minimum import limit</th>
</tr>
</thead>
<tbody>
<tr>
<td>1.19.4</td>
<td><code class="notranslate">softlimit -a 400000000 python3</code></td>
</tr>
<tr>
<td>1.19.5</td>
<td><code class="notranslate">softlimit -a 1200000000 python3</code></td>
</tr>
<tr>
<td>1.20.0rc2</td>
<td><code class="notranslate">softlimit -a 1200000000 python3</code></td>
</tr>
<tr>
<td>1.19.3</td>
<td><code class="notranslate">softlimit -a 1200000000 python3</code></td>
</tr>
<tr>
<td>1.19.2</td>
<td><code class="notranslate">softlimit -a 400000000 python3</code></td>
</tr>
</tbody>
</table>
<p dir="auto">Note this is with nothing but <code class="notranslate">import numpy</code>.  The overhead difference ~800MB matches the limit changes we needed to get builds running.</p>
<h3 dir="auto">Reproducing code example:</h3>
<p dir="auto">You can reproduce this just with loading of the module for version 1.19.5 on ubuntu:20.04 with daemontools.  See below for Dockerfile to set up clean 20.04 to reproduce.</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import numpy as np"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">numpy</span> <span class="pl-k">as</span> <span class="pl-s1">np</span></pre></div>
<p dir="auto">Invoked with <code class="notranslate">softlimit -a 1073741824 python3</code></p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="root@88e4e56b16a1:~# pip3 install numpy==1.19.5
Collecting numpy==1.19.5
  Downloading numpy-1.19.5-cp38-cp38-manylinux2010_x86_64.whl (14.9 MB)
     |████████████████████████████████| 14.9 MB 10.7 MB/s 
Installing collected packages: numpy
  Attempting uninstall: numpy
    Found existing installation: numpy 1.19.4
    Uninstalling numpy-1.19.4:
      Successfully uninstalled numpy-1.19.4
Successfully installed numpy-1.19.5
root@88e4e56b16a1:~# softlimit -a 1073741824 python3
Python 3.8.5 (default, Jul 28 2020, 12:59:40) 
[GCC 9.3.0] on linux
Type &quot;help&quot;, &quot;copyright&quot;, &quot;credits&quot; or &quot;license&quot; for more information.
&gt;&gt;&gt; import numpy
Traceback (most recent call last):
  File &quot;&lt;stdin&gt;&quot;, line 1, in &lt;module&gt;
  File &quot;/usr/local/lib/python3.8/dist-packages/numpy/__init__.py&quot;, line 143, in &lt;module&gt;
    from . import lib
  File &quot;/usr/local/lib/python3.8/dist-packages/numpy/lib/__init__.py&quot;, line 40, in &lt;module&gt;
    from .arraypad import *
  File &quot;/usr/local/lib/python3.8/dist-packages/numpy/lib/arraypad.py&quot;, line 533, in &lt;module&gt;
    def pad(array, pad_width, mode='constant', **kwargs):
  File &quot;/usr/local/lib/python3.8/dist-packages/numpy/core/overrides.py&quot;, line 180, in decorator
    source_object = compile(
MemoryError
&gt;&gt;&gt; "><pre class="notranslate"><code class="notranslate">root@88e4e56b16a1:~# pip3 install numpy==1.19.5
Collecting numpy==1.19.5
  Downloading numpy-1.19.5-cp38-cp38-manylinux2010_x86_64.whl (14.9 MB)
     |████████████████████████████████| 14.9 MB 10.7 MB/s 
Installing collected packages: numpy
  Attempting uninstall: numpy
    Found existing installation: numpy 1.19.4
    Uninstalling numpy-1.19.4:
      Successfully uninstalled numpy-1.19.4
Successfully installed numpy-1.19.5
root@88e4e56b16a1:~# softlimit -a 1073741824 python3
Python 3.8.5 (default, Jul 28 2020, 12:59:40) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
&gt;&gt;&gt; import numpy
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "/usr/local/lib/python3.8/dist-packages/numpy/__init__.py", line 143, in &lt;module&gt;
    from . import lib
  File "/usr/local/lib/python3.8/dist-packages/numpy/lib/__init__.py", line 40, in &lt;module&gt;
    from .arraypad import *
  File "/usr/local/lib/python3.8/dist-packages/numpy/lib/arraypad.py", line 533, in &lt;module&gt;
    def pad(array, pad_width, mode='constant', **kwargs):
  File "/usr/local/lib/python3.8/dist-packages/numpy/core/overrides.py", line 180, in decorator
    source_object = compile(
MemoryError
&gt;&gt;&gt; 
</code></pre></div>
<p dir="auto">But ok with 1.19.4:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="root@88e4e56b16a1:~# pip3 install numpy==1.19.4
Collecting numpy==1.19.4
  Using cached numpy-1.19.4-cp38-cp38-manylinux2010_x86_64.whl (14.5 MB)
Installing collected packages: numpy
  Attempting uninstall: numpy
    Found existing installation: numpy 1.19.5
    Uninstalling numpy-1.19.5:
      Successfully uninstalled numpy-1.19.5
Successfully installed numpy-1.19.4
root@88e4e56b16a1:~# softlimit -a 1073741824 python3
Python 3.8.5 (default, Jul 28 2020, 12:59:40) 
[GCC 9.3.0] on linux
Type &quot;help&quot;, &quot;copyright&quot;, &quot;credits&quot; or &quot;license&quot; for more information.
&gt;&gt;&gt; import numpy
&gt;&gt;&gt; 
"><pre class="notranslate"><code class="notranslate">root@88e4e56b16a1:~# pip3 install numpy==1.19.4
Collecting numpy==1.19.4
  Using cached numpy-1.19.4-cp38-cp38-manylinux2010_x86_64.whl (14.5 MB)
Installing collected packages: numpy
  Attempting uninstall: numpy
    Found existing installation: numpy 1.19.5
    Uninstalling numpy-1.19.5:
      Successfully uninstalled numpy-1.19.5
Successfully installed numpy-1.19.4
root@88e4e56b16a1:~# softlimit -a 1073741824 python3
Python 3.8.5 (default, Jul 28 2020, 12:59:40) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
&gt;&gt;&gt; import numpy
&gt;&gt;&gt; 

</code></pre></div>
<p dir="auto">Dockerfile for reproducing - vanilla 20.04 + python3 + pip + softlimit</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cat Dockerfile
from ubuntu:20.04

RUN apt-get update &amp;&amp; apt-get install -y \
        python3 python3-pip \
        daemontools \
        &amp;&amp; apt-get clean"><pre class="notranslate"><code class="notranslate">cat Dockerfile
from ubuntu:20.04

RUN apt-get update &amp;&amp; apt-get install -y \
        python3 python3-pip \
        daemontools \
        &amp;&amp; apt-get clean
</code></pre></div>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="docker build -t numpylimit .
docker run --rm -it numpylimit /bin/bash"><pre class="notranslate"><code class="notranslate">docker build -t numpylimit .
docker run --rm -it numpylimit /bin/bash
</code></pre></div>
<h3 dir="auto">Error message:</h3>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="&gt;&gt;&gt; import numpy
Traceback (most recent call last):
  File &quot;&lt;stdin&gt;&quot;, line 1, in &lt;module&gt;
  File &quot;/usr/local/lib/python3.8/dist-packages/numpy/__init__.py&quot;, line 143, in &lt;module&gt;
    from . import lib
  File &quot;/usr/local/lib/python3.8/dist-packages/numpy/lib/__init__.py&quot;, line 40, in &lt;module&gt;
    from .arraypad import *
  File &quot;/usr/local/lib/python3.8/dist-packages/numpy/lib/arraypad.py&quot;, line 533, in &lt;module&gt;
    def pad(array, pad_width, mode='constant', **kwargs):
  File &quot;/usr/local/lib/python3.8/dist-packages/numpy/core/overrides.py&quot;, line 180, in decorator
    source_object = compile(
MemoryError"><pre class="notranslate"><code class="notranslate">&gt;&gt;&gt; import numpy
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "/usr/local/lib/python3.8/dist-packages/numpy/__init__.py", line 143, in &lt;module&gt;
    from . import lib
  File "/usr/local/lib/python3.8/dist-packages/numpy/lib/__init__.py", line 40, in &lt;module&gt;
    from .arraypad import *
  File "/usr/local/lib/python3.8/dist-packages/numpy/lib/arraypad.py", line 533, in &lt;module&gt;
    def pad(array, pad_width, mode='constant', **kwargs):
  File "/usr/local/lib/python3.8/dist-packages/numpy/core/overrides.py", line 180, in decorator
    source_object = compile(
MemoryError
</code></pre></div>
<h3 dir="auto">NumPy/Python version information:</h3>
<p dir="auto">1.19.5</p>
<p dir="auto">Let me know if there is any more information I can give you.</p>