<p dir="auto">When using the openblas based pypi package for numpy on windows, a simply import allocates a lot (25-30MB/core) of committed memory per CPU core by default. This leads to MemoryErrors due to memory exhaustion, when all memory of a machine is committed used up. This can easily happen when using multiprocessing or other multi process architectures on a server with many cpu cores. Basically kills the server, unless you have at least (core*core)*30MB of RAM free.</p>
<p dir="auto">This is made worse by packages that simply import numpy for functionality unrelated to openblas, like <code class="notranslate">openpyxl</code> that only import some type definitions from numpy, so the memory usage comes as a big surprise.</p>
<p dir="auto">The problem why this happens is OpenBLAS usage of VirtualAlloc to allocate buffers on init. Using the MEMORY_COMMIT flag right away to emulate Linux mmap behaviour, instead of using only MEMORY_RESERVE and a later MEMORY_COMMIT on first actual use of the buffers, so forcing the reservation of physical (commited) memory instead of just virtual memory.</p>
<p dir="auto">Setting the OPENBLAS_NUM_THREADS=1 envvar fixes the memory allocation, but i did not see any direct mention of it related to memory usage, only to threading behaviour.</p>
<p dir="auto">It would be nice if just importing the module did not allocate and commit all that memory.</p>
<h3 dir="auto">Reproducing code example:</h3>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import numpy"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">numpy</span></pre></div>
<p dir="auto">Check the memory usage with sysinternal vmmap.<br>
<a href="https://docs.microsoft.com/en-us/sysinternals/downloads/vmmap" rel="nofollow">https://docs.microsoft.com/en-us/sysinternals/downloads/vmmap</a></p>
<p dir="auto">The usage in the 'private data' section (private bytes in process explorer) grows by megabytes, proportional to the number of CPU cores.</p>
<p dir="auto">This can be reduced to zero growth when <code class="notranslate">OPENBLAS_NUM_THREADS</code> is set to 1 before the import.</p>
<h3 dir="auto">Numpy/Python version information:</h3>
<blockquote>
<blockquote>
<blockquote>
<p dir="auto">print(numpy.<strong>version</strong>, sys.version)<br>
('1.16.2', '2.7.15 (v2.7.15:ca079a3ea3, Apr 30 2018, 16:30:26) [MSC v.1500 64 bit (AMD64)]')</p>
</blockquote>
</blockquote>
</blockquote>
<p dir="auto">Related to <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="907717829" data-permission-text="Title is private" data-url="https://github.com/vscode-restructuredtext/snooty-parser/issues/24" data-hovercard-type="issue" data-hovercard-url="/vscode-restructuredtext/snooty-parser/issues/24/hovercard" href="https://github.com/vscode-restructuredtext/snooty-parser/issues/24">vscode-restructuredtext/snooty-parser#24</a> and <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="907722755" data-permission-text="Title is private" data-url="https://github.com/networkx/networkx/issues/4857" data-hovercard-type="issue" data-hovercard-url="/networkx/networkx/issues/4857/hovercard" href="https://github.com/networkx/networkx/issues/4857">networkx/networkx#4857</a></p>
<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/1829149/120243452-77d12480-c21c-11eb-82ab-a8d7f160e4b9.png"><img src="https://user-images.githubusercontent.com/1829149/120243452-77d12480-c21c-11eb-82ab-a8d7f160e4b9.png" alt="image" style="max-width: 100%;"></a></p>
<p dir="auto">I expect <code class="notranslate">import numpy</code> to commit close to 0 memory.</p>
<h3 dir="auto">Reproducing code example:</h3>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import numpy as np"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">numpy</span> <span class="pl-k">as</span> <span class="pl-s1">np</span></pre></div>
<h3 dir="auto">Error message:</h3>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="  File &quot;C:\Users\Ark\AppData\Local\Programs\Python\Python37\lib\site-packages\numpy\__init__.py&quot;, line 153, in &lt;module&gt;
    from . import ma
  File &quot;C:\Users\Ark\AppData\Local\Programs\Python\Python37\lib\site-packages\numpy\ma\__init__.py&quot;, line 42, in &lt;module&gt;
    from . import core
  File &quot;&lt;frozen importlib._bootstrap&gt;&quot;, line 983, in _find_and_load
  File &quot;&lt;frozen importlib._bootstrap&gt;&quot;, line 967, in _find_and_load_unlocked
  File &quot;&lt;frozen importlib._bootstrap&gt;&quot;, line 677, in _load_unlocked
  File &quot;&lt;frozen importlib._bootstrap_external&gt;&quot;, line 724, in exec_module
  File &quot;&lt;frozen importlib._bootstrap_external&gt;&quot;, line 818, in get_code
  File &quot;&lt;frozen importlib._bootstrap_external&gt;&quot;, line 917, in get_data
MemoryError"><pre class="notranslate"><code class="notranslate">  File "C:\Users\Ark\AppData\Local\Programs\Python\Python37\lib\site-packages\numpy\__init__.py", line 153, in &lt;module&gt;
    from . import ma
  File "C:\Users\Ark\AppData\Local\Programs\Python\Python37\lib\site-packages\numpy\ma\__init__.py", line 42, in &lt;module&gt;
    from . import core
  File "&lt;frozen importlib._bootstrap&gt;", line 983, in _find_and_load
  File "&lt;frozen importlib._bootstrap&gt;", line 967, in _find_and_load_unlocked
  File "&lt;frozen importlib._bootstrap&gt;", line 677, in _load_unlocked
  File "&lt;frozen importlib._bootstrap_external&gt;", line 724, in exec_module
  File "&lt;frozen importlib._bootstrap_external&gt;", line 818, in get_code
  File "&lt;frozen importlib._bootstrap_external&gt;", line 917, in get_data
MemoryError
</code></pre></div>


<h3 dir="auto">NumPy/Python version information:</h3>
<p dir="auto">1.19.5 3.7.4 (tags/v3.7.4:e09359112e, Jul  8 2019, 20:34:20) [MSC v.1916 64 bit (AMD64)]</p>