<h3 dir="auto">Describe the issue:</h3>
<p dir="auto">I have a .npy file generated from an h5py dataset. The data is a pure Numpy array (no subclass) and the file has been written by Numpy, but the dtype contains metadata set by h5py.<br>
In this particular case, the metadata contains a <code class="notranslate">&lt;class 'bytes'&gt;</code> expression, which cannot be handled by <code class="notranslate">safe_eval</code> . I don't care about the metadata, but it renders the entire file unreadable.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content=" File &quot;/home/sjoerd/miniconda3/envs/active-papers/lib/python3.9/site-packages/numpy/lib/format.py&quot;, line 776, in read_array
    shape, fortran_order, dtype = _read_array_header(
  File &quot;/home/sjoerd/miniconda3/envs/active-papers/lib/python3.9/site-packages/numpy/lib/format.py&quot;, line 632, in _read_array_header
    raise ValueError(msg.format(header)) from e
ValueError: Cannot parse header: &quot;{'descr': [('paper_ref', ('|O', {'vlen': &lt;class 'bytes'&gt;})), ('path', ('|O', {'vlen': &lt;class 'bytes'&gt;}))], 'fortran_order': False, 'shape': (), }                                    \n&quot;"><pre lang="text" class="notranslate"><code class="notranslate"> File "/home/sjoerd/miniconda3/envs/active-papers/lib/python3.9/site-packages/numpy/lib/format.py", line 776, in read_array
    shape, fortran_order, dtype = _read_array_header(
  File "/home/sjoerd/miniconda3/envs/active-papers/lib/python3.9/site-packages/numpy/lib/format.py", line 632, in _read_array_header
    raise ValueError(msg.format(header)) from e
ValueError: Cannot parse header: "{'descr': [('paper_ref', ('|O', {'vlen': &lt;class 'bytes'&gt;})), ('path', ('|O', {'vlen': &lt;class 'bytes'&gt;}))], 'fortran_order': False, 'shape': (), }                                    \n"
</code></pre></div>
<p dir="auto">I  would consider this a bug in <code class="notranslate">np.save</code> . Serialization of a dtype descriptor that results in an unreadable header should fail loudly. It would be nice if the <code class="notranslate">np.save</code> could then be re-tried with the metadata stripped.</p>
<h3 dir="auto">Reproduce the code example:</h3>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import numpy as np
np.load(&quot;file.npy&quot;)"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">numpy</span> <span class="pl-k">as</span> <span class="pl-s1">np</span>
<span class="pl-s1">np</span>.<span class="pl-en">load</span>(<span class="pl-s">"file.npy"</span>)</pre></div>
<h3 dir="auto">Error message:</h3>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="Traceback (most recent call last):
  File &quot;/home/sjoerd/bayesian_inference_fm/aptool-dump.py&quot;, line 65, in dump
    arr = np.load(child_target + &quot;.npy&quot;)
  File &quot;/home/sjoerd/miniconda3/envs/active-papers/lib/python3.9/site-packages/numpy/lib/npyio.py&quot;, line 432, in load
    return format.read_array(fid, allow_pickle=allow_pickle,
  File &quot;/home/sjoerd/miniconda3/envs/active-papers/lib/python3.9/site-packages/numpy/lib/format.py&quot;, line 776, in read_array
    shape, fortran_order, dtype = _read_array_header(
  File &quot;/home/sjoerd/miniconda3/envs/active-papers/lib/python3.9/site-packages/numpy/lib/format.py&quot;, line 632, in _read_array_header
    raise ValueError(msg.format(header)) from e
ValueError: Cannot parse header: &quot;{'descr': [('paper_ref', ('|O', {'vlen': &lt;class 'bytes'&gt;})), ('path', ('|O', {'vlen': &lt;class 'bytes'&gt;}))], 'fortran_order': False, 'shape': (), }                                    \n&quot;"><pre class="notranslate">Traceback (most recent call last):
  File <span class="pl-s"><span class="pl-pds">"</span>/home/sjoerd/bayesian_inference_fm/aptool-dump.py<span class="pl-pds">"</span></span>, line 65, <span class="pl-k">in</span> dump
    arr = np.load(child_target + <span class="pl-s"><span class="pl-pds">"</span>.npy<span class="pl-pds">"</span></span>)
  File <span class="pl-s"><span class="pl-pds">"</span>/home/sjoerd/miniconda3/envs/active-papers/lib/python3.9/site-packages/numpy/lib/npyio.py<span class="pl-pds">"</span></span>, line 432, <span class="pl-k">in</span> load
    <span class="pl-k">return</span> format.read_array(fid, allow_pickle=allow_pickle,
  File <span class="pl-s"><span class="pl-pds">"</span>/home/sjoerd/miniconda3/envs/active-papers/lib/python3.9/site-packages/numpy/lib/format.py<span class="pl-pds">"</span></span>, line 776, <span class="pl-k">in</span> read_array
    shape, fortran_order, dtype = _read_array_header(
  File <span class="pl-s"><span class="pl-pds">"</span>/home/sjoerd/miniconda3/envs/active-papers/lib/python3.9/site-packages/numpy/lib/format.py<span class="pl-pds">"</span></span>, line 632, <span class="pl-k">in</span> _read_array_header
    raise ValueError(msg.format(header)) from e
ValueError: Cannot parse header: <span class="pl-s"><span class="pl-pds">"</span>{'descr': [('paper_ref', ('|O', {'vlen': &lt;class 'bytes'&gt;})), ('path', ('|O', {'vlen': &lt;class 'bytes'&gt;}))], 'fortran_order': False, 'shape': (), }                                    \n<span class="pl-pds">"</span></span></pre></div>
<h3 dir="auto">Runtime information:</h3>
<p dir="auto">Numpy version:</p>
<p dir="auto">1.24.0<br>
3.9.16 | packaged by conda-forge | (main, Feb  1 2023, 21:39:03)<br>
[GCC 11.3.0]</p>
<p dir="auto">show_runtime:</p>
<p dir="auto">WARNING: <code class="notranslate">threadpoolctl</code> not found in system! Install it by <code class="notranslate">pip install threadpoolctl</code>. Once installed, try <code class="notranslate">np.show_runtime</code> again for more detailed build information<br>
[{'simd_extensions': {'baseline': ['SSE', 'SSE2', 'SSE3'],<br>
'found': ['SSSE3',<br>
'SSE41',<br>
'POPCNT',<br>
'SSE42',<br>
'AVX',<br>
'F16C',<br>
'FMA3',<br>
'AVX2'],<br>
'not_found': ['AVX512F',<br>
'AVX512CD',<br>
'AVX512_KNL',<br>
'AVX512_KNM',<br>
'AVX512_SKX',<br>
'AVX512_CLX',<br>
'AVX512_CNL',<br>
'AVX512_ICL']}}]</p>
<h3 dir="auto">Context for the issue:</h3>
<p dir="auto">A pure Numpy array (no subclasses) saved with <code class="notranslate">np.save</code> should be readable by <code class="notranslate">np.load</code>. This issue gives a counter-example.</p>
<p dir="auto">I could try to write a patch to fix this issue, if you are interested.</p>
<p dir="auto">The dtype description in an NPY file can be written in a way that prevents the file from being read. The attached array in <a href="https://github.com/numpy/numpy/files/3439471/bad_array.pkl.zip">bad_array.pkl.zip</a> exhibits this problem. In this case the written dtype is <code class="notranslate">[('id', '|V16'), ('has_fit', '|b1'), ('timeax', ('|O', {'vlen': dtype('float64')})), ('', '|V8'), ('profile', ('|O', {'vlen': dtype('float64')})), ('', '|V8'), ('area', '&lt;f8')]</code>.</p>
<p dir="auto">(Note that it was obtained from via h5py - if this is an issue with the format of the arrays produced by that library, I'm happy to file a bug there instead.)</p>
<h3 dir="auto">Reproducing code example:</h3>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import numpy as np
import pickle
import io

with open('bad_array.pkl', 'rb') as f:
    arr = pickle.load(f)

buf = io.BytesIO()
np.save(buf, arr)
buf.seek(0)
np.load(buf)"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">numpy</span> <span class="pl-k">as</span> <span class="pl-s1">np</span>
<span class="pl-k">import</span> <span class="pl-s1">pickle</span>
<span class="pl-k">import</span> <span class="pl-s1">io</span>

<span class="pl-k">with</span> <span class="pl-en">open</span>(<span class="pl-s">'bad_array.pkl'</span>, <span class="pl-s">'rb'</span>) <span class="pl-k">as</span> <span class="pl-s1">f</span>:
    <span class="pl-s1">arr</span> <span class="pl-c1">=</span> <span class="pl-s1">pickle</span>.<span class="pl-en">load</span>(<span class="pl-s1">f</span>)

<span class="pl-s1">buf</span> <span class="pl-c1">=</span> <span class="pl-s1">io</span>.<span class="pl-v">BytesIO</span>()
<span class="pl-s1">np</span>.<span class="pl-en">save</span>(<span class="pl-s1">buf</span>, <span class="pl-s1">arr</span>)
<span class="pl-s1">buf</span>.<span class="pl-en">seek</span>(<span class="pl-c1">0</span>)
<span class="pl-s1">np</span>.<span class="pl-en">load</span>(<span class="pl-s1">buf</span>)</pre></div>
<h3 dir="auto">Error message:</h3>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Traceback (most recent call last):
  File &quot;scripts/badnpy.py&quot;, line 11, in &lt;module&gt;
    np.load(buf)
  File &quot;/Users/colingavin/Code/908/clarent2/algo/py/.venv/lib/python3.7/site-packages/numpy/lib/npyio.py&quot;, line 453, in load
    pickle_kwargs=pickle_kwargs)
  File &quot;/Users/colingavin/Code/908/clarent2/algo/py/.venv/lib/python3.7/site-packages/numpy/lib/format.py&quot;, line 712, in read_array
    shape, fortran_order, dtype = _read_array_header(fp, version)
  File &quot;/Users/colingavin/Code/908/clarent2/algo/py/.venv/lib/python3.7/site-packages/numpy/lib/format.py&quot;, line 578, in _read_array_header
    d = safe_eval(header)
  File &quot;/Users/colingavin/Code/908/clarent2/algo/py/.venv/lib/python3.7/site-packages/numpy/lib/utils.py&quot;, line 1139, in safe_eval
    return ast.literal_eval(source)
  File &quot;/Users/colingavin/.pyenv/versions/3.7.2/Python.framework/Versions/3.7/lib/python3.7/ast.py&quot;, line 91, in literal_eval
    return _convert(node_or_string)
  File &quot;/Users/colingavin/.pyenv/versions/3.7.2/Python.framework/Versions/3.7/lib/python3.7/ast.py&quot;, line 79, in _convert
    map(_convert, node.values)))
  File &quot;/Users/colingavin/.pyenv/versions/3.7.2/Python.framework/Versions/3.7/lib/python3.7/ast.py&quot;, line 74, in _convert
    return list(map(_convert, node.elts))
  File &quot;/Users/colingavin/.pyenv/versions/3.7.2/Python.framework/Versions/3.7/lib/python3.7/ast.py&quot;, line 72, in _convert
    return tuple(map(_convert, node.elts))
  File &quot;/Users/colingavin/.pyenv/versions/3.7.2/Python.framework/Versions/3.7/lib/python3.7/ast.py&quot;, line 72, in _convert
    return tuple(map(_convert, node.elts))
  File &quot;/Users/colingavin/.pyenv/versions/3.7.2/Python.framework/Versions/3.7/lib/python3.7/ast.py&quot;, line 79, in _convert
    map(_convert, node.values)))
  File &quot;/Users/colingavin/.pyenv/versions/3.7.2/Python.framework/Versions/3.7/lib/python3.7/ast.py&quot;, line 90, in _convert
    return _convert_signed_num(node)
  File &quot;/Users/colingavin/.pyenv/versions/3.7.2/Python.framework/Versions/3.7/lib/python3.7/ast.py&quot;, line 63, in _convert_signed_num
    return _convert_num(node)
  File &quot;/Users/colingavin/.pyenv/versions/3.7.2/Python.framework/Versions/3.7/lib/python3.7/ast.py&quot;, line 55, in _convert_num
    raise ValueError('malformed node or string: ' + repr(node))
ValueError: malformed node or string: &lt;_ast.Call object at 0x10b965a58&gt;"><pre class="notranslate"><code class="notranslate">Traceback (most recent call last):
  File "scripts/badnpy.py", line 11, in &lt;module&gt;
    np.load(buf)
  File "/Users/colingavin/Code/908/clarent2/algo/py/.venv/lib/python3.7/site-packages/numpy/lib/npyio.py", line 453, in load
    pickle_kwargs=pickle_kwargs)
  File "/Users/colingavin/Code/908/clarent2/algo/py/.venv/lib/python3.7/site-packages/numpy/lib/format.py", line 712, in read_array
    shape, fortran_order, dtype = _read_array_header(fp, version)
  File "/Users/colingavin/Code/908/clarent2/algo/py/.venv/lib/python3.7/site-packages/numpy/lib/format.py", line 578, in _read_array_header
    d = safe_eval(header)
  File "/Users/colingavin/Code/908/clarent2/algo/py/.venv/lib/python3.7/site-packages/numpy/lib/utils.py", line 1139, in safe_eval
    return ast.literal_eval(source)
  File "/Users/colingavin/.pyenv/versions/3.7.2/Python.framework/Versions/3.7/lib/python3.7/ast.py", line 91, in literal_eval
    return _convert(node_or_string)
  File "/Users/colingavin/.pyenv/versions/3.7.2/Python.framework/Versions/3.7/lib/python3.7/ast.py", line 79, in _convert
    map(_convert, node.values)))
  File "/Users/colingavin/.pyenv/versions/3.7.2/Python.framework/Versions/3.7/lib/python3.7/ast.py", line 74, in _convert
    return list(map(_convert, node.elts))
  File "/Users/colingavin/.pyenv/versions/3.7.2/Python.framework/Versions/3.7/lib/python3.7/ast.py", line 72, in _convert
    return tuple(map(_convert, node.elts))
  File "/Users/colingavin/.pyenv/versions/3.7.2/Python.framework/Versions/3.7/lib/python3.7/ast.py", line 72, in _convert
    return tuple(map(_convert, node.elts))
  File "/Users/colingavin/.pyenv/versions/3.7.2/Python.framework/Versions/3.7/lib/python3.7/ast.py", line 79, in _convert
    map(_convert, node.values)))
  File "/Users/colingavin/.pyenv/versions/3.7.2/Python.framework/Versions/3.7/lib/python3.7/ast.py", line 90, in _convert
    return _convert_signed_num(node)
  File "/Users/colingavin/.pyenv/versions/3.7.2/Python.framework/Versions/3.7/lib/python3.7/ast.py", line 63, in _convert_signed_num
    return _convert_num(node)
  File "/Users/colingavin/.pyenv/versions/3.7.2/Python.framework/Versions/3.7/lib/python3.7/ast.py", line 55, in _convert_num
    raise ValueError('malformed node or string: ' + repr(node))
ValueError: malformed node or string: &lt;_ast.Call object at 0x10b965a58&gt;
</code></pre></div>
<h3 dir="auto">Numpy/Python version information:</h3>
<p dir="auto">1.17.0 3.7.2 (default, Jul 14 2019, 10:22:38)<br>
[Clang 9.0.0 (clang-900.0.39.2)]</p>