<p dir="auto">Trying to create a numpy array from ctypes and ran into a problem which seems to be because of the <em>pack</em> attribute in ctypes. Anyone seen this before?</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import sys  
import ctypes
import numpy as np

print(sys.version)
print(ctypes.__version__)
print(np.__version__)

class POINT(ctypes.Structure):
    _fields_ = [(&quot;x&quot;, ctypes.c_int),
                (&quot;y&quot;, ctypes.c_float)]  

class POINT_BUG(ctypes.Structure):
    _pack_ = 4
    _fields_ = [(&quot;x&quot;, ctypes.c_int),
                (&quot;y&quot;, ctypes.c_float)]      

n = np.array((POINT*5)())
print(n)

#This fails
nb = np.array((POINT_BUG*5)())"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">sys</span>  
<span class="pl-k">import</span> <span class="pl-s1">ctypes</span>
<span class="pl-k">import</span> <span class="pl-s1">numpy</span> <span class="pl-k">as</span> <span class="pl-s1">np</span>

<span class="pl-en">print</span>(<span class="pl-s1">sys</span>.<span class="pl-s1">version</span>)
<span class="pl-en">print</span>(<span class="pl-s1">ctypes</span>.<span class="pl-s1">__version__</span>)
<span class="pl-en">print</span>(<span class="pl-s1">np</span>.<span class="pl-s1">__version__</span>)

<span class="pl-k">class</span> <span class="pl-v">POINT</span>(<span class="pl-s1">ctypes</span>.<span class="pl-v">Structure</span>):
    <span class="pl-s1">_fields_</span> <span class="pl-c1">=</span> [(<span class="pl-s">"x"</span>, <span class="pl-s1">ctypes</span>.<span class="pl-s1">c_int</span>),
                (<span class="pl-s">"y"</span>, <span class="pl-s1">ctypes</span>.<span class="pl-s1">c_float</span>)]  

<span class="pl-k">class</span> <span class="pl-v">POINT_BUG</span>(<span class="pl-s1">ctypes</span>.<span class="pl-v">Structure</span>):
    <span class="pl-s1">_pack_</span> <span class="pl-c1">=</span> <span class="pl-c1">4</span>
    <span class="pl-s1">_fields_</span> <span class="pl-c1">=</span> [(<span class="pl-s">"x"</span>, <span class="pl-s1">ctypes</span>.<span class="pl-s1">c_int</span>),
                (<span class="pl-s">"y"</span>, <span class="pl-s1">ctypes</span>.<span class="pl-s1">c_float</span>)]      

<span class="pl-s1">n</span> <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-en">array</span>((<span class="pl-v">POINT</span><span class="pl-c1">*</span><span class="pl-c1">5</span>)())
<span class="pl-en">print</span>(<span class="pl-s1">n</span>)

<span class="pl-c">#This fails</span>
<span class="pl-s1">nb</span> <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-en">array</span>((<span class="pl-v">POINT_BUG</span><span class="pl-c1">*</span><span class="pl-c1">5</span>)())</pre></div>
<p dir="auto">Produces ouput:</p>
<blockquote>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="2.7.10 (default, May 23 2015, 09:40:32) [MSC v.1500 32 bit (Intel)]
1.1.0
1.10.4
[(0, 0.0) (0, 0.0) (0, 0.0) (0, 0.0) (0, 0.0)]
C:/Users/Mark/Desktop/bugreport.py:23: RuntimeWarning: Item size computed from the PEP 3118 buffer format string does not match the actual item size.
  (&quot;y&quot;, ctypes.c_float)]
Traceback (most recent call last):
  File &quot;&lt;stdin&gt;&quot;, line 1, in &lt;module&gt;
  File &quot;C:\Python27\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py&quot;, line 685, in runfile
    execfile(filename, namespace)
  File &quot;C:\Python27\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py&quot;, line 71, in execfile
    exec(compile(scripttext, filename, 'exec'), glob, loc)
  File &quot;C:/Users/Mark/Desktop/bugreport.py&quot;, line 23, in &lt;module&gt;
    nb = np.array((POINT_BUG*5)())
TypeError: long() argument must be a string or a number, not 'POINT_BUG'"><pre class="notranslate"><code class="notranslate">2.7.10 (default, May 23 2015, 09:40:32) [MSC v.1500 32 bit (Intel)]
1.1.0
1.10.4
[(0, 0.0) (0, 0.0) (0, 0.0) (0, 0.0) (0, 0.0)]
C:/Users/Mark/Desktop/bugreport.py:23: RuntimeWarning: Item size computed from the PEP 3118 buffer format string does not match the actual item size.
  ("y", ctypes.c_float)]
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "C:\Python27\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 685, in runfile
    execfile(filename, namespace)
  File "C:\Python27\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 71, in execfile
    exec(compile(scripttext, filename, 'exec'), glob, loc)
  File "C:/Users/Mark/Desktop/bugreport.py", line 23, in &lt;module&gt;
    nb = np.array((POINT_BUG*5)())
TypeError: long() argument must be a string or a number, not 'POINT_BUG'
</code></pre></div>
</blockquote>
<p dir="auto">This seems like a bug inherited from ctypes, so maybe we need a workaround:</p>
<p dir="auto">Start off with a very simple struct:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="class foo(ctypes.Structure):
    _fields_ = [('a', ctypes.c_uint8), ('b', ctypes.c_uint32)]
f = foo()"><pre class="notranslate"><span class="pl-k">class</span> <span class="pl-s1">foo</span>(<span class="pl-s1">ctypes</span>.<span class="pl-v">Structure</span>):
    <span class="pl-s1">_fields_</span> <span class="pl-c1">=</span> [(<span class="pl-s">'a'</span>, <span class="pl-s1">ctypes</span>.<span class="pl-s1">c_uint8</span>), (<span class="pl-s">'b'</span>, <span class="pl-s1">ctypes</span>.<span class="pl-s1">c_uint32</span>)]
<span class="pl-s1">f</span> <span class="pl-c1">=</span> <span class="pl-en">foo</span>()</pre></div>
<p dir="auto">We'd expect this to insert padding, and it does:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="&gt;&gt;&gt; memoryview(f).itemsize  # ok
8"><pre class="notranslate"><span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-en">memoryview</span>(<span class="pl-s1">f</span>).<span class="pl-s1">itemsize</span>  <span class="pl-c"># ok</span>
<span class="pl-c1">8</span></pre></div>
<p dir="auto">But that padding doesn't show up in the format string:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="&gt;&gt;&gt; memoryview(f).format
'T{&lt;B:a:&lt;I:b:}'"><pre class="notranslate"><span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-en">memoryview</span>(<span class="pl-s1">f</span>).<span class="pl-s1">format</span>
<span class="pl-s">'T{&lt;B:a:&lt;I:b:}'</span></pre></div>
<p dir="auto">The docs for <code class="notranslate">struct</code> say that</p>
<blockquote>
<p dir="auto">No padding is added when using non-native size and alignment, e.g. with ‘&lt;’, ‘&gt;’ "</p>
</blockquote>
<p dir="auto">So this format has size 5, which contradicts <code class="notranslate">itemsize</code>.</p>
<p dir="auto">Sure enough, we catch this in our PEP3118 parser:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="&gt;&gt;&gt; np.array(f)
RuntimeWarning: Item size computed from the PEP 3118 buffer format string does not match the actual item size.
Out[45]: array(&lt;__main__.foo object at 0x7f2715469bf8&gt;, dtype=object)"><pre class="notranslate"><span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">np</span>.<span class="pl-en">array</span>(<span class="pl-s1">f</span>)
<span class="pl-v">RuntimeWarning</span>: <span class="pl-v">Item</span> <span class="pl-s1">size</span> <span class="pl-s1">computed</span> <span class="pl-k">from</span> <span class="pl-s1">the</span> <span class="pl-v">PEP</span> <span class="pl-c1">3118</span> <span class="pl-s1">buffer</span> <span class="pl-s1">format</span> <span class="pl-s1">string</span> <span class="pl-s1">does</span> <span class="pl-c1">not</span> <span class="pl-s1">match</span> <span class="pl-s1">the</span> <span class="pl-s1">actual</span> <span class="pl-s1">item</span> <span class="pl-s1">size</span>.
<span class="pl-v">Out</span>[<span class="pl-c1">45</span>]: <span class="pl-s1">array</span>(<span class="pl-c1">&lt;</span><span class="pl-s1">__main__</span>.<span class="pl-s1">foo</span> <span class="pl-s1">object</span> <span class="pl-s1">at</span> <span class="pl-c1">0x7f2715469bf8</span><span class="pl-c1">&gt;</span>, <span class="pl-s1">dtype</span><span class="pl-c1">=</span><span class="pl-s1">object</span>)</pre></div>
<p dir="auto">I suspect there's already an issue for this - perhaps we need a PEP3118 tag.</p>
<p dir="auto">The fix would be to special case <code class="notranslate">ctypes</code> objects in the array constructor, and not call <code class="notranslate">memoryview</code> on them because it doesn't actually work.</p>
<hr>
<p dir="auto">Worse yet, numpy doesn't stand a chance with an <em>unpadded</em> struct:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="class foo(ctypes.Structure):
    _fields_ = [('a', ctypes.c_uint8), ('b', ctypes.c_uint32)]
    _pack_ = 1
f = foo()"><pre class="notranslate"><span class="pl-k">class</span> <span class="pl-s1">foo</span>(<span class="pl-s1">ctypes</span>.<span class="pl-v">Structure</span>):
    <span class="pl-s1">_fields_</span> <span class="pl-c1">=</span> [(<span class="pl-s">'a'</span>, <span class="pl-s1">ctypes</span>.<span class="pl-s1">c_uint8</span>), (<span class="pl-s">'b'</span>, <span class="pl-s1">ctypes</span>.<span class="pl-s1">c_uint32</span>)]
    <span class="pl-s1">_pack_</span> <span class="pl-c1">=</span> <span class="pl-c1">1</span>
<span class="pl-s1">f</span> <span class="pl-c1">=</span> <span class="pl-en">foo</span>()</pre></div>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="&gt;&gt;&gt; memoryview(f).itemsize  # ok
5
&gt;&gt;&gt; memoryview(f).format  # WAT
'B'"><pre class="notranslate"><span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-en">memoryview</span>(<span class="pl-s1">f</span>).<span class="pl-s1">itemsize</span>  <span class="pl-c"># ok</span>
<span class="pl-c1">5</span>
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-en">memoryview</span>(<span class="pl-s1">f</span>).<span class="pl-s1">format</span>  <span class="pl-c"># WAT</span>
<span class="pl-s">'B'</span></pre></div>