struct.pack is much slower in python 2.6 when working with numpy arrays
Question: # The basic question

Hello everyone. I believe that I found an issue with python 2.6, struct.pack,
and numpy arrays. The issue is that the following code is incredibly slow when
I run it using python 2.6 (but it is sufficiently fast when I run it using
python 2.7 or 2.5).

    
    
    import numpy as np
    import struct
    
    x = np.mat(np.random.randint(0,3000,(1000,1000)))
    
    z = struct.pack('<'+'H'*x.size,*np.asarray(x).reshape(-1).astype(int))
    

I need to be able to run something similar to this numerous times for an
application that I am working on and right now my only option is to really run
it using python 2.6. I am asking if there is a faster way that I can do this
using python 2.6 or if I should just spend the time trying to get a system
administrator to install python 2.7 or 3.0 for me.

As far as why I need to be able to run on python 2.6 and how I figured out
that this line of code was the issue see below.

# The long story

I am working on a project where I am using python to modify some files that
are input into another program so that I can run a Monte Carlo simulation on
that program. Everything was going great, I wrote all the code using python 3,
made sure it also all worked in python 2.7 and ran a couple test cases on my
underpowered computer. Things went pretty well. It took about 30 seconds to
run a single test case. Then I went to port my analysis to a server that we
have that is much faster/more powerful then my laptop.

I got everything working on the server but have run into a huge problem. It
now takes 30 minutes to run through a single test case. After a little
investigation I found out that the server only has python version 2.6
installed and I believe that this is the problem. Further, I believe that the
following line in particular is the issue

    
    
    z = struct.pack('<'+'H'*x.size,*np.asarray(x).reshape(-1).astype(int))
    

where x is a numpy matrix, numpy has been imported as np, and struct has also
been imported. The reason I believe that this line is the issue follows.
First, when running this line in python 2.6 I get the following warning:

    
    
    DeprecationWarning: struct integer overflow masking is deprecated
    

despite the fact that I am absolutely certain that the contents of the array I
am trying to pack falls within the bounds of an unsigned short integer.
Running this same exact code using python 2.7 and 3.5 I do not get the
deprecation warning either.

Further, I attempted a short test using time it. I ran the following code
using python 2.7 and 3.0 on my local machine and got the following results:

    
    
    python3 -m timeit "import numpy as np;import struct;x=np.mat(np.random.randint(0,3200,(1000,1000)));z = struct.pack('<'+'H'*x.size,*np.asarray(x).reshape(-1).astype(int))"
    10 loops, best of 3: 467 msec per loop
    
    python2.7 -m timeit "import numpy as np;import struct;x=np.mat(np.random.randint(0,3200,(1000,1000)));z = struct.pack('<'+'H'*x.size,*np.asarray(x).reshape(-1).astype(int))"
    10 loops, best of 3: 468 msec per loop
    

which was pretty good. I then tried to run that using python 2.6

    
    
    python2.6 -m timeit "import numpy as np;import struct;x=np.mat(np.random.randint(0,3200,(1000,1000)));z = struct.pack('<'+'H'*x.size,*np.asarray(x).reshape(-1).astype(int))"
    10 loops, best of 3: 34.4 sec per loop
    

which is an absurd 7350% increase in time...

The question I'm asking is--Is there some way that I can get around this
bottleneck using python 2.6 or should I just try to get the system managers
for the server to upgrade to python 3.5 or 2.7?"


Answer: Why not avoid the `struct` module altogether and let `numpy` handle the binary
conversion for you?

For example:

    
    
    import numpy as np
    
    x = np.random.randint(0, 3200, (1000, 1000))
    z = x.astype('<u2').tostring()
    

`'<u2'` specifies little-endian (`<`) unsigned int (`u`) 16-bit (`2`). It's
identical to `<H`.

Just to prove that they're identical:

    
    
    z1 = x.astype('<u2').tostring()
    z2 = struct.pack('<'+'H'*x.size,*np.asarray(x).reshape(-1).astype(int))
    assert z1 == z2
    

And for timings (note: this is on python 2.7), the numpy version is ~100x
faster:

    
    
    In [7]: %timeit x.astype('<u2').tostring()
    100 loops, best of 3: 1.33 ms per loop
    
    In [8]: %timeit struct.pack('<'+'H'*x.size,*np.asarray(x).reshape(-1).astype(int))
    1 loops, best of 3: 118 ms per loop
    

