<p dir="auto"><em>Original ticket <a href="http://projects.scipy.org/numpy/ticket/1631" rel="nofollow">http://projects.scipy.org/numpy/ticket/1631</a> on 2010-10-10 by <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/nstange/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/nstange">@nstange</a>, assigned to <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/charris/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/charris">@charris</a>.</em></p>
<p dir="auto">Hi everybody,</p>
<p dir="auto">for numpy-1.5.0, the testsuite crashes here with an SIGBUS here:<br>
"Test filled w/ flexible dtype ... Bus Error (core dumped)" (numpy-1.5.0/numpy/ma/tests/test_core.py)<br>
and here:<br>
"Test that 'exotic' formats are processed properly ... Bus Error (core dumped)" (numpy-1.5.0/numpy/ma/tests/test_mrecords.py).</p>
<p dir="auto">The problem is that SPARC processors require alignment, that is 8 byte values have to be aligned at 8 byte memory boundraries.<br>
The good point is: numpy knows that the array is misaligned (verified using gdb) and thus, the fix is easy (see attached diff 02_put_mask_only_on_behaved_arrays.diff): The check for ISCONTINOUS isn't enough in PyArray_PutMask (item_selection.c), also check for ISBEHAVED (= ISALIGNED &amp;&amp; ISWRITABLE).</p>
<h6 dir="auto"></h6>
<p dir="auto">But now, another problem arises:<br>
a SIGSEGV here:<br>
"Tests fields retrieval"<br>
(numpy-1.5.0/numpy/ma/tests/test_mrecords.py:77).</p>
<p dir="auto">The problem is in _copy_same_shape(numpy-1.5.0/numpy/core/src/multiarrayctors.c:732): dest-&gt;dimensions == NULL.<br>
Please also note that maxaxis == -1 at that point.<br>
Check the attached diff "04_copy_from_same_shape_zerodim_fix.diff" for details. There's one point in the diff I'm unsure about: The right position of<br>
PyArray_INCREF(src);<br>
PyArray_XDECREF(dest);<br>
I tried to resemble the original "logical" position, but since I have no clue about Python's reference counting, please have a look.</p>
<h6 dir="auto"></h6>
<p dir="auto">While debugging the last SIGSEGV-issue, I stumbled over another mistake (at least I think so):</p>
<p dir="auto">In PyArray_IterAllButAxis (numpy-1.5.0/numpy/core/src/multiarray/iterators.c), the minaxis won't be set if the first nonzero stride is the smallest one.<br>
See attached diff (03_fix_iterallbutaxis_minstride_search.diff) for a fix.</p>
<h6 dir="auto"></h6>
<p dir="auto">The reason why I poste those three issues into one report is that the testsuite still doesn't succeed and thus, I'm unsure if I've broken sth. with my diffs.</p>
<h1 dir="auto">What I get now is:</h1>
<h2 dir="auto">FAIL: Ticket <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="7728996" data-permission-text="Title is private" data-url="https://github.com/numpy/numpy/issues/1897" data-hovercard-type="issue" data-hovercard-url="/numpy/numpy/issues/1897/hovercard" href="https://github.com/numpy/numpy/issues/1897">#1897</a> second test</h2>
<p dir="auto">Traceback (most recent call last):<br>
File "/pf/m/m222086/xas/solaris10/python2/python-2.7-ve0-gcc/lib/python2.7/sit<br>
e-packages/numpy/core/tests/test_regression.py", line 1255, in test_structured_a<br>
rrays_with_objects2<br>
assert sys.getrefcount(strb) == numb<br>
AssertionError:<br>
7 = &lt;module 'sys' (built-in)&gt;.getrefcount('aaaa')<br>
array([[(0L, 'aaaa'), (0L, 'aaaa')]],<br>
dtype=[('f0', '&gt;i8'), ('f1', '|O4')]) = &lt;module 'numpy' from '/pf/m/m2<br>
22086/xas/solaris10/python2/python-2.7-ve0-gcc/lib/python2.7/site-packages/numpy<br>
/<strong>init</strong>.pyc'&gt;.array([[(0,'aaaa'),(1,'bbbb')]], 'i8,O')</p>
<blockquote>
<blockquote>
<p dir="auto">array([[(0L, 'aaaa'), (0L, 'aaaa')]],<br>
dtype=[('f0', '&gt;i8'), ('f1', '|O4')])[array([[(0L, 'aaaa'), (0L, 'aaaa<br>
')]],<br>
dtype=[('f0', '&gt;i8'), ('f1', '|O4')]).nonzero()] = array([[(0L, 'aaaa'<br>
), (0L, 'aaaa')]],<br>
dtype=[('f0', '&gt;i8'), ('f1', '|O4')]).ravel()[:1]<br>
assert &lt;module 'sys' (built-in)&gt;.getrefcount('bbbb') == 7<br>
assert &lt;module 'sys' (built-in)&gt;.getrefcount('aaaa') == 7 + 2</p>
</blockquote>
</blockquote>
<h1 dir="auto"></h1>
<h2 dir="auto">FAIL: Test filled w/ mvoid</h2>
<p dir="auto">Traceback (most recent call last):<br>
File "/pf/m/m222086/xas/solaris10/python2/python-2.7-ve0-gcc/lib/python2.7/sit<br>
e-packages/numpy/ma/tests/test_core.py", line 518, in test_filled_w_mvoid<br>
assert_equal(tuple(test), (1, default_fill_value(1.)))<br>
File "/pf/m/m222086/xas/solaris10/python2/python-2.7-ve0-gcc/lib/python2.7/sit<br>
e-packages/numpy/ma/testutils.py", line 94, in assert_equal<br>
return _assert_equal_on_sequences(actual, desired, err_msg='')<br>
File "/pf/m/m222086/xas/solaris10/python2/python-2.7-ve0-gcc/lib/python2.7/sit<br>
e-packages/numpy/ma/testutils.py", line 66, in _assert_equal_on_sequences<br>
assert_equal(actual[k], desired[k], 'item=%r\n%s' % (k,err_msg))<br>
File "/pf/m/m222086/xas/solaris10/python2/python-2.7-ve0-gcc/lib/python2.7/sit<br>
e-packages/numpy/ma/testutils.py", line 98, in assert_equal<br>
raise AssertionError(msg)<br>
AssertionError:<br>
Items are not equal:<br>
item=1</p>
<p dir="auto">ACTUAL: 2.0<br>
DESIRED: 1e+20</p>
<blockquote>
<blockquote>
<p dir="auto">raise AssertionError('\nItems are not equal:\nitem=1\n\n ACTUAL: 2.0\n DESIR<br>
ED: 1e+20')</p>
</blockquote>
</blockquote>
<p dir="auto">At least not segfaults/bus errors anymore ;).<br>
Btw.: I don't know what mvoid is, but have a look at the following:</p>
<p dir="auto">myuid@myhost:~$ ~/xas/solaris10/python2/python-2.7-ve0-gcc/bin/python         Python 2.7 (r27:82500, Oct  9 2010, 17:26:38)<br>
[GCC 4.5.1] on sunos5<br>
Type "help", "copyright", "credits" or "license" for more information.</p>
<blockquote>
<blockquote>
<blockquote>
<p dir="auto">import numpy as np<br>
import numpy.ma as ma<br>
import numpy.ma.core<br>
from numpy.ma.core import *<br>
x = ma.array([(1,2.)], mask=[(0,1)], dtype=[('a', int), ('b', float)])<br>
print x.filled()<br>
[(1, 1e+20)]<br>
x = mvoid((1,2.), mask=[(0,1)], dtype=[('a', int), ('b', float)])<br>
print x.filled()<br>
(1, 2.0)</p>
</blockquote>
</blockquote>
</blockquote>
<p dir="auto">Since the non-working mvoid-mask could have an impact on results, I cannot release that build to our the scientists at our site. Do you have any idea what the issue could be?<br>
Do you know what a mvoid is? I can't find any documentation about it. I'm seriously thinking about just removing that class from my numpy build (If I knew that there weren't any dependants outside of numpy)...</p>
<p dir="auto">Wishes</p>
<p dir="auto">Nicolai</p>
<p dir="auto"><em>Original ticket <a href="http://projects.scipy.org/numpy/ticket/1094" rel="nofollow">http://projects.scipy.org/numpy/ticket/1094</a> on 2009-04-24 by trac user Gerard, assigned to <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/charris/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/charris">@charris</a>.</em></p>
<p dir="auto">Running the numpy (version 1.3.0, but also occurs on 1.2.1) autotester fails on masked array test case with a bus error. Used system is Solaris 5.8:<br>
uname -a<br>
SunOS wsasd529 5.8 Generic_117350-18 sun4u sparc SUNW,Sun-Blade-1500</p>
<p dir="auto">output of "numpy.test(verbose=10)":</p>
<p dir="auto">...<br>
Tests how masks are initialized at the creation of Maskedarrays. ... ok<br>
Check the use of ndmin ... ok<br>
Make sure we're not losing the original mask w/ ndmin ... ok<br>
Tests creaating a masked array from alist of masked arrays. ... ok<br>
test_deepcopy (test_core.TestMaskedArray) ... ok<br>
Test printing a masked array w/ fancy dtype. ... ok<br>
Test filled w/ flexible dtype ... Bus error</p>
<p dir="auto">Bus error is caused by a misaligned memory write (see output of purify below):</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="MAW: Misaligned memory write
  This is occurring while in:
        DOUBLE_fastputmask [arraytypes.inc.src:2282]
                   s_val = *vals;
                   for (i = 0; i &lt; ni; i++) {
                       if (mask[i]) {
        =&gt;                 in[i] = s_val;
                       }
                   }
               }
        PyArray_PutMask [multiarraymodule.c:4687]
                       }
                   }
                   else {
        =&gt;             func(dest, mask-&gt;data, ni, values-&gt;data, nv);
                   }
               }

        array_putmask  [multiarraymodule.c:4600]
                                                &amp;array, &amp;mask, &amp;values)) {
                   return NULL;
               }
        =&gt;     return PyArray_PutMask((PyArrayObject *)array, values, mask);
           }

           /*NUMPY_API
        PyCFunction_Call [methodobject.c:77]
        PyEval_EvalFrameEx [ceval.c:3564]
        PyEval_EvalFrameEx [ceval.c:3650]
        PyEval_EvalCodeEx [ceval.c:2831]
        PyEval_EvalFrameEx [ceval.c:3659]
        ..."><pre class="notranslate"><code class="notranslate">MAW: Misaligned memory write
  This is occurring while in:
        DOUBLE_fastputmask [arraytypes.inc.src:2282]
                   s_val = *vals;
                   for (i = 0; i &lt; ni; i++) {
                       if (mask[i]) {
        =&gt;                 in[i] = s_val;
                       }
                   }
               }
        PyArray_PutMask [multiarraymodule.c:4687]
                       }
                   }
                   else {
        =&gt;             func(dest, mask-&gt;data, ni, values-&gt;data, nv);
                   }
               }

        array_putmask  [multiarraymodule.c:4600]
                                                &amp;array, &amp;mask, &amp;values)) {
                   return NULL;
               }
        =&gt;     return PyArray_PutMask((PyArrayObject *)array, values, mask);
           }

           /*NUMPY_API
        PyCFunction_Call [methodobject.c:77]
        PyEval_EvalFrameEx [ceval.c:3564]
        PyEval_EvalFrameEx [ceval.c:3650]
        PyEval_EvalCodeEx [ceval.c:2831]
        PyEval_EvalFrameEx [ceval.c:3659]
        ...
</code></pre></div>
<p dir="auto">Commenting the execution of the fastputmask in multiarraymodule.c (and thus running the 'slower' version already implemented there) solves the problem:</p>
<p dir="auto">multiarraymodule.c, line 4676 (and further):</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="    func = self-&gt;descr-&gt;f-&gt;fastputmask;
    /*if (func == NULL) {*/
        for (i = 0; i &lt; ni; i++) {
            tmp = ((Bool *)(mask-&gt;data))[i];
            if (tmp) {
                src = values-&gt;data + chunk*(i % nv);
                memmove(dest + i*chunk, src, chunk);
            }
        }
    /*}
    else {
        func(dest, mask-&gt;data, ni, values-&gt;data, nv);
    }*/"><pre class="notranslate"><code class="notranslate">    func = self-&gt;descr-&gt;f-&gt;fastputmask;
    /*if (func == NULL) {*/
        for (i = 0; i &lt; ni; i++) {
            tmp = ((Bool *)(mask-&gt;data))[i];
            if (tmp) {
                src = values-&gt;data + chunk*(i % nv);
                memmove(dest + i*chunk, src, chunk);
            }
        }
    /*}
    else {
        func(dest, mask-&gt;data, ni, values-&gt;data, nv);
    }*/
</code></pre></div>
<p dir="auto">added here are the comment blocks such that func is not executed anymore.</p>
<p dir="auto">'Minimal' piece of python code to reproduce:</p>
<p dir="auto">from numpy.ma.core import *</p>
<p dir="auto">f = array([(1,1)], dtype=[('i',int),('f',float)])</p>
<p dir="auto">f.mask = [1,1]</p>
<p dir="auto">f.filled()</p>