Title: Initializing a memmap on identical files produces different arrays
Tags: python;numpy
Question: I am translating some algorithms and am checking my work by comparing outputs. I have a .img file created by IDL and a .img file created by Python. The two outputs are identical bitwise, but when I open a memmap to continue processing, the Python file loses some data.

```&gt;&gt;&gt; mm = np.memmap("X:/eordentl/Processing Algorithms/processed_data_py/
        C5704B-00045Z-03_verytop_2017_08_17_03_28_29/
        C5704B-00045Z-03_verytop_2017_08_17_03_28_29_SWIRcalib.img", 
        dtype="float32", mode="r", offset=0, shape=(333,320,285))
&gt;&gt;&gt; mm[0]
memmap([[ 0.        ,  0.        ,  0.        , ...,  0.        ,
          0.        ,  0.        ],
        [ 0.        ,  0.        ,  0.        , ...,  0.        ,
          0.        ,  0.        ],
        [ 0.        ,  0.        ,  0.        , ...,  0.        ,
          0.        ,  0.        ],
        ...,
        [ 0.3388325 ,  0.30027774,  0.32364482, ...,  0.40399274,
          0.42492244,  0.41838124],
        [ 0.28071803,  0.30334   ,  0.32874447, ...,  0.10009208,
          0.10556249,  0.05749646],
        [ 0.09928307,  0.1659135 ,  0.36206895, ...,  0.00572116,
         -0.00990769,  0.00214016]], dtype=float32)
&gt;&gt;&gt; mm2=np.memmap("X:/eordentl/Processing Algorithms/processed_data_idl/
        C5704B-00045Z-03_verytop_2017_08_17_03_28_29_SWIRcalib.img", 
        dtype="float32", mode="r", offset=0, shape=(333,320,285))
&gt;&gt;&gt; mm2[0]
memmap([[ 0.01608727,  0.03300966,  0.04274924, ...,  0.07621645,
          0.07274907,  0.07459512],
        [ 0.06294538,  0.07551169,  0.07973923, ...,  0.42498964,
          0.38354877,  0.396222  ],
        [ 0.34490117,  0.3083234 ,  0.27291125, ...,  0.32263884,
          0.31246758,  0.3155154 ],
        ...,
        [ 0.3388325 ,  0.30027774,  0.32364482, ...,  0.40399274,
          0.42492244,  0.41838124],
        [ 0.28071803,  0.30334   ,  0.32874447, ...,  0.10009208,
          0.10556249,  0.05749646],
        [ 0.09928307,  0.1659135 ,  0.36206895, ...,  0.00572116,
         -0.00990769,  0.00214016]], dtype=float32)
&gt;&gt;&gt; import filecmp
&gt;&gt;&gt; filecmp.cmp("X:/eordentl/Processing Algorithms/processed_data_py/C5704B-00045Z-03_verytop_2017_08_17_03_28_29/C5704B-00045Z-03_verytop_2017_08_17_03_28_29_SWIRcalib.img","X:/eordentl/Processing Algorithms/processed_data_idl/C5704B-00045Z-03_verytop_2017_08_17_03_28_29_SWIRcalib.img")
True
```

I am incredibly confused. These are not the only differences, see

```&gt;&gt;&gt; np.count_nonzero(mm2-mm)
109685
```

though the number is more likely due to floating point error.

Given that the files are bit by bit identical (verified through FC /b on windows cmd), what could be causing the difference, and how can I fix it?

Comment: For the first comment, the Python-generated one always has the zeros regardless of order of initialization (tested in new session). Am working on the second one now.

Comment: And yes, I did close the python-created file. (Phew!)

Comment: Haha, true. Just finished checking the with `mmap.mmap`, with the same results (python-created has 0's, IDL-created does not).

Comment: I think at this point I should say how I created the python file. I used `output = open(outfile, "wb")`, then wrote into the file with `output.write(arr.tobytes())`. I don't think there should be anything wrong with that (considering that the content of the two files appears to be identical).

Comment: Figured it out (I think). I was using the spectral library to interface with ENVI files. The spectral files handles memmaps to images pretty poorly, and I ended up with multiple maps open to a bankscarla@example.org. I rewrote the code to access the file using only one memmap, and that worked.

Comment: What happens if you map the files in the opposite order? Does the second one get the non-zero files no matter which one is mapped first, or does the Python-generated one get the non-zero files no matter which one is mapped first?

Comment: Also, what happens if you use the stdlib `mmap.mmap` to map the two files? Are the mmaps equal then? What if you then construct arrays off those mmaps using an explicit `buffer` (as described in the `numpy.memmap` docs)?

Comment: And one last thing: You did close or flush the Python-created file before memmapping it, right?

Comment: If I were you, I would have said damn rather than phew there. Sure, a silly mistake makes you feel dumb for a couple minutes, but it has the advantage of being easy for someone else to spot, and trivial for you to fix; if you didn't do anything silly, it's probably going to be a lot more of a pain to debug… But anyway, now you've ruled it out.

Comment: Well, now we know the problem isn't in numpy. And IIRC (although I'd have to check the code), numpy doesn't even use Python's mmap; it calls the POSIX or Windows functions directly, so if they're both getting the same results, it's probably not either of them doing anything weird. Which means… I'm stumped. Hopefully someone else can help.



{'source': 'StackExchangeDataset', 'subset': 'stackoverflow.com', 'metadata': {'AnswerCount': 0, 'CommentCount': 10, 'ContentLicense': 'CC BY-SA 4.0', 'CreationDate': '2018:00:06/26/18', 'FavoriteCount': 2.0, 'PostTypeId': 1, 'Tags': '<python><numpy>', 'Title': 'Initializing a memmap on identical files produces different arrays', 'ViewCount': 58, 'id': 51033645, 'user_AboutMe': '<p>WHEEEEEEEEEEEEEE</p>\n', 'user_AccountId': 4932263.0, 'user_DisplayName': 'Skynet_0', 'user_Reputation': 121.0, 'user_Views': 13.0, 'user_id': 3971637.0}}