MemoryError when creating and writing
Question: I'm relatively new at coding and was working in python on taking a large
amount (~2.0 GB) of data from an output file and turning it into a readable
and sorted list. My major issue is creating a test file of that size. The
input file will be a long array that is something around 2.56*10^8 (row) by 1
(column) The end result is something around a 6.4*10^7 (row) by 4 (column)
array and displaying it. To create a sample array, I have been using this code
(note that the size is not final, its just as large as I could get it by
increasing the size by powers of 2).

    
    
    import numpy as np
    import subprocess as subp
    from array import array
    
    keepData = 1
    
    if(not keepData):
      subp.call(['rm', 'Bertha.DAT']) #removes previous file if present
    
    girth = int(8e6) #number of final rows
    
    girthier = girth*4
    bigger_tim = np.zeros(girthier) #initial array
    
    File = 'Bertha.DAT'
    bid = open(File, 'wb')
    for ii in range(0,girth):
        tiny_tim = 100*(2*np.random.rand(1,3)-1)
        bigger_tim[ii*4]=4
        bigger_tim[ii*4+1]=tiny_tim[0,0]
        bigger_tim[ii*4+2]=tiny_tim[0,1]
        bigger_tim[ii*4+3]=tiny_tim[0,2]
        #for loop that inputs values in the style of the input result
    
    line.tofile(bid) #writes into file
    bid.close()
    

This code works for creating files that are 250MB, but they cannot create any
larger than that. Any help is greatly appreciated.

EDIT:

I am also adding in my second code to see if there is a problem there as well
due to large memory usage.

    
    
    import numpy as np
    import pandas as pd
    
    girth = int(24e6)
    
    Matrix = np.zeros((girth,4))
    
    Bertha = np.fromfile('Bertha.DAT',dtype = float,count = -1, sep = "")
    
    for jj in range(0,girth):
        Matrix[jj,0] = Bertha[jj*4]
        Matrix[jj,1] = Bertha[jj*4+1]
        Matrix[jj,2] = Bertha[jj*4+2]
        Matrix[jj,3] = Bertha[jj*4+3] 
    
    Table = pd.DataFrame({'Atomic Number':Matrix[:,0], 'X Position':Matrix[:,1], 'Y Position':Matrix[:,2], 'Z Position':Matrix[:,3]})
    print Table
    

EDIT: First code runs with up to 24e6 as the "girth" value, but gets the
following error with 32e6:

    
    
    Traceback (most recent call last):
    
      File "<ipython-input-1-cb13d37b70b9>", line 1, in <module>
        runfile('D:/WinPython-32bit-2.7.6.3/Big_Bertha.py', wdir='D:/WinPython-32bit-2.7.6.3')
    
       File "D:\WinPython-32bit-2.7.6.3\python-2.7.6\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 540, in runfile
    execfile(filename, namespace)
    
      File "D:/WinPython-32bit-2.7.6.3/Big_Bertha.py", line 19, in <module>
        bigger_tim = np.zeros(girthier) #initial array
    
    MemoryError
    

This looks like I cannot create the initial dummy matrix to store values due
to insufficient memory.

The second one has a very similar issue but with a different error at 24e6 as
the "girth" value.

    
    
    Traceback (most recent call last):
    
      File "<ipython-input-1-268052dcc4e8>", line 1, in <module>
        runfile('D:/WinPython-32bit-2.7.6.3/binaryReader.py', wdir='D:/WinPython-32bit-2.7.6.3')
    
       File "D:\WinPython-32bit-2.7.6.3\python-2.7.6\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 540, in runfile
        execfile(filename, namespace)
    
      File "D:/WinPython-32bit-2.7.6.3/binaryReader.py", line 14, in <module>
        Bertha = np.fromfile('Bertha.DAT',dtype = float,count = -1, sep = "")
    
    MemoryError
    


Answer: The error you receive comes from the fact that python cannot allocate any more
memory.

In your second example you are allocating a numpy table with 32 million rows
and 4 columns. With the usual double precision floats this alone is 1 GiB. The
row `np.fromfile =` needs to load a very big file, as the file length should
match `Matrix`, i.e. you will need to have at least 1 GiB of data from the
file.

And there it is: 1 GiB + 1 GiB = 2 GiB, which is the maximum for 32-bit
python. (And that is why 24 million rows is ok.) That is why the error is
thrown when the data is loaded from the file. Also, the limit is not 2 GiB for
user's data, it is 2 GiB in total, which may be much less in practice.

There are several things you can do:

  1. Do not create the empty table. Load the data from the file and reshape it to the shape you want (four columns and as many rows as it takes):

`m = np.fromfile("Bertha.DAT").reshape(-1,4)`

  2. Use something else as the data type than float (which is an 8-bit double). If you do not run into precision trouble, use 'float32' (or 'f4'). However, you cannot change the data type in `np.fromfile` as there it determines the data type and order in the file, as well.

  3. Use 64-bit python. If you deal with large data, that is the way to go. It consumes a bit more memory in some cases (and a lot in some others, but not with numpy), but if your computer has a lot of RAM, even very large tables work very well.

If you are interested in seeing how much your objects occupy memory, the `sys`
module has a nice function `sys.getsizeof` for they purpose, e.g.
`sys.getsizeof(Bertha)`.

* * *

There are a few stylistic things in your code which you might want to fix. One
is about naming your variables, they should be in lower case (class names are
capitalised). For this type of information it is very useful to read through
the `PEP 8` recommendation. (The name `Matrix` is a bit unfortunate in any
case, as there is something called `numpy.matrix`.)

Another thing that catches my eye is that you are iterating through a numpy
array with a for loop. That is usually a warning sign of something done in a
very slow way. There are rare cases where you need to do that, but usually
there are very concise and fast ways to manipulate the arrays.

