It appears I've run out of 32-bit address space. What are my options?
Question: I'm trying to take the covariance of a large matrix using `numpy.cov`. I get
the following error:

    
    
    Python(22498,0xa02e3720) malloc: *** mmap(size=1340379136) failed (error code=12)
    *** error: can't allocate region
    *** set a breakpoint in malloc_error_break to debug
    
    Process Python bus error
    

It seems that this is not uncommon for 32-bit machines/builds (I have a 64-bit
mac os x 10.5, but using a 32-bit python and numpy build as I had trouble
building numpy+scipy+matplotlib on a 64-bit installation).

So at this point what would be the recommended course of action that will
allow me to proceed with the analysis, if not switching machines (none others
are available to me at the moment)? Export to fortran/C? Is there a simple(r)
solution? Thanks for your suggestions.


Answer: To be at your place, I would try to "pickle" (save) the matrix on my hard
drive, close python , then in command line re-open the pickeled file and do my
computation on a "fresh python" instance.

I would do that because maybe your problem is before computing the covariance.

    
    
    import cPickle
    import numpy
    M = numpy.array([[1,2],[3,4]]) # here it will be your matrix
    cPickle( M , open( "~/M.pic", "w") ) # here it's where you pickle the file
    

Here you close python. Your file should be saved in you home directory as
"M.pic".

    
    
    import cPickle
    import numpy
    M = cPickle.load( open( "~/M.pic", "r") )
    M = numpy.coa( M )
    

If it still does not work, try setting a "good" dtype for your data. numpy
seams to use dtype 'float64' of 'int64' by default. This is huge and if you do
not need this precision, you might want to reduce it to 'int32' or 'float32'.

    
    
    import numpy
    M = numpy.array([[1,2],[3,4]] , dtype.float32 )
    

Indeed, I can guarantee you that C/Fortran is not an option for you. Numpy is
already written in C/Fortran and probably by people cleverer than you and me
;)

By curiosity, how big is your matrix? how big is your pickled file?

