python memory exception when not at full memory usage
Question: I am using Ubuntu 64bit 12.04. My machine has 64gigs of RAM. I am running a
script where I have to store ~9gig of data into a dictionary.

It is a simple dictionary where keys are 30 characters and value is just a
integer.

However, the script is throwing a memory exception at around 58% memory usage.
What is going on here? Is there a max limit to dictionary size?


Answer: I don't think there is a max value that limits the size of a dictionary in
Python. Assuming your script is running under Unix, you can increase the
memory limit your process can consume via standard library module
[resource](https://docs.python.org/2/library/resource.html).

    
    
    >>> import resource
    >>> resource.setrlimit(resource.RLIMIT_AS, (10**9, 10**9))
    

You may also want to periodically check the memory usage with
`resource.getrusage()` function. The resulting object has the attribute
`ru_maxrss`, which gives total memory usage for the calling process.

    
    
    >>> import resource
    >>> resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
    >>> 20631552
    

By this way, at least you can make sure that it is your script which eats the
memory.

