Managing dictionary memory size in python
Question: I have a program which imports a text file through standard input and
aggregates the lines into a dictionary. However the input file is very large
(1Tb order) and I wont have enough space to store the whole dictionary in
memory (running on 64Gb ram machine). Currently Iv got a very simple clause
which outputs the dictionary once it has reached a certain length (in this
case 100) and clears the memory. The output can then be aggregated at later
point.

So i want to: output the dictionary once memory is full. what is the best way
of managing this? Is there a function which gives me the current memory usage?
Is this costly to keep on checking? Am I using the right tactic?

    
    
    import sys
    X_dic = dict()
    
    # Used to print the dictionary in required format
    def print_dic(dic):
        for key, value in dic.iteritems():
            print "{0}\t{1}".format(key, value)
    
    for line in sys.stdin:
        value, key = line.strip().split(",")      
    
        if (not key in X_dic):
            X_dic[key] = []                            
    
        X_dic[key].append(value)
    
        # Limit size of dic.
        if( len(X_dic) == 100):
            print_dic(X_dic)              # Print and clear dictionary
            X_dic = dict()
    
    
    # Now output
    print_dic(X_dic)
    


Answer: The module `resource` provides some information on how much resources (memory,
etc.) you are using. See
[here](http://pythonforbiologists.com/index.php/measuring-memory-usage-in-
python/http://pythonforbiologists.com/index.php/measuring-memory-usage-in-
python/) for a nice little usage.

On a Linux system (I don't know where you are) you can watch the contents of
the file `/proc/meminfo`. As part of the _proc_ file system it is updated
automatically.

But I object to the whole strategy of monitoring the memory and using it up as
much as possible, actually. I'd rather propose to dump the dictionary
regularly (after 1M entries have been added or such). It probably will speed
up your program to keep the dict smaller than possible; also it presumably
will have advantages for later processing if all dumps are of similar size. If
you dump a huge dict which fit into your whole memory when nothing else was
using memory, then you later will have trouble re-reading that dict if
something else is currently using some of your memory. So then you would have
to create a situation in which nothing else is using memory (e. g. reboot or
similar). Not very convenient.

