Keep persistent variables in memory between runs of Python script
Question: Is there any way of keeping a result variable in memory so I don't have to
recalculate it each time I run the beginning of my script? I am doing a long
(5-10 sec) series of the exact operations on a data set (which I am reading
from disk) every time I run my script. This wouldn't be too much of a problem
since I'm pretty good at using the interactive editor to debug my code in
between runs; however sometimes the interactive capabilities just don't cut
it.

I know I could write my results to a file on disk, but I'd like to avoid doing
so if at all possible. This should be a solution which generates a variable
the first time I run the script, and keeps it in memory until the shell itself
is closed or until I explicitly tell it to fizzle out. Something like this:

    
    
    # Check if variable already created this session
    in_mem = var_in_memory() # Returns pointer to var, or False if not in memory yet
    if not in_mem:
        # Read data set from disk
        with open('mydata', 'r') as in_handle:
            mytext = in_handle.read()
        # Extract relevant results from data set
        mydata = parse_data(mytext)
        result = initial_operations(mydata)
        in_mem = store_persistent(result)
    

I've an inkling that the [shelve](http://docs.python.org/library/shelve.html)
module might be what I'm looking for here, but looks like in order to open a
shelve variable I would have to specify a file name for the persistent object,
and so I'm not sure if it's quite what I'm looking for.

Any tips on getting shelve to do what I want it to do? Any alternative ideas?


Answer: You can achieve something like this using the `reload` global function to re-
execute your main script's code. You will need to write a wrapper script that
imports your main script, asks it for the variable it wants to cache, caches a
copy of that within the wrapper script's module scope, and then when you want
(when you hit ENTER on stdin or whatever), it calls `reload(yourscriptmodule)`
but this time passes it the cached object such that yourscript can bypass the
expensive computation. Here's a quick example.

**wrapper.py**

    
    
    import sys
    import mainscript
    
    part1Cache = None
    if __name__ == "__main__":
        while True:
            if not part1Cache:
                part1Cache = mainscript.part1()
            mainscript.part2(part1Cache)
            print "Press enter to re-run the script, CTRL-C to exit"
            sys.stdin.readline()
            reload(mainscript)
    

**mainscript.py**

    
    
    def part1():
        print "part1 expensive computation running"
        return "This was expensive to compute"
    
    def part2(value):
        print "part2 running with %s" % value
    

While `wrapper.py` is running, you can edit `mainscript.py`, add new code to
the `part2` function and be able to run your new code against the pre-computed
part1Cache.

