python: improve performance and/or method to avoid memory error creating, saving and deleting variable variables
Question: I have been fighting against a function giving me a memory error and thanks to
your support ([Python: how to split and return a list from a function to avoid
memory error](http://stackoverflow.com/questions/17097483/python-how-to-split-
and-return-a-list-from-a-function-to-avoid-memory-error)) I managed to sort
the issue; however, since I am not a pro-programmer I would like to ask for
your opinion on my method and how to improve its performance (if possible).

The function is a generator function returning all cycles from an n-nodes
digraph. However, for a 12 nodes digraph, there are about 115 million cycles
(each defined as a list of nodes, e.g. [0,1,2,0] is a cycle). I need all
cycles available for further processing even after I have extracted some of
their properties when they were first generated, so they need to be stored
somewhere. So, the idea is to cut the result array every 10 million cycles to
avoid memory error (when an array is too big, python runs out of RAM) and
create a new array to store the following results. In the 12 node digraph, I
would then have 12 result arrays, 11 full ones (containing 10 million cycles
each) and the last containing 5 million cycles.

However, splitting the result array is not enough since the variables stay in
RAM. So, I still need to write each one to the disk and delete it afterwards
to clear the RAM.

As stated in [How do I do variable variables in
Python?](http://stackoverflow.com/questions/1373164/how-do-i-do-variable-
variables-in-python), using 'exec' to create variable variable names is not
very "clean" and dictionary solutions are better. However, in my case, if I
store the results in a single dictionary, it will run out of memory due to the
size of the arrays. Hence, I went for the 'exec' way. I would be grateful if
you could comment on that decision.

Also, to store the arrays I use numpy.savez_compressed which gives me a 43 Mb
file for each 10million cycles array. If it is not compressed it creates a 500
Mb file. However, using the compressed version slows the writing process. Any
idea how to speed the writing and/or compressing process?

A simplified version of the code I wrote is as follows:

    
    
    nbr_result_arrays=0
    result_array_0=[]
    result_lenght=10000000
    tmp=result_array_0 # I use tmp to avoid using exec within the for loop (exec slows down code execution) 
    for cycle in generator:
        tmp.append(cycle)    
        if len(tmp) == result_lenght:
            exec 'np.savez_compressed(\'results_' +str(nbr_result_arrays)+ '\', tmp)'
            exec 'del result_array_'+str(nbr_result_arrays)
            nbr_result_arrays+=1
            exec 'result_array_'+str(nbr_result_arrays)+'=[]'
            exec 'tmp=result_array_'+str(nbr_result_arrays)
    

Thanks for reading,

Aleix


Answer: How about using
[itertools.islice](http://docs.python.org/2/library/itertools.html#itertools.islice)?

    
    
    import itertools
    import numpy as np
    
    for i in itertools.count():
        tmp = list(itertools.islice(generator, 10000000))
        if not tmp:
            break
        np.savez_compressed('results_{}'.format(i), tmp)
        del tmp
    

