Writing large memory structures to disk with limited Memory resources (Python)
If anyone must know, I solved this problem by converting my large memory structure into three smaller memory structures (order dependent) which I could then pickle. When I load the memory structure I have to also re concatenate the memories into a larger structure. The memory saving isn't much however it is a workaround I was able to do for now. This solution is of course structure dependent.
I have a large memory bank of information (in a class that has a large dictionary) that needs to be loaded into memory and updated as information is received once the data has been compiled into this super large structure (up to 20GB) I need to then save this updated memory bank to disk for later loading. However with pickle I haven't been able to find a way I can pickle a file by streaming the data as it serializes it(I can't exceed 25.5 GB). If you notice between having both a 20GB structure and needing to have the serialized pickle it well exceeds my memory resources.
Is there a way to have pickle stream the information as it is serialized, or will I have to make my own function to write the memory to file(s) myself?
Is there a way to keep memory costs low (offloading the memory from from to disk as the process is completed)?