Profile Memory Allocation in Python (with support for Numpy arrays)
Can you just save/pickle some of the arrays to disk in tmp files when not using them? That's what I've had to do in the past with large arrays. Of course this will slow the program down, but at least it'll finish. Unless you need them all at once?
I have a program that contains a large number of objects, many of them Numpy arrays. My program is swapping miserably, and I'm trying to reduce the memory usage, because it actually can't finis on my system with the current memory requirements.
I am looking for a nice profiler that would allow me to check the amount of memory consumed by various objects (I'm envisioning a memory counterpart to cProfile) so that I know where to optimize.
I've heard decent things about Heapy, but Heapy unfortunately does not support Numpy arrays, and most of my program involves Numpy arrays.