How to handle huge volume of data in limited RAM
This question makes me remember the early 80's. Memory used to be expensive and we invented swapping. The (high level part of the) OS sees more memory than actually present, and pages are copied on disk. When one is needed another one is swapped off, and the page is copied back into memory. Performances are awful, but at least it works.
Your question is rather broad, but a rule of thumb says that if you can process your data in batches, explicitely loading batches of data is much more efficient, but if the algorythm is too complex or requires actions on any data at any moment, just let the swap care for it.
So add a swap file of significantly greater than the memory you think you need (with the given sizes, I would try 100 or 200 Gbytes), start the processing before leaving office, and you could have results next day in the morning.
I have to process a large volume of data ( feature maps of individual layers for around 4000 images) which sizes more than 50 GB after some point. The processing involves some calculation after which around 2MB file is written to the HDD.
Since the free ram is around 40GB my process crashes after some point. Can anyone suggest a better approach to either divide or process this 50GB data such that the computation can be made within the available ram. For e.g. some in memory compression approach
I am just looking for hints to the possible approaches to this problem.
Thanks