How to cache big data in memory (efficiently) in complex variables across executions of Python scripts?
I am trying to call (from Java Spring beans) Python Pytorch scripts that contains trained neural networks for the use: my Pytorch neural networks are neural functions that accepts state, encodes it and returns the action, decodes all this is according to learned/trained policy.
So - each time when I am trying to invoke Python script I should construc torch.nn, load weights and biases from some external store (DB, file) and then execute this nn to get single answer, pretty expensive operation.
How can I keep torch.nn instance (with loaded weights and biases) in memory and make it available immediately for each execution of Python script?
Memcached is not the solution, because it can keep string or binary values only and it is quite expensive to serialized and deserialize torch.nn instance. One suggestion was Tensorflow Serving, I am currently researching it, so - I don't yet know whether this is the answer.
It is quite possible that Tensorflow has some caching technologies which I can use for the Pytroch as well?

If I understand your use case correctly, what you need is a model server that keeps the model loaded and ideally also handles any exceptions from incorrect data.
One rather straightforward way to transform your inference script into a tensorflow-serving-like callable service is the python library flask. Another way seems to be a new tool called torchserve.
