Is there a general guideline for experience replay size, and how to store?
If yes, how can I go about this without that much RAM?


Your value of 56GB seems correct to me, assuming you include a multiplier of 4 for the "4 frames per state representation" used in the DQN/Atari paper. However you should note that in the original paper, the images were converted to greyscale and downsampled to 110×84 prior to use in state representation. This made them 16 times smaller than the frames in your problem, so the whole data set would fit into 4GB.

If you must use larger images, you could store them on disk - maybe in a database - and load on demand. That may unfortunately create an I/O bottleneck that slows learning, although you should still be able to work with it. You can parallelise fetching from the database for mini-batches with the learning process, and this is similar to the mini-batch generators used for things like ImageNet training. You can also work on improving disk performance using optimisations such as parallel disk arrays or SSDs.

You could also pre-process the frames using a hidden layer embedding from a generic computer vision network trained on e.g. ImageNet, and store that representation, not the raw pixel values. This may limit self-discovery of important low-level features by the learning agent, but then again it may still be worth a shot, as lower-level image features are often very similar across different problems, and transfer learning in e.g. image classification has been quite successful using that approach.

More likely practical answers used by RL researchers at least up to a certain scale is one of:


Store less states in replay memory. Base the size of replay on the memory you have available. Yes this may compromise the learning, but there is no special magic about the number 50,000 and if you are optimising resource use you may have to decide between how efficiently a system learns with 10,000 fast replay memory size or 50,000 slower I/O-based replay memory size.
Buy more memory. The big name research labs working in deep RL are funded well enough that they can afford to throw money at this problem. One estimate for how much AlphaGo Zero hardware cost is $25million, so you can imagine that loading a few machines with 128GB+ RAM if they thought it necessary for any reason on any other flagship project would not be a major blocker.


If you look at what OpenAI are doing at the cutting edge of video game playing, you can see that their hardware setup is equally monstrous. It is not clear whether they have an issue with storing experience as they use a different algorithm, or needing RAM in general, but if they did, it is also clear they could quite happily finance maximum RAM on their training rigs.

Do note I am not a RL researcher myself (just a hobbyist) and have not asked anyone else what they would do when faced with this problem.