Can ConvLSTMs outuput images?
If you only need 5 frames to predict the next frame, then I'd recommend a U-Net architecture, wich is basically a CNN encoder/decoder network in which the decoder uses the intermediate features produced in the encoder as well as its own features to produce an output image. Also, in additional to using a conventional L2 loss for the output image, you can always add an additional GAN loss to make the image look more realistic.

If using a longer history of frames can help, then I recommend taking a look at "Recurrent Environment Simulators" and combining it with the ideas above.

Hope it helps!