How to arrange the dataset/images for CNN+LSTM
I'll expand my answer if there is any interest. I propose building a simple test model using Vgg and two custom fully connected layers that end with a single sigmoid per the document below. If you prefer early fusion I recommend using stacked black and white images.
https://www.cs.cmu.edu/~rahuls/pub/cvpr2014-deepvideo-rahuls.pdf



if you then want to go the route of LSTM, then this graphic from Karpathy will help. Use the many to one architecture. The inputs to the RNNS or LSTMS vary by their implementation. You would most likely want to pass the final convolutional layer though a fully connect layer and then into the RNN/LSTM. https://karpathy.github.io/2015/05/21/rnn-effectiveness/