Structure of Convolutional Neural Network to analyze a sequence of frames I want to apply a CNN to a series of image sequences to classify that sequences of frames/images in two groups/categories. We are in a binary classification problem. My dataset is composed by a lot of 'batches' of frames. For example, each batch of frames could be compose of 20 frames of 64x64 pixels. One important thing is that the order of that 20 frames is important. If you shuffle the order of that 20 frames the output could change.

For all this, I want to create a CNN to solve this binary classification problem. I'm usig Keras and TensorFlow. 

What is my question? Well, I'm not sure if I have to use a TimeDistributed layer or not. The input shape of the neural network is the following one: (20, 64, 64, 1), whose meaning is: 20 frames with a 64x64 size (1 channel - grayscale). Should I use a TimeDistributed layer?
I'm not sure if I have to use a TimeDistributed layer or not


You definitely don't have to use TimeDistributed. You have other choices, that may be equally valid, depending on your data:


Flatten your example data into 81920 features per example and use a simple Dense layer.
Use a Conv3D layer.
Use some form of RNN, such as LSTM.


From your data description, I would expect either a TimeDistributed or 3D CNN based approach to be a good first bet. Intuition suggests that the CNN would work better if there was minor change between frames (because it has capability to directly find subtle frame differences), the TimeDistributed approach would work better processing larger changes (because it will ignore frame differences until the fully-connected layer).


  Should I use a TimeDistributed layer?


Only you can answer this, by trying it and measuring performance of your classifier. However, it should function correctly, and intuition suggests it would be a good choice if your frames are in sequence but visually disjoint.