Shaping data for ConvLSTM for many-to-one image model
The problem was for each ConvLSTM layer I was using keep_dims = True which means that the number of dimensions in the input is reflected in the ouput. To obtain a many to one model I just needed to set keep_dims = False. I was then able to manipulate the size of the output using dense layers and convolutions. In this way I could have a video sequence as an input and a single 2D image as the output and I was able to obtain the relationship: 

input shape : (10, 50, 256, 400, 1)
output shape:     (10, 256, 400, 1)

Where I was able to get a 2D segmentation for each fragment in the video sequence.

It is not exactlly what I needed, but was much better than getting a segmentation result for every frame in the video sequence.