How to pretrain/select CNN for Biomedical Video Analysis
Data
I am trying to train a model in the biomedical domain with a rather specialized task (flow prediction).
My input consists of video clips and I would like to predict either a single image or a video. I have pixel wise ground truth for every frame of the video but the number of videos is very limited.
I do however have a lot of unlabeled data of related scenes, which is why I considered transfer learning/pretraining of some sort.

Architecture
Architecture wise I am considering a CNN, RNN combination where the CNN provides a representation of the input frames for the RNN to learn about the temporal relationship between input frames.


Now my question is: What kind of CNN do I use and on what do I pretrain it? Since I am working with biomedical data I would assume image net as well as most other image data sets do not really help as the image content is very different. Are there any data sets/tasks/networks I could use for this purpose?

Some people recommend shallow autoencoders to pretrain networks if there is not much labeled data, but lots of unlabeled data. A reference would be Géron's book "Hands-On Machine Learning with Scikit-Learn and TensorFlow", but there are also many articles and tutorials online.
