Sample K representative frames within a video
The question misses a lot of detail, so I will try to make an educated guess. 

I don't know of any specific algorithm for the task you are trying to achieve, but the first step towards your solution would be perhaps the computation of the average of the magnitude of the optical flow throughout the whole duration of a video. Why: you said that an interesting feature in your videos is the fact that the camera moves less while filming something interesting. Then, the magnitude of optical flow vectors will be expectedly smaller. If you plot the average magnitude of optical flow throughout time of the video, you could start to see parts where there is potentially something interesting here and there.

Edit: look-up at the algorithm used by Youtube for "smart" thumbnail selection. E.g. https://stackoverflow.com/questions/28263511/smart-video-thumbnail-generator-algorithm