Use deep learning to rank video scenes
It’s seems like quite challenging problem; at least you would need quite a lot of annotated data and computational power.

The approaches/optimizations you could consider:


To make scene change detection and take short piece out of each
To introduce some kind of “novelty” metric and try to maximize
it to get most different parts of video   
To convert video to kind of vector with existing solution like
r-cnn and yolo and then process it with recurrent networks.   
The task seems to be very close to video capturing/summarization, you
can take inspiration there  
Also, the attention approach might be handy, look, for example
self-attention for video,
semantic attention for
video