How to label a video for supervised learning?
Try Computer Vision Algorithms and Applications by Richard Szeliski and label propagation in video sequences.
To get a good grasp on the topic also Supervised Sequence Labeling with Recurrent Neural Networks. I proposed these resources as they cover the labeling, machine learning and are very in-depth materials (book, dissertation and the paper about propagating labels).

Video is the natural extension of the image, but to get fully working methods we also have to account for scene movement and changes. And I belive that in the video point is referenced by $x, y, t$.