Can an object's movement (instead of its appearance) be used to classify it?
Machine learning systems work based on input data. The form of that data is irrelevant. It may need some configuring on the technical side of things, but the general ability to learn from the dataset remains the same.
ML systems are built to learn how to interpret data, without you needing to predefine exactly what it should be looking for.

But even then, you can reason that videos are no different from images. A video is essentially nothing but a flipbook of images. If you were to paste every frame of the video in a long sequence, you would have one big picture.
The rest should follow as usual: if a machine learning system can be taught to interpret images, it can evidently also interpret these "frame sequence" images, which means it's able to interpret the dataset that we colloquially call a video.

That being said, since a video contains much more information (a 5 second 30fps video is 150x as much data as a single image with the same resolution), the required learning process increases exponentially.

So yes, it's perfectly possible, but it will require more processing power and training as there are more input variables to account for.



There are ways to reduce this increase in complexity. For example, to check if someone is drunk or not, rather than push the entire video through the learning algorithm, you instead preprocess the video to figure out the person's gait or even just their footsteps (relative location and timing) and only push this processed data through.

This dramatically cuts down on the complexity of the network needed; at the cost of requiring preprocessing which may introduce issues (bad preprocessing = bad learning).