Why is video classification still not that accurate?
I've used py-faster-rcnn on video feeds and found it to be pretty usable in a sunny environment. Although on a GTX660, I could only process one image every 600ms, which may highlight a limitation. It still takes a lot of time to compute an inference on a deep net, so for a minimally fast framerate, of 10fps, you're still limited to a speed of 100ms.

There might also be other artifacts due to camera dynamics such as image blurring, sensor saturation, or low light levels that might be more present in video feeds of natural scenes as opposed to the fairly well exposed images in static image datasets.