Applying ConvNets to classify motion/video data How would someone go about using deep learning to classify sign language gestures? For example, suppose I had video files of many different gestures. For any given gesture, I might have many videos of it, and each video would be comprised of many frames. When trying to classify MNIST digits in images the dimensions of the input are comparatively simple: height, width, and RGB channels. How would gestures (multiple frames over time) be accounted for? Would time be a fourth dimension? What should the neural net's architecture look like so as not to overfit? Should I use something instead of a convoluted neural network?

Edit:

I know that there are probably clever ways to hand-code predictors for sign language, but I'm more interested in how to architect the neural net and take advantage of the time component of the data (where there is value in the transition of video frames over time). Classifying gestures is a simplification of the actual problem I'm trying to solve, so I'm looking for an approach that would be generalizable to other types of problems where it might be necessary to look at many frames of the video to predict the target variable.
Signs are the visual equivalent of "words", and just as words can be decomposed into smaller parts, so can signs be broken down into smaller meaningful/useful units. For example, in American Sign Language (ASL), signs are decomposed into parameters such as handshape, orientation, movement, location on the body. Each parameter can take on a finite set of values, for example handshapes can be closed fist, open hand, cocked index, etc. I would expect a neural network would need to learn these parameters and their values, while learning to ignore distractor and nonsense configurations. Perhaps initial training includes learning to differentiate between signs and non-signs. And just as there are different spoken language, there are different sign languages as well. Moreover, there exist distinct sign language dialects like Black ASL that will need to be taken into account by your system.