Applying ConvNets to classify motion/video data
Signs are the visual equivalent of "words", and just as words can be decomposed into smaller parts, so can signs be broken down into smaller meaningful/useful units. For example, in American Sign Language (ASL), signs are decomposed into parameters such as handshape, orientation, movement, location on the body. Each parameter can take on a finite set of values, for example handshapes can be closed fist, open hand, cocked index, etc. I would expect a neural network would need to learn these parameters and their values, while learning to ignore distractor and nonsense configurations. Perhaps initial training includes learning to differentiate between signs and non-signs. And just as there are different spoken language, there are different sign languages as well. Moreover, there exist distinct sign language dialects like Black ASL that will need to be taken into account by your system.