Neural Networks: Incorporate feature-specific knowledge
This 2013 paper integrates deep learning models with structured hierarchical Bayesian models, for the net to learn novel concepts from very few training examples. It shows encouraging results.

Another approach is to import a general-purpose top-performing neural net, and replace the top fully connected layers with initialised ones (including an output layer that corresponds to the number of classes in your specific task). OverFeat is an example, for image classification - it was trained on ImageNet.