Face recognition system with a sparse dataset
You can apply transfer learning.  Leverage one of the existing datasets and deep learning networks that can classify faces e.g. celebrities.  

Add/remove layers on top of the feature extraction layers and use your small data set.  Provided that the base dataset / network is large and rich enough you should be able to apply it to your dataset.

See https://cs231n.github.io/transfer-learning/