Research paths/areas for improving the performance of CNNs when faced with limited data
Some research areas that come to mind which can be useful when faced with a limited amount of data:

Regularization: Comprises different methods to prevent the network from overfitting, to make it perform better on the validation data but not necessarily on the training data. In general, the less training data you have, the stronger you want to regularize. Common types include:

Injecting noise in the network, e.g., dropout.

Adding regularization terms to the training loss, e.g., L1 and L2 regularization of the weights, but also confident output distributions can be penalized.

Reducing the number of parameters in the network to make it unable to fit the training data completely and thus unable to overfit badly. Interestingly, increasing the number of parameters for large models can also improve the validation performance.

Early stopping of training. For example, if one part of the training set is set aside and not used to update the weights, training can be stopped when the observed loss on this part of the training set is observed to start to increase.



Generating new training data:

Data augmentation: Ways to augment existing training examples without removing the semantics, e.g., slight rotations, crops, translations (shifts) of images.

Data interpolation, e.g., manifold mixup.

Using synthetic data, e.g., frames from video games or other CGI.



Transfer learning: When you take a neural network that has already been trained on another, much larger dataset of the same modality (images, sounds, etc.) as your dataset and fine-tune it on your data.

Multitask learning: Instead of training your network to perform one task, you give it multiple output heads and train it to perform many tasks at once, given that you have that the labels for the additional tasks. While it may seem that this is a more difficult for the network, the extra tasks have a regularizing effect.

Semi-supervised learning: If you have much unlabeled data that labeled data, you can combine supervised learning with unsupervised learning. Much like with multitask learning, the extra task introduced by the unsupervised learning also has a regularizing effect.


Other interesting methods can be found in systems that perform one-shot learning, which inherently implies very little training data. These systems often uses slightly modified network architectures. For example, facial recognition systems can learn to recognize a face from only a single photo, and usually use a triplet loss (or similar) of a vector encoding of the face, instead of cross-entropy loss of the output of a softmax layer normally used for image classification.
Zero-shot learning also exists (e.g., zero-shot machine translation), but this is a completely different type of problem setup and requires multiple data modalities.