How does rotating an image and adding new 'rotated classes' prevent overfitting?
How can data augmentation reduce overfitting?

You write that you can already maybe see how data augmentation can help prevent overfitting in general, but it sounds a bit uncertain and it's still asked in the title of the question, so I'll address this first:

Generally, when we use Machine Learning for classification problems, we would ideally learn a classifier that can perform well on a population. An example of a population would be: the set of all handwritten characters in the entire world. Generally, we don't have that complete population available for training, we only have a (much smaller) training dataset. If a training set is large enough, it might be a good approximation of the true population we're interested in (a "dense sampling" of the space we're interested in), but it's still just that; an approximation.

We say that a learning algorithm is overfitting if it performs singificantly better on the training set than it is on the population (which we generally approximate again using a separate test set). 

Now, data augmentation (like adding rotations / translations of images in the training set to the training set) can help combat overfitting because it bridges the gap between training set and population. The population (all handwritten characters in the entire world) will likely include characters at various offsets from the middle (e.g. translations) and at various rotations. So, data augmentation is simply adding more examples (and possibly more varied examples) to our training set, which importantly are considered to be a part of the population we're interested in. If, for example, the population we are interested in were only the set of all handwritten characters at a specific position in the image (e.g., centered), then augmenting the dataset by adding various translations would not help; we'd be adding instances that are outside the population we want to learn about.



Why doesn't adding extra classes for rotations cancel out augmentations?

There are two possible explanations I can come up with:


Maybe the "extra-class" rotations are different from the "data augmentation" rotations.


Here is the exact quote that's relevant from the paper:


  "To reduce the risk of overfitting, we performed data augmentation by randomly translating and rotating character images. We also created new classes through 90◦, 180◦ and 270◦ rotations of existing data."


That first sentence is not 100% clear in my opinion. I imagine the translations they use for data augmentation are relatively small (e.g. offsets of a few pixels), so maybe the rotations they use for data augmentation are also only "small" rotations (for example, between -10◦ and +10◦). The "larger" rotations (multiples of 90◦) described in the second sentence may then no longer be a part of the "data augmentation to reduce the risk of overfitting" in the first sentence; they're simply parts of a different action performed to increase the number of classes in the dataset (and, I imagine, for each of these larger rotations they may again perform "smaller rotations" for data augmentations).

This explanation is kind of hypothetical though, it's not 100% clear from the paper exactly what they mean here in my opinion.


"Overfitting" can have a slightly different interpretation in the case of one-shot learning than in traditional learning.


Note that this paper is about "one-shot learning", where the goal is to be able to classify accurately after being presented only a single example ("one shot") of a never-before-seen class. In such one-shot problems, you could in some sense say that an algorithm might "overfit" to the "distribution of classes" if it can only perform one-shot learning well on a certain set of similar classes, but not on others. 

For example, if you only train one-shot learning on a set of handwritten characters that are "upright" (close to 0 rotation), your algorithm might be able to perform well in terms of one-shot learning when presented with new classes (new handwritten characters) that are also upright, but might be incapable of proper one-shot learning when presented with new classes (new handwritten characters) that are upside-down.