Why does adding data augmentation decrease training accuracy a tiny bit?
The obvious reasons why data augmentation might reduce the train accuracy is -
As you know, deep learning models are data hungry. If the model don't get enough data to recognize the patterns then it will try to memorize the dataset. Bigger models tend to memorize the data instead of finding patterns, because they are big enough to do so. When model memorizes the training data it will definitely perform very good on training set and poorly on validation set.
And as you said data augmentation is a regularization technique. In regularization your model weights are penalized more to make sure they don't over fit. As a result, your model cannot perform well on training set (depending on how much regularzation is used), but as an advantage model will try to find generalized patterns in the dataset and this will also help at the time of validation.
I could find one research paper which has exhaustive experiments about data augmentation and regularization.