Expected behavior of adversarial attacks on deep NN?
This seems to be a known problem, and intuitively seems reasonable. You might be interested in the paper Adversarial Training Can Hurt Generalization.
The authors suggest that this might be because training on the perturbed data requires the model to learn more robust features, which means more samples are required to obtain performance comparable to a model that is not adversarially trained.
You could try collecting or generating additional samples to see if this leads to an improvement. They also mention that in their experiments on the MNIST dataset, using Xavier initialisation led to a significant benefit, so you could experiment with that too.