Should I prevent augmented data to leak to the test/cross validation sets
Yes! 

The accuracy of your algorithm should be tested on images that you expect to receive. This would not include data that has been rotated, shifted, blurred, etc. However, augmenting your data in such ways can and usually will lead to better results.

Split your data first. Then augment only the training data while staying away from the validation data. Only use your testing data when you are confident that your solution leads to good results on your validation set without overfitting it.