after augmentation validation accuracy going down?
You can not compare validation accuracies if the validation datasets are fundamentally different.

What you were doing initially was to validate your model on a limited set of images. Augmentation is applied to make your dataset more representative of your deployment environment. 

If in your deployment environment you expect images to be drawn from the same subspace as your augmented dataset, the validation accuracy of the first case will not be a true representative of your model's real world performance.