Data augmentation when using flow_from_directory in CNN
Augmentations often rely on the nature of your data. Imagine if an the result of an augmentation would be logical in your context.

For example let's say you have a cats vs dogs dataset. The images here can be flipped left to right. On the other hand in the MNIST dataset it makes no sense to flip the images at all (what good would it be feeding a flipped '3' to your model). Some augmentations might actually confuse the model (e.g. an upside down '9' might look like a '6').

Furthermore, if augmetation is making your model worse, try smaller augmentations (e.g. stick to affine transformations with small ranges - $[-5%, +5%]$ rotation/translation/scaling).