Unbalanced dataset on image classification, is it better to lose samples and balance it?
Instead of downsampling the dataset, you should try using the Upsampling technique. Sometime downsampling leads to loss of data. Use augmentation techniques to increase the size of your dataset.