How to maintain accuracy with more classes?
There can be multiple explenations.

But here are some:

Images in train test and valid that are poor for some classes are not representative in train (poperties of these objects are not to be found hence you cant predict on something that you did not atleast partially learn on)- covariate shift

You say adding data is problematic and I understand but just note that more data beats everything else by a mile. Seriously. Hence augment the data, A LOT!. And make sure that for every class properties are representative and to be found in every train test and valid set.

I can start talking about architecture but these are marginal benefits at this point, make sure you have this point right. Without that every discussion about architeture if futile.