How to Classify an Image in a Class and a Subclass?
The simple solution

Have one classifier which can classify all the combinations:


Fresh banana
Rotten banana
Fresh mango
Rotten mango
...


The good solution

Or have one network with multiple outputs. See the master Thesis of Marvin Teichmann for example. 
https://arxiv.org/abs/1612.07695

So one path of the network would be:


Fresh vs Rotten


The other path of the network would be 


Banana vs Mango vs ...


The difference to the simple solution is that you can give a probability to the cases, e.g. for the ground truth "fresh banana" you might say 99% banana but only 42% fresh.

The obvious solution

This one should not be done: have multiple classifiers