How to approach the PlantVillage dataset?
I think this approach should work. So we need to clasify the plant and its disease. As mentioned, the disease maybe be associated with the plant's species.


Build a standard CNN or Siamese CNN for identifying the plant species. The model should be able classify the name/species of the plant even when it has a disease of some another plant ( The classification should not be baised upon the plant's disease ).
So, now, after the above model is ready, we build a 2nd model for disease classification. But the classification of the disease maybe be dependent on the plant species. So, for the 2nd model we will have 2 features namely, the image ( raw sample ) and the output of the 1st model ( plant name ). So, the 2nd model's prediction will be based on these two features.


The basic idea will be:

Model 1: input = image, output = plant name.
Model 2: input = ( image, plant name ), output = disease name. 

So this is a system of models. That's easy if you are working with Keras or PyTorch.

Hope that helps.