Improving misclassification for one class in a multi-class classification task
The main problem is that there are too many Escas in your dataset. If you look at the confusion matrix, the Esca column gets predicted (wrongly and correctly) much more that the others. This is clearly a symptom of a skewed data set.
Try augmenting your images to generate a larger 'super-dataset'. Then sample a 'sub-dataset' from that 'super-dataset', such that it has an equal distribution across all classes. Train on the 'sub-dataset'.
If you can't augment your data to have a better distribution across the 4 classes, here are some ideas:

Modify the loss function to more aggressively penalize Black Rots classified as Escas.
Split into two networks; the first differentiates between
Leaf Blight-Healthy-Black Rot/Esca, and the second differentiates between
Black Rot-Esca.