Design a branching CNN
I'm not sure what the state of the art is right now, but I had a similar problem for which I designed a neural network. What I did was make a model that predicted the class probabilities from the inputs (in your case the image) and then made another model that took both the normal inputs and my probability predictions. I saw a very large increase in the accuracy for the second part after I had fed the class predictions into the feature prediction section.

You could design this as one model with two outputs where the class is an intermediate layer but still an output. Your loss function would be adding the loss for the class prediction and the feature prediction. Weighting the loss for the class prediction would cause the model to lean more towards correctly predicting the class over the other features. If you must absolutely predict the class first and then the features, you can first train the model on the class loss, then train it again with the feature loss after freezing the weights associated with/preceding the class predictions.