feature importance after classification
You may use Permutation importance
 - Get your base-line score
 - Permutate a feature values. May replace with Random values
- Calculate the score again
- The dip is the feature importance for that Feature
- Repeat for all the Features

....Breiman and Cutler also described permutation importance, which measures the importance of a feature as follows. Record a baseline accuracy (classifier) or R2 score (regressor) by passing a validation set or the out-of-bag (OOB) samples through the Random Forest. Permute the column values of a single predictor feature and then pass all test samples back through the Random Forest and recompute the accuracy or R

To check the importance for the individual Class i.e. 0/1
Extrapolate the same to check if the increase is more for False-Positive or False-Negative.
Read Beware Default Random Forest Importances for more explanation.
Few other quotes from the page-

Any machine learning model can use the strategy of permuting columns to compute feature importances. This fact is under-appreciated in academia and industry.


The permutation mechanism is much more computationally expensive than the mean decrease in impurity mechanism, but the results are more reliable. The permutation importance strategy does not require retraining the model after permuting each column; we just have to re-run the perturbed test samples through the already-trained model.