Class weights for imbalanced data in multilabel problems
I think what you are looking for in this case is "cost-sensitive classification" - you can look it up on google scholar to find some papers. You'll likely have to define "costs" for each type of misclassification.

Alternatively, if you are doing one-vs-rest classification, you can reweight/upsample/downsample differently for each class.

If your dataset is small, you could also try to set weights as the objective of some constrained optimization objective that would make them more balanced.