Imbalanced Dataset (Transformers): How to Decide on Class Weights? I'm using SimpleTranformers to train and evaluate a model.
Since the dataset I am using is severely imbalanced, it is recommended that I assign weights to each label. An example of assigning weights for SimpleTranformers is given here.
My question, however, is: How exactly do I choose what's the appropriate weight for each class? Is there a specific methodology, e.g., a formula that uses the ratio of the labels?
Follow-up question: Are the weights used for the same dataset &quot;universal&quot;? I.e., if I use a totally different model, can I use the same weights or should I assign different weights depending on the model.
p.s.1. If it makes any difference, I'm using roBERTa.
p.s.2. There is a similar question here, however, I believe that my question is not a duplicate because a) the attached question is about Keras where my question is about Transformers, and b) I'm also asking about general recommendations of how weight values are decided where the attached question is not.
I am not sure about the model you are using but I might explain what the procedure is for ML in general. You have three &quot;vanilla&quot; solutions for coping with unbalanced supervised dataset.

Reweighing class label so that there is the same number (calculated as sum of weights for given label) of samples per label. For example if a label with maximum number of samples has $n_{max}$ samples, and some other class has $n_i$ samples, then you would assign a weight $w_i=\frac{n_{max}}{n_i}$.
Undersampling - a basic procedure that gets rids of all the additional samples so that you end up with a balanced dataset.
Oversampling - creating copies of the unbalanced classes (with less samples than the $n_{max}$)
Hope that helps,

Max