Training a model purely on weak labels
Based on a quick read of the paper linked in the comment:

I just don't understand if your model (BERT or not) is only trained on these weak labels then you are treating them as &quot;ground-truth&quot;,

Correct, but only for the training: the model is trained to recognize the labels obtained with a &quot;quick and dirty&quot; method.

and more importantly, don't you already know how to create &quot;ground-truth&quot; (by the rules-based system) ??? What's the point of the 2nd step?

No, because the real ground truth they are interested in is not those from the &quot;quick and dirty&quot; method. If they were, it would indeed be sufficient to run their rule-based system. The goal is to predict the labels obtained in what the authors call the &quot;Gold Standard Corpus&quot;, which was manually annotated and never seen by the model.
Typically the quick and dirty method will result in some classification errors. The point of tuning the model with these labels is to see if the model can extrapolate from these low-quality labels to high-quality labels. This ability to generalize beyond the training data is based on the underlying semantic information contained in the original BERT-like model. For example this model might be able to associate a specific sport like swimming with &quot;physical activity&quot;, even though the weak supervision doesn't contain this association.