Should I oversample my validation data to get better F1 score and PRC?
what you encounter are real-world problems rarely taught in classes.

For training, I would test SKLearn's class_weight = &quot;balanced&quot; or class_weight={0:0.995, 1:0.005}. It's a very robust technique.
For testing, you can't fiddle with the class_weight. It's meant to simulate the real-world data.
Make sure you don't overfit. E.G. Decision Tree max_leaf_node 5-10 OR (not both) max_depth 3-5.

Your results aren't that bad. A precision 1 of 0.03 is &quot;low&quot;, but you only had 36 labels. Roughly speaking, the model labeled 1,000 as 1 (100/0.03 * 36). i.e. 97% false alarm. But with a good recall, you got most of the labels.
Putting it differently: in a dataset of around 8,000 you got 1,000 labeled as 1. That's around 12%. Out of those 1,000 there are about 30 real 1's. You get most of them (around 75%).
Imagine this was Churn Prediction. You don't want to lose customers. Without ML, you might send all your 8,000 customers a discount of 10%. That's expensive. With ML, you would only send 1,000 customers a discount and get most of them wanting to leave. That's a strong improvement over no model.
The same argument applies to many other cases such as Predictive Maintenance.