How to control the amount of positives in classification? I have a basic, yet quite complex problem to solve right now. Let's say we have a training set of 20,000 samples in my training set, out of which 3 to 4% is flagged as "True", the rest is flagged as "False". I want to train a classifier (typically XGBClassifier or LGBMClassifier are those I worked a bit with).

What I'm currently doing is that I find the best parameters using GridSearchCV. But my objective is to minimize the amount of samples I would flag as "True" once I try on the test set. Should I train the algorithm with a typical F1 metric, and ONLY THEN find the best threshold that suits my needs? Or should I create a custom metric that would implicitly force the algorithm not to flag too many samples as positive?

Hope this is clear!
as desertnaut was saying, I think you are talking about class imbalance.

If you meet class imbalance "in the wild" there are a couple of ways to tackle this. Your approach of creating a custom metric would fall under cost function based approaches. You can use cost functions that handle false negatives as worse than false positives (which is usually the case when we're talking about class imbalance).

Another way would be to use sampling based approaches. These are rather simple and I personally tend to use these more often since it's easier to play around with aka experiment with them. Two extremes that can be combined as you wish:


Boosting one class / Oversampling: Look at the distribution of Class A and Class B. Boost the class you want to have more effect to have more datasamples changing your algorithm.
Undersampling: Look at the distribution of Class A and Class B. Remove datasets of the class you want to have less effect of your algorithm.


In reality I usually go for a combination of both to have more variance in my data. For example boosting Class A by 110% and reducing Class B to 80% to achieve a balanced dataset (although it sounds to me that you want to have an imbalance just the other way round. Take care about bias with that though)