How to control the amount of positives in classification?
as desertnaut was saying, I think you are talking about class imbalance.

If you meet class imbalance "in the wild" there are a couple of ways to tackle this. Your approach of creating a custom metric would fall under cost function based approaches. You can use cost functions that handle false negatives as worse than false positives (which is usually the case when we're talking about class imbalance).

Another way would be to use sampling based approaches. These are rather simple and I personally tend to use these more often since it's easier to play around with aka experiment with them. Two extremes that can be combined as you wish:


Boosting one class / Oversampling: Look at the distribution of Class A and Class B. Boost the class you want to have more effect to have more datasamples changing your algorithm.
Undersampling: Look at the distribution of Class A and Class B. Remove datasets of the class you want to have less effect of your algorithm.


In reality I usually go for a combination of both to have more variance in my data. For example boosting Class A by 110% and reducing Class B to 80% to achieve a balanced dataset (although it sounds to me that you want to have an imbalance just the other way round. Take care about bias with that though)