Custom metrics for unbalanced classes problem in RandomForest or SVM
I think I understand what you are trying to do. First, let me attempt to obtain clarity on two concepts that may be getting conflated (if not for you, then for other users).

When a random forest is being fit, it typically uses entropy or gini impurity for categorical outcome variables, and mean squared error or mean absolute error for numerical outcome variables -- this is the model building step. However, when the fit model is being evaluated, the scoring method is used to understand the models performance -- this is the step where we learn how accurate the model might be.

I've seen that sklearn allows specifying a custom scoring method, but as far as I know, I don't believe you can specify your own criterion parameter in the RandomForestClassifier or RandomForestRegressor methods/APIs.