How to implement a GridSearchCV custom scorer that is dependent on a training feature?
You should be able to do this, but without make_scorer.
The &quot;scoring objects&quot; for use in hyperparameter searches in sklearn, as those produced by make_scorer, have signature (estimator, X, y).  Compare with metrics/scores/losses, such as those used as input to make_scorer, which have signature (y_true, y_pred).
So the solution is just to define your own &quot;scoring object&quot; directly, and reference the passed X to do the computation you want.  See The User Guide for some more details.