Custom thresholds on categorical classification When assessing a binary classification task, it is possible to search for particular threshold in order to have better score on some metrics (f1,recall,etc) through numerous methods. Unfortunately, it looks like the method cannot be applied on categorical classification (more than two classes) task. I've thought about training a Simple Classifier (SVC,Log,...,Tree) on top of a already trained model in order to find best thresholds to apply on outputs to maximize similarity of results.
My proposed workflow is to train a model on training set and evaluate it, then use outputs on training set of this model as entry for another model which may learn how to handle threshold to maximize similarity to ground truth on training set (all of those steps being validated using the same strategy).
I've never heard about it and I want to know whether it makes sense ?
What you are talking about sounds a lot like meta-learning (use outputs of several models and use another model to train on that predictions). You can read through more detailed example here (link to machinelearningmastery)
Should you do it for your binary classification task - no. (this in not a strict &quot;no&quot;, but generally you should not do it given your task can really benefit from such approach, but this has little to do with classification threshold)
About threshold search, you can check this thread with quite good answers:
https://stats.stackexchange.com/a/386433/232020
From your question, it sounds like you are dealing with the binary dataset, not sure what you mean by  Unfortunately, the method seems not to be scaled on categorical classification.