Custom thresholds on categorical classification
What you are talking about sounds a lot like meta-learning (use outputs of several models and use another model to train on that predictions). You can read through more detailed example here (link to machinelearningmastery)
Should you do it for your binary classification task - no. (this in not a strict &quot;no&quot;, but generally you should not do it given your task can really benefit from such approach, but this has little to do with classification threshold)
About threshold search, you can check this thread with quite good answers:
https://stats.stackexchange.com/a/386433/232020
From your question, it sounds like you are dealing with the binary dataset, not sure what you mean by  Unfortunately, the method seems not to be scaled on categorical classification.