Quantifying binary classification uncertainty
First things first: be sure to use a validation set in order to determine the optimal threshold, do not use the final test set.
Another general point: it depends what you mean by &quot;false alarm&quot;, but in general you should not assume that it is possible for the classifier to keep the error rate under a predefined value (unless you have already done an experiment which shows this, of course). If by &quot;false alarm&quot; you mean only one type of error then it is usually possible to keep one type of error under a threshold at the expense of the other type of error. For instance if the goal is to minimize false positive errors, then one can let the classifier make as many false negative errors as necessary in order to keep the false positive rate under the threshold.
Now about quantifying uncertainty:

The first approach that you propose makes sense, but as you said it's not symmetric and also it wouldn't allow to convey uncertainty in any practical way. Ideally you'd want to be able to say something like &quot;there's an X% chance that the prediction is correct&quot; but you cannot obtain that from a difference with the threshold.
As far as I understand your second approach goes in the right direction but I'm not sure if the details are clear. So what you could do is to determine experimentally the rate of error (either FPR or TPR depending on which side of the threshold) for different intervals of predicted $\hat{y}$. For example if $T=0.9$ you can calculate the FNR for every interval $[0.0,0.1]\ [0.1,0.2]\ ...\ [0.8,0.9]$. This way for any predicted $\hat{y}$ you can provide the corresponding error rate.