Replace human judgements with mathematical approach/theories I would like to give a context of what I did.

1) Let's say there are two dictionaries (dict A and dict B) each containing a list of words/terms as shown below.



2) Now my task is to find matching words for dict A in dict B

3) I use an automated tool (fuzzy-matching/similarity) to do the above task and the output looks like below



4) Once I get the output as above, you can see that there are some records with match % less than 100. It is totally possible that dict B didn't have the exact matching term. It's fine.

5) So, what I do is review terms that have match % less than 50. Meaning I take those terms (that are less that are 50% match) and check for a related term in dict B again. Doing this, I am able to update the output like below. Because we know through human experience that sore throat lozenge and strepsils are related (matching is better now when compared to earlier where it was mapped to orange (totally irrelevant)). So this problem is more of a semi-automated task rather than full-blown ML task



So, now my question is (not on NLP or ML but below)

1) But how can I prove that choosing 50% as the threshold for manual review is the right one? Because this is a subjective thing/ based on individual judgment. Meaning I could have chosen 30% or 40% as well, it could have saved my time in manually reviewing terms

2) Meaning, this 50% isn't written in stone but what I am looking for is some theory/mathematical/statistical approach through which I can arrive at this threshold value rather than based on my judgment/subjective which I cannot defend/justify?

Can you people share some views/techniques on how can this be done in a systematic approach?
When it comes to choosing a threshold, I can see 3 approaches:


Make an educated guess


This is what you are currently doing. You pick a value and would need to argue why this is a reasonable threshold. Obviously, the argument is as strong as the assumptions you make.


Unsupervised way


If you compute the matching score for all pairs between A and B, you can analyze the histogram of these scores. Most likely, you will have quite a lot of scores near 0 and a few scores near 100. Then, you can decide to pick a threshold. This itself can be done in different ways. You can pick the median/mean matching score, this becomes close to option 1 but at least, the number comes from a specific mathematical concept. You can use the Jenks-Fisher algorithm (here is a Python implementation). In summary, the algorithm will find a threshold that splits your data into clusters which minimize the intra-cluster deviation. This would be better than the median/mean since it's expected that the data will be quite skewed.


Supervised way 


If you somehow have access to which matches are confirmed/overturned by humans, you could use entropy/information gain to find the best threshold. This is similar to building a decision tree of depth 1 (a decision stump), where your input is the matching score and the target is a binary variable (whether or not a human says this is a match).

So your data would be something like this:

score, is_match
0.0, 0
0.1, 0
0.2, 1
0.3, 0
0.4, 1
0.5, 0
0.6, 0
0.7, 0
0.8, 1
0.9, 1
1.0, 1




My personal recommendation would be to use the Jenks-Fisher algorithm (option 2)