While mulling this over in the back of my mind while working on other stuff for several weeks, it has morphed into several ways to improve some of the shortcomings of TextCat, as well as ways to report a somewhat meaningful score (which TextCat be default isn't really well-equipped to do).
So, I've converted this to an EPIC and I'll add a number of sub-tasks to it.
I've kept the original description below.
Some potential measures of confidence that could be applied to TextCat are laid out in "TextCat and Confidence"; briefly, we could consider the number of language guesses TextCat gives, compare the score of the second-best guess to the best guess, or compare a score to the theoretical worst possible score for a given string.
It's possible to test any (or all) of those measures against existing query sets and try to find a threshold for better/worse language identification performance.
Trey's language evaluation sets: the corpora are small, but we have a reasonable approximation of "the truth" for each identification.
Mikhail's TextCat A/B test data: the corpora are probably larger (not sure on the breakdown by wiki), and a lot more data can be gotten relatively easily without a lot of manual work to get it; evaluation could be based on clicks, which depend not only on language ID, but also whether there's anything relevant in the interwiki results.
Randomly selected queries: We could identify them by language, and the number of results from a cross-wiki search could be used as a proxy for language identification quality. Data is effectively unlimited and easy to get, though evaluation would be less direct.
The confidence thresholding could happen internally to TextCat, based on a threshold parameter, since it already fails to return a value when too many candidates are likely (by default: >5, within 5%).
T151796: Should usage of textcat be tracked in the search satisfaction schema?
TJones added subscribers: debt, mpopov, EBernhardson.
debt moved this task from needs triage to This Quarter on the Discovery-Search board.
debt raised the priority of this task from Normal to High.
It’s straightforward to run a few tests using a few possible confidence measures. We have the language evaluation sets (small, but manually categorized) and @mpopov's A/B Test data (large but categories are only heuristic: clicked/unclicked).
If a confidence measure gives improved results for either case, it’s worth pursuing.
TJones added a project: Epic.
TJones created subtask T149318: Add support for limiting min input length for TextCat.
TJones created subtask T149322: Bucketing & Bonuses for TextCat.
TJones created subtask T149324: TextCat Improvement Deployment.
debt closed subtask T151230: Consider Additional Unknown n-gram Penalty as Resolved.
Deskana edited projects, added Discovery-Search; removed Discovery-Search (Current work).
Deskana lowered the priority of this task from High to Normal.
This task requires the attention of @TJones, and given the efforts we're devoting to language analyser work which we've deemed more important, I'm moving this task out of the sprint to reflect the reality of the situation that this won't be worked on soon.
mmodell moved this task from needs triage to later on... on the Discovery-Search board.
Closing this because after looking into it a while back I decided that internal confidence isn't really a thing for TextCat to do, and easy things to improve the quality of TextCat results were done.
TJones closed subtask T155670: Investigate Ratio of First to Second Result Scores as a Confidence Measure as Declined.