Identify outliers for annotation in text data
The idea is to find the documents which are not well represented in the current labeled data. The first point is indeed a bit vague and can probably be interpreted in different ways. My interpretation would be something like this:

For every document $d_u$ in the unlabeled data, count the number of words in common with every document $d_l$ in the labeled data. This value is the &quot;match score&quot; between $d_u$ and $d_l$.

Note: I think that this value should be normalized, for example using the overlap coefficient. Note that other similarity measures could be used as well, for instance cosine-TFIDF.


As output from the above step, for a single document $d_u$ one obtains a &quot;match score&quot; for every labeled document. The average across the labeled documents gives the &quot;average match&quot; for $d_u$.