{'fraction_non_alphanumeric': 0.028193832599118944, 'fraction_numerical': 0.0026431718061674008, 'mean_word_length': 4.308411214953271, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
HMM Tagging on Coherent vs. Nonsense Text

Linguists would very much like to know whether the two are the same or at least whether the latter subsumes the former. A partial answer, no, is given by the fact that HMM-taggers have not yet been found that tag 100%. The discrepancy could be non-existent (but not yet shown) or could be due to inherent limitations in HMM:s (a question which could in principle be resolved by intractable amounts of brute force). But, the discrepancy could also be due to a wrong understanding part-of-speech tagging in language, or the necessity of semantics. We will look at it from this angle. If it can be shown that taggers perform just as well on text when there is no semantic coherence, this is a good argument that the discrepancy is due to semantics. Tagging is an art over three kinds of data: text, lexicon and tag sequence. Normally, a tagger is given a (semantically coherent) text annotated with a tag sequence. The tag lexicon is either given independently or extracted from the annotated text. In our case, we have done tests varying the said parametres away from what is normally the case.