{'fraction_non_alphanumeric': 0.011428571428571429, 'fraction_numerical': 0.0, 'mean_word_length': 5.258928571428571, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Training hidden Markov models on incomplete sequences

This paper deals with the algorithms of training hidden Markov models on sequences with missing observations. The method of imputation using Viterbi algorithm and the method of marginalization of missing observations are studied. These two methods are compared to the standard methods of dealing with missing data: imputation of gaps using the mode (the most frequent value) of nearest observations and gluing of observable parts of the sequence. The studied methods appeared to be more effective than the standard ones. Although the marginalization method proved to be a bit more effective than the method of imputation using Viterbi algorithm.