{'fraction_non_alphanumeric': 0.012, 'fraction_numerical': 0.0, 'mean_word_length': 4.994011976047904, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Procedures of extending the alphabet in combined coding for prediction by partial string matching in text compression

The present work describes the PPM — Prediction by Partial (string) Matching algorithm for lossless compression of text. It studies the procedure of extending the source alphabet for the PPM encoding in order to allow using of symbols not yet present in the source alphabet at the very beginning of the process of text encoding. The work describes the procedure of extending the alphabet and presents the changes needed to be performed in the PPM algorithm so that it can use the extended generated alphabet. There are three ways proposed in extending the source alphabet: manual, running over the text, and adaptive. The arithmetical compression algorithm is used in order to encode the text words with the help of the statistic model derived from the PPM. Experimental results on different formats of text files and useful conclusions for text lossless compression are presented.