{'fraction_non_alphanumeric': 0.013333333333333334, 'fraction_numerical': 0.0, 'mean_word_length': 5.352112676056338, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Conservative stemming for search and indexing

In this paper, we describe a stemmer which is designed to stem conservatively to orthographically correct word forms and recognizing words which do not need to be stemmed, such as proper nouns. We compare the performance of our stemmer with several other stemmers and propose further work to make this stemmer more effective for information retrieval, topic detection, and other linguistic applications.