{'fraction_non_alphanumeric': 0.017595307917888565, 'fraction_numerical': 0.0, 'mean_word_length': 5.762376237623762, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Phrase Segmentation Model using Collocation and Translational Entropy

In this paper, we propose a phrase segmentation model for the phrase-based statistical machine translation. We observed that good translation candidates generated by a conventional phrase-based SMT decoder have lexical cohesion and show more uniform translation for each phrase segment. Based on the observation, we propose a novel phrase segmentation model using collocation between two adjacent words and translation entropy of phrase segments. Experimental results show that the proposed model significantly improves the translation quality in both English-to-Korean and English-to-Chinese translation tasks.