PKUSEG: A Toolkit for Multi-Domain Chinese Word Segmentation
Ruixuan Luo Jingjing Xu Yi Zhang Zhiyuan Zhang Xuancheng Ren Xu Sun


Enhancing Python Compiler Error Messages
via Stack Overflow
    Emillie Thiselton and Christoph Treude
School of Computer Science
University of Adelaide

emillie.thiselton@gmail.com, christoph.treude@adelaide.edu.au

    
============================================================================================================================================================




    Chinese word segmentation (CWS) is a fundamental step of Chinese natural language processing. In this paper, we build a new toolkit, named PKUSEG, for multi-domain word segmentation. Unlike existing single-model toolkits, PKUSEG targets multi-domain word segmentation and provides separate models for different domains, such as web, medicine, and tourism. Besides, due to the lack of labeled data in many domains, we propose a domain adaptation paradigm to introduce cross-domain semantic knowledge via a translation system. Through this method, we generate synthetic data using a large amount of unlabeled data in the target domain and then obtain a word segmentation model for the target domain. We also further refine the performance of the default model with the help of synthetic data. Experiments show that PKUSEG achieves high performance on multiple domains. The new toolkit also supports POS tagging and model training to adapt to various application scenarios. The toolkit is now freely and publicly available for the usage of research and industry.[<https://github.com/lancopku/pkuseg-python/>]





§ INTRODUCTION

Chinese word segmentation is a fundamental task of Chinese processing.  Since words define the basic semantic unit of Chinese, the quality of segmentation directly influences the performance of downstream tasks. 

In recent years, Chinese word segmentation has undergone great development. The best-performing systems are mostly based on conditional random fields (CRF) <cit.>. However, despite the promising results, these approaches heavily rely on feature engineering. To tackle this problem, many researchers <cit.> explore neural networks to automatically learn better representations. 





Recently, there arise several public segmentation toolkits, such as jieba, HanLP, and so on. For efficiency, they are built on traditional segmentation models, like perceptron <cit.> or CRF, rather than time-consuming neural networks.  These toolkits only provide a single coarse-grained segmentation model, mostly trained on news domain data. In real-world applications, the domain of text varies and the text from different domains has different domain-specific segmentation rules. This increases the difficulty of segmentation and drops the performance of existing toolkits on text from various domains.








To address this challenge, we propose a multi-domain segmentation toolkit, PKUSEG, based on the work of <cit.>: We adopt a fast and high-precision model CRF as implementation. PKUSEG includes multiple pre-trained domain-specific segmentation models. Since some domains may be of low resources, we use a pre-training technique to improve the quality of segmentation. We first pre-train a coarse-grained model on a mixed corpus, including millions of data from news and web domains. Then, we fine-tune the coarse-grained model on specific domain data to get fine-grained models. 

To train a model for some domains that do not have any manually annotated data, but have a lot of unlabeled data, we further propose an unsupervised domain adaptation approach named masked Attention Augmentation. Due to the lack of labeled data in the target domain, these domains lack the semantic knowledge of the target domain, so it is difficult to correctly segment the proper noun or syntactic structure of the target domain. To address these issues, we propose to learn word-level semantic knowledge of the target domain by introducing translation. We first obtain the English translation of the data through a translation model. Then, we train a large Transformer model for segmentation and masked language model tasks. The model takes the translation pairs as input and tries to obtain the semantic knowledge of the target domain by learning the word-level alignment between the two languages. The Transformer model is then used to generate synthetic data for the target domain. We use synthetic data to further train the target domain model or improve the general model performance.

In addition to provided segmentation models, PKUSEG also allows users to train a new model on their domain data.  Furthermore, POS tagging is supported in PKUSEG to adapt to various scenarios. Experimental results show that PKUSEG achieves high performance on multi-domain datasets. 

In summary, PKUSEG has the following characteristics:

    
  * Good out-of-the-box performance. The default word segmentation model provided by PKUSEG is trained on a large-scale, curated, multi-domain dataset, which shows stable and high performance across various domains.
    
  * Domain-specific pre-trained models. PKUSEG also comes with multiple pre-trained models that are fine-tuned on texts of different domains, which further elevates domain-specific performance, suitable for analyzing in-domain texts. 
    
  * Easy transfer learning. For advanced users, PKUSEG supports transfer learning based on the default multi-domain model. Users can fine-tune the model on their custom segmented texts.
    
  * Unsupervised domain adaptation. For the domains which do not have any manually annotated data, but have a lot of unlabeled data, we propose an unsupervised domain adaptation paradigm. And our method successfully trains the domain-specific model without any target domain annotated data.
    
    
    
    
    
    
  * POS tagging. PKUSEG also provides users with POS tagging interfaces for further lexical analysis.






§  IMPLEMENTATION

This section gives a detailed description of toolkit implementation.




 §.§ Conditional Random Field





Despite the better performance, we do not use neural networks as an implementation due to the time-consuming training process. Instead, we use a well-performing and fast-training model, CRF, as implementation, considering the trade-off between time cost and high accuracy.  We optimize the weights of CRF by maximizing the log-likelihood of the tags of the reference sequence. When calculating the log-likelihood, the log-likelihood function can be calculated by the recursive algorithm in linear time. During inference, the Viterbi <cit.> algorithm is adopted. The goal is to find the sequence of tags by dynamic programming.


 
 




 §.§ ADF Algorithm


For CRF with many high-dimensional features, the amount of parameters is very large, leading to expensive training costs. To address this problem, we use adaptive online gradient descent based on feature frequency information (ADF) <cit.> for training.
The ADF algorithm does not use a single learning rate for all parameters like stochastic gradient descent (SGD), instead, it turns the learning rate into a vector with the same dimension as the parameters. The learning rate of each parameter is automatically adjusted according to the frequency of the parameter. The idea is that the feature with higher frequency will be more adequate. 







 §.§ Pre-training

To handle the problem of low-resource for some domains, we adopt a pre-training technique in PKUSEG following the work of <cit.>. We mix news and web data together as pre-training data. News data comes from dataset PKU provided by the Second International Chinese Word Segmentation Bakeoff. Web data comes from the Weibo dataset provided by NLPCC-ICCPOL 2016 Shared Task <cit.>. A hybrid dataset CTB  is also involved in pre-training. In the process of fine-tuning,  models are initialized with the pre-trained model and trained on domain-specific data.  So far PKUSEG supports four fine-grained domains, including news, medicine, tourism, and web.  Considering the covered domains are limited, we also provide a pre-trained model for generalization. 




 §.§ A Large-Scale Vocabulary

One major difficulty of multi-domain segmentation is spare domain-specific words. It is hard to cover all of these words on the training set. Therefore,  to increase the coverage rate of PKUSEG, we automatically build a large-scale domain vocabulary. The word resource is crawled from Sogou website[<https://pinyin.sogou.com/dict/>] and extracted from the training data of PKU, MSRA, Weibo, and CTB.  In total, we extract almost 850K words.  The distribution of words is shown in Table <ref>. 







 §.§ Derive Synthetic Data with masked Attention Augmentation


In the process of practical application, because the cost of data annotation is very expensive, the domain where the model is deployed often faces a situation where unlabeled samples are abundant but labeled samples are seriously insufficient. In order to make full use of the existing labeled data and unlabeled data of the target domain to train a domain-specific model, we propose a masked Attention Augmentation (m𝒜^2) approach <cit.> to derive synthetic data. Our method is able to train a segmentation model for the target domain without any target domain labeled data. 

An overview of m𝒜^2 is illustrated in Figure <ref>. Specifically, we first use Google Translate to obtain an English translation of the source domain and target domain data. Then we train a Transformer model of the target domain. Given a translation pair, we randomly mask some tokens and encourage the model to reconstruct them. Since the tokens of both languages can be masked, this masked language model task encourages the model to recover the tokens based on the prompts in another language, and further enhance the alignment between the two languages. Meanwhile, we train the word segmentation task on the source domain. The final objective is the sum of the masked language model task and the Chinese word segmentation task. The m𝒜^2 approach prompts the model to learn about the crucial word-level semantic knowledge of the target domain. And then, the well-trained Transformer model is used to derive synthetic data from the target domain. The synthetic data is finally used for training domain-specific models or improving the generalization performance of the default model. So far, we obtain unlabeled data of three domains from Wikipedia, including science, art, and entertainment. And we use the news domain as our source domain.


This figure is from <cit.>.
 


§ USAGE

PKUSEG has high precision performance along with user-friendly interfaces. It is developed based on standard python3  libraries. PKUSEG supports common running platforms,  such as Windows, Linux, and MacOS. 





 §.§ Installation


 PKUSEG offers two user-friendly installation methods. Users can easily install it with PyPI and the corresponding models will be downloaded at the same time. A typical command is:



pip3 install pkuseg



Users also can install PKUSEG from GitHub. After downloading the project code from GitHub, users can run the following command to install PKUSEG:



python setup.py build_ext -i



Noting that the downloaded project from GitHub does not include pre-trained models, users need to additionally download them from GitHub or train a new model.



 §.§ Segmentation

The followings are a detailed introduction to segmentation interfaces.




  
Domain-specific Segmentation.
 If a  user is aware of the domain of the text to be segmented, then he/she can use the domain-specific model. An example code for specifying a model is shown in Figure <ref>. If a model is toolkit-provided,  users can directly use the domain name to call it, e.g, “medicine”, “tourism”, “web”, and “news”. The model is automatically loaded based on the parameter “model_name”. If the model is user-trained, “model_name” refers to the model path.
 



  
Coarse-grained Segmentation.
Although PKUSEG is designed to satisfy the situation where users know the domain of the text to be segmented, we also provide a coarse-grained model in case that the user can not distinguish the target domain. The coarse-grained model works under the default mode.  Figure <ref> shows an example code using the default mode. 






  
User-defined Dictionary.
To better recognize new words, users can add a dictionary to cover the words that do not occur in the dictionary of PKUSEG. The provided dictionary file should follow the following format. Each row has a single word and the dictionary file is encoded in the UTF-8 format. Figure <ref> shows the usage of a user-defined dictionary.






  
Model Training.

PKUSEG also allows users to train a new model from scratch with their training data. Figure <ref> is an example code for showing how to train a new model. 




  
Segmentation with POS Tagging. 
In addition to segmentation, PKUSEG  can also label POS tags for words in a sentence. The usage of POS tagging interfaces is shown in Figure <ref>.

 

  








§ EXPERIMENT

This section evaluates the performance of PKUSEG.


 §.§ Settings

We compare PKUSEG with the well-known CWS toolkits, THULAC, and jieba. It should be noted that they are trained on a heterogeneous mass of text to satisfy the requirement of generic segmentation. While PKUSEG is designed for multi-domain segmentation and the training of PKUSEG models is more targeting. The different application scenes make they can not be directly compared. Only for providing a reference for users, we use the training API provided by THULAC and JIEBA and train new models with the same training set of PKUSEG to compare the performance. We evaluate the PKUSEG on the following CWS benchmarks which stand for different domains: MSRA (news), WEIBO (web text), and CTB8 (hybrid text).







 §.§ Dataset



  
MSRA & PKU. MSRA and PKU are from the news domain and provided by the Second International Chinese Word Segmentation Bakeoff. 



  
CTB8. Chinese Tree Bank is a hybrid domain dataset.[<https://catalog.ldc.upenn.edu/LDC2013T21>] It consists of approximately 1.5 million words from Chinese newswire, government documents, magazine articles, various broadcast news, and broadcast conversation programs, web newsgroups, and weblogs.



  
Weibo. This dataset comes from the NLPCC-ICCPOL 2016 Shared Task. Different from the popularly used newswire datasets, this dataset consists of many informal micro-texts.



  
Medicine & News & Tourism. The corpus
is originally constructed in <cit.> by
annotating multi-domain texts. 



  
Science & Art & Entertainment. To evaluate the performance of domain-specific models constructed by our unsupervised method, we manually annotated about 100 sentences for each of the three domains on Wikipedia.







          
















 §.§ Out-of-domain Results

To show the effect of domain knowledge on segmentation performance, we train a model on the CTB8 dataset and report its performance on different datasets. Here we choose CTB8 as an example because CTB8 is a hybrid dataset. The results are shown in Table <ref>. 



Here are the results without using vocabulary. We can see that the performance drops obviously on out-of-domain datasets. Since different domains have their unique segmentation standard, it is not suitable to provide one single model for various domain data. The result demonstrates the necessity of fine-grained segmentation toolkits. 




 §.§ Pre-training Results

We combine existing large-scale datasets together, including  PKU (news), Weibo (web), and CTB8 (hybrid), and use them as pre-training data to obtain a coarse-grained model. Then the coarse-grained model is used to fine-tune domain-specific models. 





Table <ref> shows the effect of pre-training. The pre-training models perform much better in terms of average score, especially on datasets with lower resource (e.g., tourism). 























         








 §.§ Default Performance

Considering the fact that many users tend to use the default mode to test performance,  with the default model and vocabulary of PKUSEG. We also report experimental results on the default mode. The results are shown in Table <ref>.



As we can see, the performance of the default model performs worse than that of domain-specific models. Therefore, we recommend users use domain-specific models, rather than the default model if the user can distinguish the domain of the text. 

To learn more about the practical application of PKUSEG, we also show some segmentation examples that randomly crawled from articles that cover the domains of medicine, travel, web text, and news. The segmentation results are shown in Table <ref>. PKUSEG has high accuracy when dealing with words that need professional domain knowledge.

UTF8gbsn


































    


    















    






















 §.§ Performance in Default Setting

Considering the fact that many users tend to use the default model to test performance, we compare the performance of different toolkits with their default settings. The comparison result is shown in Table <ref>. As shown, PKUSEG receives higher F1 scores on all datasets that stand for different domains compared with jieba and THULAC. Consequently, it has the best average performance.





 §.§ Unsupervised Domain-specific Performance



To verify the practicability of domain-specific models constructed by our unsupervised approach, we conduct experiments with data from three domains of science, art, and entertainment, and compare the performance of our domain-specific models with the default PKUSEG model. The results are shown in Table <ref>. 




The domain-specific models have an average +0.92 improvement in the three domains.





 §.§ Refined Default Performance


To improve the generalization performance of the default model, we combine all the synthetic data and annotated data together, and then train the refined PKUSEG model with this large mixed dataset. We compare this refined model with the original default PKUSEG model on both in-domain datasets (the training sets of the original default model) and out-of-domain datasets (the unannotated domains from Wikipedia). The experimental results are shown in Table <ref> and Table  <ref>.

The refined model has no performance drop in the in-domain datasets while having +0.67 F1 improvement on the out-of-domain datasets. The results show that the refined model achieves better generalization performance.





§ CONCLUSION AND FUTURE WORK

In this paper, we propose a new toolkit PKUSEG for multi-domain Chinese word segmentation. PKUSEG provides simple and user-friendly interfaces for users. Experiments on widely-used datasets demonstrate that PKUSEG performs well with high accuracy.  So far PKUSEG supports domains like medicine, tourism, web, and news.  In the future, we plan to release more domain-specific models and improve the efficiency of PKUSEG further.





acl_natbib

