A Gap-Based Framework for Chinese Word Segmentation via Very Deep Convolutional Networks
Zhiqing Sun Gehui Shen Zhihong Deng
cs.CL

Two- and one-dimensional gap solitons in spin-orbit-coupled systems
with Zeeman splitting
    Hidetsugu Sakaguchi^1 and Boris A. Malomed^2,3
    December 30, 2023
=========================================================================================

@topnum0
@botnum0

	Most previous approaches to Chinese word segmentation can be roughly classified into character-based and word-based methods. The former regards this task as a sequence-labeling problem, while the latter directly segments character sequence into words. However, if we consider segmenting a given sentence, the most intuitive idea is to predict whether to segment for each gap between two consecutive characters, which in comparison makes previous approaches seem too complex. Therefore, in this paper, we propose a gap-based framework to implement this intuitive idea. Moreover, very deep convolutional neural networks, namely, ResNets and DenseNets, are exploited in our experiments. Results show that our approach outperforms the best character-based and word-based methods on 5 benchmarks, without any further post-processing module (e.g. Conditional Random Fields) nor beam search.






§ INTRODUCTION


	Unlike English, Chinese are written without explicit word delimiters, which makes word segmentation a fundamental and preliminary task in Chinese natural language processing. Recently, neural approaches for Chinese Word Segmentation (CWS) are attracting huge interest and a great deal of neural models have given competitive results to the best statistical models.

	Previous neural approaches to CWS can be roughly classified into character-based and word-based. The former regards this task as a sequence-labeling problem, while the latter directly segments character sequence into words.

	Since <cit.>, most character-based methods use {B, I, E, S} labels to denote the Beginning, Internal, End of a word and a Single-character word, respectively. To get the label scores for each character, tensor neural network <cit.>,  recursive neural network <cit.>, long-short-term-memory (RNN-LSTM) <cit.> and convolutional neural network (CNN) <cit.> have been proposed. A transition score [A]_i, j for jumping from i to j labels in successive characters is then introduced to handle the label-label transition and give a structured output. With the help of transition scores, a |s| × 4 label score sequence can be decoded into a |s| × 1 label sequence for inference, where |s| is the number of the characters in a sentence. A post-processing module such as Conditional Random Fields (CRF) <cit.> or maximum margin criterion <cit.> can be used to enforce structure consistency and provide an objective function.

	In word-based framework, <cit.> proposed a transition-based model which decodes a sentence from left-to-right incrementally.  <cit.> and <cit.> proposed to score candidate segmented outputs directly. <cit.> introduced partial-word into word-based models. For these word-based models, LSTMs <cit.> or their variants are used for feature extraction, maximum margin criterion is used for training and beam search is used for inference. 

	Despite of the great success these methods achieved, there are still problems in both character-based and word-based frameworks. The main problem of the character-based framework is the use of post-processing modules, e.g., CRF or maximum margin criterion. In computer vision literature, current state-of-the-art models tend to be in an end-to-end scheme and directly get output from the neural networks <cit.>, which in comparison make these post-processing modules seem overdesigned.  Moreover, the use of {B, I, E, S} labels may produce redundant information. For example, “B” may be more similar to “S” or “I” than “E”. These redundancies are not considered in all previous character-based models. An explanation we offer for the extensive using of these post-processing modules in the current state-of-the-art character-based models is that they are not good at capturing character combination features. Besides, the word-based models suffer from the problem of non-parallel, and they can only use the word segmentation information from the previous time steps.

	In this paper, we propose a concise and efficient approach that overcomes the problems of character-based and word-based models: To improve the feature combination, we introduce deep convolutional neural networks to extract features for segmentation. Moreover, we directly predict segmentation for each gap between two consecutive characters. Thus, we need not structure our scores and avoid inference decoding. Because technically speaking, our framework are segmenting based on the gaps, we refer to our approach as Gap-Based Convolutional Networks (Gap-Based ConvNets). Figure <ref> illustrates how our model works.

	We evaluate Gap-Based ConvNets on 5 different benchmark datasets, namely CTB6, PKU, MSR, AS and CityU. As a pure supervised model, Our approach outperforms the current state-of-the-art pure supervised results on all of these benchmarks by a large margin, while are also competitive with the best semi-supervised results. We hope that our simple framework will open up a new way and serve as a solid baseline for CWS research. We also hope that this paper can help ease future research in other sequence-labeling tasks. 
	
	The contributions of this paper could be summarized as follows.

  
  * We propose an end-to-end framework for CWS that directly classify the gaps between two consecutive characters and our results outperform the state-of-the-art character-based and word-based methods.
  
  * Very deep neural networks are first introduced for CWS, in which we propose residual blocks and dense blocks to integrate multiple level character features. We also show that deeper neural networks can achieve better performance in CWS.




§ GAP-BASED FRAMEWORK

	Our gap-based framework is described  in detail in this section. 
	First of all, if we consider segmenting a given sentence s (character sequence) into chunks (words), the most intuitive idea is to predict whether to segment for each gap between two consecutive characters. That is to say, |s|-1 predictions can determine the segmentation of the sentence s.



 §.§ Gap Feature Representation

	
	We follow previous works and consider both uni-character embedding and bi-character embedding when transforming the one-hot sparse discrete sequence s into real-valued representation.
	
	Two separated look-up tables are used to generate a 2-dimension representation u of size |s| × e_u for uni-character embedding and a 2-dimension representation  b of size ( |s| - 1 ) × e_b for bi-character embedding, respectively, where e_u is the dimension of uni-character embedding and e_b is the dimension of bi-character embedding.
	
	The representation of a gap is then a concatenation of the uni-character embedding of its two consecutive characters and their bi-character embedding. Temporal convolutional layers (Conv) with kernel size 1 is applied to the concatenation, in order to integrate the uni-character and bi-character information. The output of the convolutional layer is regarded as the input to the feature extraction blocks.
	
	We follow <cit.> and activate the convolutions with gated linear units (GLU) <cit.>, which is a non-linearity operation that implement a gating mechanism over the convolution Y = [A B]:
	
	
    v([A B]) = A ⊗σ(B)

	
Moreover, Batch normalization (BN) <cit.> follows each convolutional layer A before it is scaled by the gate σ(B). Biases are not used in convolutions.

	
	 In Figure <ref>, u_i and b_j denote the uni-character representation and the bi-character representation, respectively, and “Concatenate” denotes a concatenating operation. “Conv-BN-GLU, k, d” denotes a temporal convolutional operation of kernel size k and kernel number d, which is then batch normalized and activated by GLU.
	 


 §.§ Feature Extraction


Most of the previous applications of neural network to CWS use an architecture which is rather shallow (up to 5 layers). <cit.> proposed a 5-layer convolutional neural network. <cit.> compared their LSTM models in different layers and found their 1-layer LSTM model works best. <cit.>, <cit.>, <cit.> and  <cit.> also use 1-layer LSTM or bi-LSTM to extract features. These architectures are rather shallow in comparison to the deep convolutional networks which have pushed the state-of-the-art in computer vision. Besides, the use of 5-character context window <cit.> in these models shows that their models are not good at capturing character combination features.

In this section, we propose deep feature extraction blocks, namely, residual blocks and dense blocks, to capture character combination features. To the best of our knowledge, this is the first time that very deep convolutional networks have been applied to sequence-labeling.



  
Residual Block



	Traditional convolutional feed-forward networks connect the output of the  ℓ-th layer as input to the (ℓ + 1)-th layer. We can represent this scheme as a layer transition: x_ℓ = H_ℓ(x_ℓ-1). Observing that deep feed-forward networks are hard to train, ResNets <cit.> add a skip-connection that bypasses the non-linear transformations with an identity function:
	
	
    x_ℓ = H_ℓ(x_ℓ-1) + x_ℓ-1

	
	Our L-layer wide residual block design follows <cit.>. We define H_ℓ(·) as a composition of two gated linear unit (GLU), where each of them split the kernel-size-3 convolutions (Conv) into two parts, namely, A and B, and control the convolutional layer A with the gate σ(B). Each A is batch normalized (BN) before the gate.
	
	The Gap-Based ConvNets with a residual block is referred as Gap-Based ResNets in the rest of the paper. Figure <ref> illustrates the layout of an example residual block schematically. 
	


  
Dense Block


	
	The densely connected architecture <cit.> is an extension of the residual architecture. Consider the gap representation x_0 are going to pass through the dense block and our dense block has L layers. Then each layer implements a non-linear transformation H_ℓ(·) to the concatenation of all preceding layers' feature maps:
	
    x_ℓ = H_ℓ([x_0, x_1, …, x_ℓ-1])

where ℓ indexes the layer and [x_0, x_1, …, x_ℓ-1] refers to the concatenation of the feature-maps produced in layers 0, …, ℓ-1.

	Similar to the residual blocks, we define H_ℓ(·) as a gated linear unit (GLU) with a temporal convolutional layer (Conv) with kernel size 3 and Batch normalization (BN). 
	
	The number of the units in the temporal convolutional layer is referred as the growth rate k, as the concatenated layer depth grows by adding more layers. The Gap-Based ConvNets with such a dense block is referred as Gap-Based DenseNets.
	
	A temporal convolutional layer with kernel size 1 is introduced as the bottleneck layer before each convolutional layer with kernel size 3 to reduce the number of input feature maps. We set the number of kernel units in the bottleneck layers to k as the same as the growth rate. Thus, our H_ℓ will be Conv1-BN-GLU-Conv3-BN-GLU.
	
	Figure <ref> illustrates the layout of an example non-bottleneck dense block schematically. It can be noticed that a dense block can explicitly concatenate multi-level character combination features.
	


 §.§ Segmentation Prediction


	At the end of the deep feature extraction blocks, a temporal convolutional layer with kernel size 1 and unit number 2 is performed. Then, a softmax layer is attached to make the output a probability distribution. Finally we get a (|s| -1) × 2 matrix for our prediction, and the values on the second dimension represent the prediction scores of “segmentation” and “no segmentation”, respectively.
	
	We ensemble several separately trained models by taking the means of their prediction scores.
	



§ TRAINING


	Because our model directly classifies the gaps into “segmentation” and “no segmentation”, we can simply use the cross entropy (CE) as our loss function:
	
    L = - ∑_i=1^|s|-1∑_x ∈{0, 1 } p_t^(i) (x) log p_m^(i) (x)

where p_t^(i) is the true probability distribution for the i-th gap, which may be (0, 1) for “segmentation” or (1, 0) for “no segmentation”, and p_m is the probability distribution predicted by our model for the i-th gap, namely, the output of the softmax layer.

	Considering that there are plenty of annotation inconsistencies in the current datasets <cit.>, we use a label smoothing <cit.> with factor β = 0.1 to prevent overfit and boost the model robustness.

	We use Adam <cit.> with a mini-batch size of n to optimize model parameters, with an initial learning rate α_1 = 0.002 in the first 8000 steps and α_2 = 0.0002 in the rest steps. Model parameters are initialized by normal distributions as <cit.> suggested. A dropout for the gap representation with dropout rate p_1 is used to reduce overfitting. We set pretrained bi-character embedding fixed and only fine-tune the pretrained uni-character embedding.


§ EXPERIMENTS






 §.§ Experimental Settings




  
Data
We use Chinese Treebank 6.0 (CTB6) (LDC2007T36) <cit.> as our main dataset. We follow the official document and split the dataset into training, development and test data. In order to verify the robustness of our model, we additionally evaluate our models on SIGHAN 2005 bakeoff <cit.> datasets, where we randomly split 10% data from the training data as development data.

We replace all the punctuation with “<PUNC>”, English characters with “<ENG>” and Arabic numbers with “<NUM>” for all text. We also add “</s>” symbol to the beginning and the end of a sentence.

We apply word2vec <cit.> on Chinese Gigaword corpus (LDC2011T13) to get pretrained embedding of uni-characters and bi-characters. We choose 50 for both uni-character embedding size e_u and bi-character embedding size e_b. Notice that as we do not need word embedding, we do not have to automatically segment the corpus by other segmentors. The use of word embedding in the gap-based framework is left for future works.



  
Evaluation 
The standard word precision, recall and F1 measure <cit.> are used to evaluate segmentation performances. The prediction scores of 4 separately trained models with same settings are ensembled for the evaluation of different model architectures and hyper-parameters.



  
Fine-tune 
We fine-tune the hyper-parameters on the development data. We almost keep all the hyper-parameters to be the same when we evaluate our models on different datasets, except that the batch size is set to 256 for AS dataset while to 64 for other datasets, and  dropout rate p_1 is set to 0.3 for CTB6 and PKU datasets while to 0.2 for other datasets.



 §.§ Model Analysis

We perform development experiments on CTB6 dataset to verify the usefulness of various configurations and different loss objectives, respectively.



  §.§.§ Model Selection


We evaluate our Gap-Based Convolutional Networks with residual blocks or dense blocks, with different number of layers L. The main results on CTB6 are shown in Table <ref>, where we mark our best results in boldface. The dimension of the first convolutional layer (gap representation) are set to 512.

We find that our 12-layer and 28-layer Gap-Based DenseNets both achieve the best performance, with growth rate 128. Therefore, to get both the performance and the speed, we only use a 12-layer Gap-Based DenseNet with growth rate 128 in the following experiments.



  §.§.§ Gap Representation




We compare different gap representations. The results are shown in Table <ref>, where “both” represents the original model, “uni-character' ' and “bi-character' ' represent the gap representation only with uni-character embedding and bi-character embedding, respectively. And “combined” represents the combined results of a pure uni-character model and a pure bi-character model.

As can be seen from the table, by removing uni-character and bi-character embedding, the
F-score decreases to 94.8 and 95.8, respectively. We can find that the uni-character embedding are more robust and useful than bi-character embedding, which is an opposite conclusion to <cit.>. We believe this is due to the sparsity of the bi-character embedding.



In addition, we conduct a simple experiment on the combination of pure uni-character and pure bi-character models.

As the pure uni-character models and the pure bi-character models have different representations for the gaps, their predictions for the segmentation are also independent. They produce different distribution of segmentation errors, which provide the opportunity for them to learn from each other, as shown in Table <ref>.

Therefore, we provide an oracle-combined model to combine the results from these two models. For the combined results, we accept the segmentation results that both pure uni-character model and pure bi-character model accept and use what we call “oracle” to determine whose results to accept when in divergence of views. The “oracle” represents that we can always make the right choices. The combined results are also listed in Table <ref>. 

We find that while performs better than pure uni-character and pure bi-character models, our oracle-combined results also outperform the original model by a large margin, which suggests that uni-character information and bi-character information are quite complementary, and maybe the combination of uni-character information and bi-character information before the feature extraction blocks makes the original model not able to sufficiently exploit the feature combination ability of deep neural networks.

However, “oracle” means that we need to know the answer beforehand, which is impossible in practice. Therefore, a practical way to combine the results of pure uni-character and pure bi-character models is left for future investigation.




  §.§.§ Comparison with Character-Based Framework


Both our gap-based model and character-based models regard CWS task as a sequence-labeling task. However, a big difference between them is that while character-based models have to give 4 scores for each character and use a post-processing module, the gap-based models only need to give a binary classification for each gap. 

In order to see the efficiency of the gap-based schemes, we compare the character-based scheme and the gap-based scheme in the same neural network architecture, namely, Gap-Based DenseNets. To combine the character-based scheme with the original architecture, we revise the architecture by

  
  * We represent the character information by the character's uni-character embedding and the bi-character embedding of its consecutive characters.
  
  * We use a convolutional layer and softmax layer with 4 units instead of 2, which represent the scores for {B, I, E, S} in the character-based sequence-labeling scheme. 










To fully investigate the performance of character-based scheme, following <cit.>, we train the network in two approaches. The first approach use a transition matrix to model the tag dependency and CRF for structured inference, and the second use a greedy loss that directly classifies the characters into {B, I, E, S}. The results are shown in Table <ref>, where “character-based CRF” represents the first approach and “character-based greedy” represents the second.

We can see that our gap-based scheme have competitive results to the character-based scheme, while we do not have a post-processing module. Moreover, our simple classification scheme makes us more convenient to use a great deal of well-developed tricks in the large supervised learning literature, e.g., label smoothing and confidence penalty.

Table <ref> gives an example segmentation result that our gap-based framework gives the correct segmentation, while the answer from the character-based framework is wrong. It shows that our gap-based segmentation framework is able to fix some mistakes that character-based framework will make.



 §.§ Final Results


In addition to CTB6 dataset, which has been the most commonly adopted by recent segmentation research, we additionally evaluate our models on the SIGHAN 2005 bakeoff datasets, to examine cross domain robustness. Among these datasets, PKU and MSR datasets are in simplified Chinese, while AS and CityU datasets are in traditional Chinese and we have to map them into simplified Chinese before segmentation.

Our final results are shown in Table <ref> and Table <ref>, which list the results of several current state-of-the-art methods. As can be seen from Table <ref>, our proposed method gives the best pure supervised performance among both statistical and neural segments in all datasets by a large margin.

Moreover, our method are also competitive to the best semi-supervised methods, including those using rich pretraining information like mutual information<cit.>, punctuation<cit.>, automatically segmented text<cit.>, POS data<cit.> or word-context embedding<cit.>. As can be seen in Table <ref>, we outperform the best semi-supervised models on CTB6, AS and CityU datasets.

In summary, while competitive to the best semi-supervised methods, our gap-based approach gives the best pure supervised performance on all corpora. To our knowledge, we are the first to report state-of-the-art results on both CTB6 and all SIGHAN 2005 bakeoff benchmarks. It verifies that while simple, the gap-based framework is very efficient for CWS.



§ RELATED WORK


<cit.> was the first to propose to regard CWS task as character-tagging, using a maximum entropy model to give each character a label from  {B, I, E, S}. <cit.> followed this character-tagging scheme and proposed a conditional random field (CRF) to further improve the performance. Since then, this sequence-labeling CRF scheme was followed by most subsequent approaches in the literature.

<cit.> was the first to propose a neural network model and introduced character embedding for CWS, using a character window of size 5. This kind of 5-character window design still appears in recent state-of-the-art models. <cit.> used a tensor neural network to further exploit the character combination feature and introduced bi-character embedding into neural CWS models. <cit.> proposed a Gated Recursive Neural Network to combine character features. <cit.> proposed a LSTM model in order to get rid of this 5-character window design. <cit.> proposed a Dependency-based Gated Recursive Neural Network to efficiently combine local and global features. <cit.> proposed a convolutional neural network to extract features. These character-based models exploited the recent springing up neural network architectures.

<cit.> introduced the log conditional odds that a given token sequence constitutes a chunk according to a generative model. <cit.> was the first to proposed a word-based approach to CWS, which provides a direct solution to the problem. <cit.> proposed a beam-search model. <cit.> proposed a neural transition-based model with beam search that explicitly produce chunks in order. <cit.> and <cit.> proposed a model that score the candidate segmented outputs directly. These word-based models can fully exploit the word features such as word embedding.

Observing the similarity of sequence segmentation in natural language processing (NLP) literature and and semantic segmentation in computer vision literature, we may find that “simplest things are the best” Occam's razor is not applied to NLP sequence segmentation tasks. <cit.> proposed to use Fully Convolutional DenseNets to do semantic segmentation by directly do  classification on each pixel and beat other complex framework such as fully connected CRFs <cit.>. This paper provides us the idea to directly classify the gaps.

The model architecture of <cit.> is similar to ours, as both of us use convolutional neural networks to extract character combination features. However, their models are rather shallow (up to 5 layers) and only use feed forward connections, while we introduce deep feature extraction blocks that contain residual connections or dense connections, inspired by <cit.> and <cit.>. Moreover, their models are still in character-based framework, while we show that our gap-based framework can further exploit the representation power of deep neural networks.


Our approach remains much potential that can be further investigated and improved in the future. For example, our models may furthermore benefit from recently popular semi-supervised learning methods, such as word-context character embedding<cit.>, rich pretraining<cit.> and pretrained word embedding <cit.>. They can all get more information from auto-segmented text. The use of LSTM-RNN and its variants in the gap-based framework is also interesting to be investigated.




§ CONCLUSION

In this paper, we propose a novel gap-based framework for Chinese word segmentation that directly predict whether to segment for each gap between two consecutive characters. Moreover, we introduce very deep convolutional networks (residual blocks and dense blocks) for feature extraction.

Experiments show that our proposed Gap-Based ConvNets are effective to solve Chinese word segmentation task. We outperform the previous best character-based and word-based methods by a large margin. To our knowledge, we are the first to report state-of-the-art results on both CTB6 and all SIGHAN 2005 bakeoff benchmarks.


acl_natbib

