Multi-Task Learning with Contextualized Word Representations for Extented Named Entity Recognition
Thai-Hoang Pham Khai Mai Nguyen Minh Trung Nguyen Tuan Duc Danushka Bolegala Ryohei Sasano Satoshi Sekine



Primordial power spectrum and cosmology from black-box galaxy surveys

    Alan Heavens
    December 30, 2023
=======================================================================




Fine-Grained Named Entity Recognition (FG-NER) is critical for many NLP applications. While classical named entity recognition (NER) has attracted a substantial amount of research, FG-NER is still an open research domain. The current state-of-the-art (SOTA) model for FG-NER relies heavily on manual efforts for building a dictionary and designing hand-crafted features. The end-to-end framework which achieved the SOTA result for NER did not get the competitive result compared to SOTA model for FG-NER. In this paper, we investigate how effective multi-task learning approaches are in an end-to-end framework for FG-NER at different aspects. Our experiments show that using multi-task learning approaches with contextualized word representations can help an end-to-end neural network model achieve SOTA results without using any additional manual effort for creating data and designing features. 




§ INTRODUCTION




  

Fine-grained named entity recognition (FG-NER) is a special kind of named entity recognition (NER) that focuses on identifying and classifying a large number of entity categories. In traditional NER task, often less than eleven named entity (NE) categories are defined. For example, in two shared
tasks, CoNLL 2002 and CoNLL 2003 <cit.>, there were only four NE types considered: Person, Location, Organization, and Miscellaneous. From these shared tasks, ten NE categories were defined for Twitter texts <cit.>. The FG-NER, on the other hand, handles hundreds NE categories which are the fine-grained classification of coarse-grained categories. In particular, <cit.> proposed the entity hierarchy which contains 200 NE categories designed manually. Meanwhile, <cit.> used unsupervised methods for creating FG-NER category from knowledge bases such as Freebase <cit.> and YAGO <cit.>. Figure <ref> shows an example when identifying and classifying NE by traditional NER and FG-NER systems.





  

While there have been many methods proposed for classical NER <cit.>, FG-NER is still an open research domain. Unlike NER, the current SOTA model for FG-NER <cit.> requires significant manual effort for building a dictionary and designing features. The end-to-end neural network architectures have not achieved competitive results for this task. It is because the data sparseness problem of some NE categories when the size of FG-NER dataset are comparable with NER dataset while the number of NE categories is much larger. Moreover, identifying NE for FG-NER is more difficult compared to NER because these NE types are more complex and longer.


  

Recently, multi-task learning approaches have been proposed for improving the performances of NER systems <cit.>. It can be seen as a form of inductive transfer that introduces an auxiliary task as an inductive bias to help a model prefer some hypotheses over the others. Another way to improve NER systems is using contextualized word representations to learn the dependencies among words in a sentence <cit.>. From these motivations, we investigate the effectiveness of multi-task learning for the end-to-end neural network architecture in both cases uncontextualized and contextualized word representations for FG-NER task.


  

We have novel contributions in two folds. First, to the best of our knowledge, our work is the first study that concentrates on multi-task learning approach for sequence labeling problem in general and for FG-NER task in particular at different aspects including different parameter sharing schemes for multi-task sequence labeling, learning with neural language model, and learning at different word representation settings. We also give empirical analysis to understand the effectiveness of contextualized word representations for FG-NER task. Second, we propose an end-to-end neural network architecture which achieves SOTA result compared to the previous systems that require significant manual effort for building a dictionary and designing features. This neural network system, despite focusing on FG-NER task, can still be applied to any other sequence labeling problems.


  

The remainder of this paper is structured as follows. Section <ref> describes multi-task learning architectures and contextualized word representations used in our system. Section <ref> gives experimental results and discussions. Finally, Section <ref> concludes the paper.


§ APPROACH




 §.§ Single-Task Sequence Labeling Model



  

With a recent resurgence of the deep learning approaches, there have been several neural network models proposed for sequence labeling problem. Most of these models shared the same abstract architecture. In particular, each input sentence is fed to these models as a sequence of words and is transformed into a sequence of distributed representations by the word embedding layer. These distributed representations can be improved by incorporating character-level information from Convolutional Neural Network (CNN) or Long Short-Term Memory (LSTM) layer into word embedding layer.
That distributed representation sequence is then passed to the recurrent neural network layer (LSTM or a Gated Recurrent Unit (GRU)), and then a Conditional Random Field (CRF) layer takes as input the output of the recurrent neural network layer to predict the best output sequence <cit.>. 




  

In our work, we re-implement the neural network architecture in <cit.> which is the combination of CNN, bi-directional LSTM (BLSTM), and CRF models as our base model. 
For training, we minimize the negative log-likelihood function:


    E = -∑_t=1^Tlog(P(y_t|h_t))

where h_t is the output of BLSTM and y_t is the label at time step t. Decoding can be solved effectively by Viterbi algorithm to find the sequence with the highest conditional probability.




 §.§ Multi-Task Learning with Sequence Labeling Model



  

Recently, multi-task sequence learning approach has been used successfully in sequence labeling problem <cit.>. In particular, the main sequence labeling task is learned with auxiliary sequence labeling tasks during training to improve the performance of the main task. These multi-task sequence labeling models are extensions to the base model discussed above with different parameter sharing schemes. 


  

In our work, we investigate two kinds of multi-task sequence labeling models for FG-NER task including same-level-shared model and hierarchical-shared model. To train multi-task sequence labeling models, we minimize both of auxiliary and main objective function. In particular, for an input sequence x = (x_1,x_2,...,x_T), we minimize:

    E_auxiliary=-∑_t=1^Tlog(P(y^auxiliary_t|h^auxiliary_t))

 if x belongs to auxiliary data, or 

    E_main=-∑_t=1^Tlog(P(y^main_t|h^main_t))

if x belongs to main data.
h^auxiliary_t, h^main_t are the outputs of BLSTM and y^auxiliary_t, y^main_t are the labels at time step t.


  
Same-level-Shared Model
For same-level-shared model, both main and auxiliary tasks are trained and predicted at the same-level layer. Specificially, we experiment with two kinds of same-level-shared model including embedding-shared model (Figure <ref>fig:1b) which uses the same embedding layer for both main and auxiliary tasks, and separate LSTM and CRF layers for each task, and RNN-shared model (Figure <ref>fig:1c) which uses the same embedding and LSTM layers for both main and auxiliary task, and separate CRF layers for each task.  In RNN-shared model, h^auxiliary, h^main are the same and are computed from one BLSTM layer:

    h^auxiliary = h^main = BLSTM(x)

while in embedding-shared model, h^auxiliary, h^main are computed from separate BLSTM layers:

    h^auxiliary = BLSTM^auxiliary(x)


    h^main = BLSTM^main(x)




  
Hierarchical-Shared Model
For hierarchical-shared model, we train and predict different supervised tasks at different-level layers. The auxiliary and main tasks are predict by the low-level and high-level layers respectively. To avoid catastrophic interference between main and auxiliary tasks, the word representations are fed into both both low-level and high-level layers. In particular, h^auxiliary, h^main are computed as follows:

    h^auxiliary = BLSTM^auxiliary(x)


    h^main = BLSTM^main([x;h^auxiliary])




 §.§ Multi-Task Learning with Neural Language Model



  

Learning with auxiliary sequence labeling task requires additional data which may not be available for some languages. For this reason, several models have been proposed for training sequence labeling task with other unsupervised learning tasks. In particular, <cit.> trained single-task sequence labeling models with neural language model simultaneously.


  

In our work, we incorporate a word-level neural language model into both single and multi-task sequence labeling models to improve the performances. Specifically, we put the hidden state from BLSTM at each time step into softmax layer to predict the next and previous words. Note that we use two separate language models for each forward and backward passes of BLSTM. The objective function now is the combination of sequence labeling and language model objective functions and is computed as follows:


    E_joint = E +  λ(E_LM + E_LM)

where λ is a parameter controlled the impact of the language modeling task to the sequence labeling task and E_LM, E_LM are the objective functions of forward and backward language models. These objective functions are computed as follows:


    E_LM = -∑_t=1^Tlog(P(w_t-1| h_t))


    E_LM = -∑_t=1^Tlog(P(w_t+1| h_t))

where h_t, h_t are hidden states of forward and backward LSTM and w_0, w_T+1 are special tokens START, END. We investigate two kinds of incorporating neural language model into our multi-task sequence labeling model: shared-LM which  shares neural language model for both auxiliary and main sequence labeling task and unshared-LM which uses separate neural language model for each task. Figure <ref>fig:1d and Figure <ref>fig:1e show the difference between these two kinds of incorporating neural language model.



 §.§ Deep Contextualized Word Representations



  

Uncontextualized word embeddings such as Word2Vec <cit.>, GloVe <cit.> have been used widely in neural natural language processing models and have improved their performances. However, these word embeddings still have some drawbacks. In particular, it is difficult for them to represent the complex characteristics of a word and its meaning at different contexts. Recently, deep contextualized word representations have been proposed to solve these problems. <cit.> introduces the word representations which are computed from multi-layer bidirectional language model with character convolutions. Unlike  <cit.>, <cit.> use Transformer instead of BLSTM to calculate the language model. <cit.> improves <cit.> work by jointly learning both left and right context in Transformer.


  

Our proposed multi-task models can be trained with any kind of these contextualized word representations but in the scope of this paper, we only experiment with contextualized word representations described in <cit.> which are called Embeddings from Language Models (ELMo) and leave other contextualized word representations in our future works. ELMo are functions of the entire input sentence and are computed as follows:


    ELMo_t = γ∑_l=1^Ls_lh_t,l^LM

where h_t,0^LM is the input representation and h_t,l^LM is the output at l^th layer of L-layer bidirectional language model at time step t, s = (s_1,s_2,...,s_L) are the softmax-normalized weights and γ are the scalar parameter which allows the model to scale the ELMo vector. In our work, we incorporate a 2-layer bidirectional language model pre-trained on 1 Billion Word Language Model Benchmark dataset to our system. We set s_1 = 0 and s_2 = 1 which means we only use the output of 2^nd layer of the bidirectional language model as an input for the next layer in our system.




§ EXPERIMENTS




 §.§ Datasets



  

We conduct our experiments with FG-NER as our main task and POS tagging, chunking, NER, and language model as our auxiliary tasks. For FG-NER task, we use the English part of the dataset described in <cit.>. For chunking task, we use CoNLL 2000 dataset <cit.>. This dataset has only training and testing sets so we used one part of the training set for validation. For NER task, we use CoNLL 2003 and OntoNotes 5.0 datasets <cit.>. OntoNotes 5.0 dataset is also used for POS tagging task. The details of each dataset are described in Table <ref>.









 §.§ Training and Evaluation Method



  

The training procedure for multi-task sequence labeling models is as follows. For same-level-shared models, at each iteration, we first sample a task (main or auxiliary tasks) by Bernoulli trial based on sizes of datasets. Next, we sample a batch of training examples from the given task and then update gradients for both the shared parameters and the task-specific parameters according to the loss function of the given task. For hierarchical-shared models, at each iteration, we train the auxiliary (low-level) task first and then move to the main (high-level) task because selecting the task randomly hampers the effectiveness of hierarchical-shared models <cit.>. We use stochastic gradient descent algorithm with decay rate 0.05. Table <ref> shows the hyper-parameters we used in our models. 


  

We evaluate the performance of our system with F_1 score:

    F_1 = 2*𝚙𝚛𝚎𝚌𝚒𝚜𝚒𝚘𝚗*𝚛𝚎𝚌𝚊𝚕𝚕/𝚙𝚛𝚎𝚌𝚒𝚜𝚒𝚘𝚗+𝚛𝚎𝚌𝚊𝚕𝚕

Precision and recall are the percentage of correct named entities identified by the system and the percentage of identified named entities present in the corpus respectively. To compare fairly with previous systems, we use an available evaluation script provided by the CoNLL 2003 shared task[<http://www.cnts.ua.ac.be/conll2003/ner/>] to calculate F_1 score of our FG-NER system.



 §.§ Results






  
Base Model
Our base model is similar to LSTM + CNN + CRF model in <cit.>, but in contrast to their model, we implement by PyTorch instead of Theano and train sentences with same length at each batch to make the training process faster. It achieves F_1 score of 81.51% compared to 80.93% reported in their paper.


  
Deep Contextualized Word Representations
In the first experiment, we investigate the effectiveness of contextualized word representations (ELMo) compared to uncontextualized word representations (GloVe) when incorporating in our FG-NER systems (Base Model (GloVe) vs. Base Model (ELMo)). From Table <ref>, we see that using ELMo significantly improves the F_1 score of our system compared to using GloVe (from 81.51% to 82.74%).


  

To further investigate this phenomenon, we give an analysis to see which NE types are improved when using ELMo. Table <ref> shows F_1 scores of 5 NE types which are most improved and their average token lengths. While the average token length of NEs in our dataset is 1.9, the average token lengths of these NE types are much longer. It shows that ELMo helps to improve the performance of our system when identifying NEs which are long sequences. This result is understandable because Base Model (GloVe) relies on only BLSTM layer to learn the dependencies among words in sequence to predict NE labels while Base Model (ELMo) learns these dependencies by both embedding and BLSTM layers. Unlike NER, NE types in FG-NER are often more complex and longer so using only BLSTM layer is not sufficient to capture these dependencies.









  
Parameter Sharing Schemes
In the second experiment, we investigate the impact of training FG-NER with other auxiliary sequence labeling tasks including POS tagging, chunking, and NER by our multi-task sequence labeling models at different parameter sharing schemes. In particular, we compare three kinds of multi-task sequence labeling architectures including embedding-shared, RNN-shared, and hierarchical-shared models. The size of the original OntoNotes dataset is much larger than FG-NER dataset so it is difficult for our system to focus on learning FG-NER task. Thus, we sample 10,000 sentences from OntoNotes for training POS tagging and NER.


  

Table <ref> shows the performances of our multi-task sequence labeling models with GloVe and ELMo representations. In both cases, hierarchical-shared model gives the best performances. In particular, it achieves an F_1 score of 82.14% when learning with NER (CoNLL) and an F_1 score of 83.04% when learning with NER (Ontonotes) compared to F_1 scores of 81.51% and 82.74% of base model in GloVe and ELMo settings respectively. For same-level-shared models, they also achieve better results compared to base model but the differences are not very large. These results indicate that learning FG-NER with other sequence labeling tasks at different parameter sharing schemes helps to improve the performances of FG-NER system. Also, in most cases, it is more beneficial when learning the auxiliary and the main tasks at different levels (hierarchical-shared model) compared to learning at the same level (RNN-shared and embedding-shared models).


  

For same-level sharing scheme, we also see that embedding-shared model achieves better performances than RNN-shared model in most cases. The gap between these two models is larger when the auxiliary task is more different from the main task (POS tagging, chunking are more different from FG-NER compared to NER).



  
Neural Language Model
In the third experiment, we incorporate our systems including both single and multi-task sequence labeling models with neural language model. We experiment with two kinds of incorporating neural language model: shared-LM which shares neural language model for both auxiliary and main sequence labeling tasks and unshared-LM which uses separate neural language model for each task. For single-task model, incorporating neural language model helps to improve performance from 81.51% to 81.77% and from 82.74% to 82.91% in GloVe and ELMo settings respectively. For multi-task models, with shared-LM, our best result is an F_1 score of 82.99% when learning hierarchical-shared FG-NER model with NER (Ontonotes), and with unshared-LM, our best result is an F_1 score of 83.35% when learning hierarchical-shared FG-NER model with chunking. We also see that using unshared-LM helps our multi-task models achieves better performances compared to using shared-LM in most cases.




  
Comparison with SOTA System
Our best system achieves the SOTA result for FG-NER. In particular, our hierarchical-shared model with chunking as an auxiliary sequence labeling task and unshared-LM achieves an F_1 score of 83.35% compared to 83.14% of the previous SOTA model for FG-NER <cit.>. While that model requires significant manual effort for building a dictionary and designing hand-crafted features, our best model is truly end-to-end framework without using any additional information.



§ CONCLUSION


We present an experimental study on the effectiveness of using multi-task learning with contextualized word representations in FG-NER task. In particular, we examine the multi-task approach at different aspects including different parameter sharing schemes for multi-task sequence labeling, learning with neural language model, and learning at different word representation settings. Our best model, while does not use any additional manual effort for creating data and designing features, achieves an F_1 score of 83.35% which is the SOTA result compared to the previous FG-NER model.



named

