Improving Neural Sequence Labelling using Additional Linguistic Information
Mahtab Ahmed Muhammad Rifayat Samee Robert E. Mercer
















my_enumerate

  
  
  








[scale=0.4](0,.35) – (.25,0) – (1,.7) – (.25,.15) – cycle;

  
  
makecaption
  
  
  
  


B-.05emi-.025em b-.08em
    T-.1667em.7exE-.125emX







Improving Neural Sequence Labelling using Additional Linguistic Information



    Mahtab Ahmed, Muhammad Rifayat Samee, Robert E. Mercer
Department of Computer Science 

University of Western Ontario

London, Ontario, Canada 

mahme255, msamee, rmercer@uwo.ca
    December 30, 2023
=====================================================================================================================================================================================




Sequence labelling is the task of assigning categorical labels to a data sequence. In Natural Language Processing, sequence labelling can be applied to various fundamental problems, such as Part of Speech (POS) tagging, Named Entity Recognition (NER), and Chunking. In this study, we propose a method to add various linguistic features to the neural sequence framework to improve sequence labelling. Besides word level knowledge, sense embeddings are added to provide semantic information. Additionally, selective readings of character embeddings are added to capture contextual as well as morphological features for each word in a sentence. Compared to previous methods, these added linguistic features allow us to design a more concise model and perform more efficient training. Our proposed architecture achieves state of the art results on the benchmark datasets of POS, NER, and chunking. Moreover, the convergence rate of our model is significantly better than the previous state of the art models.



Sequence Labelling, Long Short Term Memory, Conditional Random Field, Linguistic Features.




§ INTRODUCTION


Linguistic sequence labelling is one of the first tasks focusing on natural language processing using deep learning and it has been well examined over the past decade <cit.>. Part of speech (POS) tagging, named entity recognition (NER), and chunking are subclasses of sequence labelling. They play a vital role in fulfilling many downstream applications,
such as relation extraction, syntactic parsing, and entity linking <cit.>. POS tagging assigns a tag to each word in a text, where a tag represents the lexical category of a word. NER is a subtask of information extraction that seeks to locate and classify named entities in text. Chunking identifies the POS and short phrases in a sentence by doing shallow parsing and also groups words into syntactically correlated phrases. These labelled texts can later be used for different applications such as machine translation, information retrieval, word sense disambiguation, and natural language understanding etc.  

Before neural sequence models, traditional algorithms were based on Hidden Markov Models (HMMs) <cit.> and Conditional Random Fields (CRFs) <cit.>. The problem with these models is they are heavily dependent on manually hand-crafted features. So it becomes difficult to apply them in real life applications because it is not practically possible to always have human expertise. Also it is very difficult to apply these  manually designed models in a new domain.

To overcome these drawbacks, Neural Network (NN) based models have been proposed in which the models are responsible for extracting higher level features from the data <cit.>. Recurrent Neural Networks (RNNs) along with its variants, Long Short Term Memory (LSTM) <cit.> and Gated Recurrent Units (GRUs) are found to work very well with sequence data as they can capture long distance dependencies <cit.>. Nevertheless, considering the overwhelming number of their parameters and the relatively small size of most human annotated sequence labelling corpora, annotations alone may not be sufficient to train complicated models. So, guiding the learning process with extra knowledge could be a wise choice <cit.>. For example, before tagging the word `flies' as either a verb or a noun in the sentence `Time flies like an arrow', having its semantic meaning would make a correct tagging straightforward. knowledge about How exactly are we going to tag the word `flies'?  Is it a verb or a noun?. However, if we have some understanding on the semantic meaning of the word `Time' here then it becomes easy to tag this. 

Knowing the sense of a word prior to sequence labelling (POS or NER) often gives the most probable tag for that word. Word senses can be obtained from a variety of sources: WordNet <cit.>, a lexical database for English that can be queried for the sense of a word given its context; the simplified  LESK algorithm <cit.> which uses the dictionary  definition of each word in a sentence as extra context to suggest the word sense; and linear algebraic methods, one <cit.> which uses a random walk on a discourse model and represents the vector of the base word as a linear combination of its probable sense vectors. 

In this paper, we propose a novel deep neural architecture for doing sequence labelling incorporating not only semantic features through word senses but also the rich morphology of the words. We provide an in depth analysis of the design of this architecture giving some insights regarding how each feature is introduced into the architecture. Our sequence model achieves state of the art results on the three sequence labelling tasks, POS, NER, and chunking, and has a training time at least four times faster than the currently available state of the art models. 
is the most famous and widely used NN model for sequence data but it as the sequence becomes longer it suffers a problem called Vanishing gradient problem <cit.>.

is  which goes over the entire sequence data through time and tries to remember it. However, as the sequence becomes longer, RNN based networks started to loose the context. With increase in network depth the gradient gets multiplied more and as a result the change in weight tends to become very small. This famous problem of RNN is called Vanishing gradient problem <cit.> and to overcome this some variants of the basic RNN is introduced such as Long Short Term Memory (LSTM) and Gated Recurrent Unit (GRU). These variants use gating mechanism to propagate new information further and at the same time forgets some previous information. 

With their layer wise structure the model can easily extract features within the features as it goes deeper through its layers. Even though these kind of networks have huge number of parameters which makes them easy to grab the underlying structure of the data, adding some morphological and semantic features with the model's extracted features certainly gives a huge boost in their performance. Also the structure of the basic NN models are not suitable to fit over the sequence data.




§ RELATED WORK

Huang et al. <cit.> propose a few models for the sequence tagging task. Apart from just word embeddings, they first use morphological as well as bigram and trigram information as their input features. Later, they use LSTM and Bidirectional LSTM (BLSTM) with CRF to do the final tagging. Lample et al. <cit.> extract character embeddings from both the left and right directions, concatenate these with word embeddings and then use a stacked LSTM along with CRF to do the tagging. Liu et al. <cit.> propose a model leveraging both word as well as character level features. It includes a language model to represent the character level knowledge along with a highway layer to avoid the feature collision. Finally it is trained jointly as a multitask learning.

Yu et al. <cit.> propose a general purpose tagger using a convolutional neural network (CNN). First, they use CNNs to extract the character level features and then concatenate it with word embeddings, position embeddings and binary features. Finally they use another CNN to get the contextual features as well as to do the tagging. Ma and Hovy <cit.> propose an end-to-end sequence labelling model using a combination of BLSTM, CNN and CRF. They use a CNN to get the character level information, concatenate it with word embeddings and then apply BLSTM to model the contextual information. Finally they generate the tags by using a sequential CRF layer. Rei <cit.> trains a language model type objective function using BLSTM-CRF to predict the surrounding words for every word in the corpus and utilizes it for sequence labelling.

The contribution of this paper combines the common themes found in these previous works (morphology encoded as character embeddings, and word embeddings) with word senses in a new architecture that integrates these embeddings and the outputs of a CNN, a BLSTM, and a CRF in novel ways.



§ THE MODEL: BLSTM-CRF


In this section, we describe our work in detail. We first explain each of the pieces of the complete architecture and then we explain how we combine those pieces to build our model. This section also explains the morphological and semantic features that we have added with our model to get the improved performance that is discussed in Section <ref>.




 §.§ Recurrent unit: Bidirectional LSTM




Recurrent neural networks (RNNs) are the best know and most widely used NN model for sequence data as they go over the entire sequence through time and try to remember it in a compressed form. Although in theory, RNNs are capable of remembering long distance dependencies, practically, as the sequence becomes longer, RNN based networks start to lose the context. As the sequence length increases, the gradient of the error function gets multiplied more and more and as a result, the change in the NN weights tends to become very small. This well known problem of RNN is called the vanishing gradient problem <cit.>. To overcome this drawback some variants of the basic RNN have been introduced such as Long Short Term Memory (LSTM) <cit.> and Gated Recurrent Unit (GRU) <cit.>. These variants use a gating mechanism to propagate new information further and at the same time forgets some previous information allowing the gradients to propagate further in time.

 
    i_t = σ(W_xix_t + W-hih_t−1 + W_cic_t−1 + b_i

 
    f_t = σ(W_xfx_t + W-hfh_t−1 + W_cfc_t−1 + b_f

 
    c_t = f_tc_t−1 + i_ttanh(W_xcx_t + W_hcht−1 + b_c)

 
    o_t = σ(W_xox_t + W_hoh_t−1 + W_coc_t + b_o)

 
    h_t = o_t tanh(c_t )





 §.§ Recurrent unit: Bidirectional LSTM


Recurrent neural networks (RNNs) are the best known and most widely used NN model for sequence data as they go over the entire sequence through time and try to remember it in a compressed form. 
Although its variant, LSTM, is very good with long term dependencies, for many sequence labelling task, it is important to keep track of these dependencies from the future as well as from the past. But LSTM has just one hidden state from the past and changes that hidden state recursively through time. An elegant solution to this problem is going over the sequence in both forward and backward directions with two hidden states and finally concatenating the output from both directions. This bidirectionality has proven to be very effective in some prior works <cit.>. The resulting network, the Bidirectional LSTM (BLSTM), is the RNN variant that is used by the model described below.



 §.§ Word Sense


As we are in the framework of sequence modeling, we needed an algorithm which gives the sense embedding of a word so that we can initialize an embedding layer at the beginning of our model with the pre-trained senses.


Knowing the sense of a word prior to tagging makes the tagging task more straightforward. Generally, polysemy is captured in standard word vectors, but the senses are not represented as multiple vectors. So we have trained an adaptive skip gram model, AdaGram, <cit.> which gives a vector for each sense of a word. It is a non-parametric Bayesian extension of the skip-gram model and is based on the constructive definition of Dirichlet process (DP) <cit.>. It can learn the required number of representations of a word automatically. The advantage of this kind of trainable model is that we can train our own corpus with it allowing the possible sense vectors to be correlated with the specific context within corpus. 

In our model, we denote a set of input words as X= {x_i}_i=1^N and their context as Y= {y_i}_i=1^N. The ith training pair (x_i, y_i) consists of words x_i = o_i with context y_i = (o_t)_t ∈ c(i), where c(i) is the index of the context words. Then, instead of maximizing the probability of generating a word given its contexts <cit.>, we maximize the probability of generating the context given its corresponding input words <cit.>. Our final objective function becomes,


    p(Y|X,θ) = ∏_i=1^N p(y_i|x_i, θ) = ∏_i=1^N∏_j=1^C p(y_ij|x_i, θ)

where, θ is the set of model parameters. The drawbacks of this objective function is that it captures just one representation of a word which goes against a word having different senses depending on the context <cit.>. To counter this, AdaGram introduces a new latent variable z  which captures the required number of senses even though the number of structure components of the data is unknown a priori. Being unsupervised, DP can increase the required number of clusters through variational inference and there always remains a probability of introducing a new cluster.In AdaGram, if the similarities of a word vector with all its existing sense vectors are below a certain threshold, a new sense is assigned to that word with a prior probability p. The prior probability of the kth meaning of word w is 

    p(z=k|w,β)=β_wk∏_r=1^k-1(1-β_wr), 
     p(β_wk|α)=Beta(β_wk1,α), 
    k=1…

where β is a latent variable and α controls the number of senses. Theoretically, it is possible to have an infinite number of senses for each word w. However, as long as we have a finite amount of data, the number of senses can not be more than the number of occurrences of that word. With more data, it can increase the complexity of the latent variables thereby allowing more distinctive meanings to be captured. Taking all the facts into account, our final objective function becomes, 

    p(Y,Z,β|X,α, θ)=∏_w=1^V∏_k=1^∞p(β_wk|α) 
    ∏_i=1^N[p(z_i|x_i,β)∏_j=1^C[p(y_ij|z_i,x_i,θ)]

where Z = {z_i}_i=1^N is a set of senses for all the words.


 §.§ Convolutional Neural Network
 

Convolutional Neural Networks (CNNs) are good for extracting n-gram features from a sentence <cit.>. They consist of kernels (i.e., a matrix of weights) which are used to go over the input word embedding matrix with a variable stride length and extract some higher level features. In our model, we use a CNN to get the bigram features. To do that, first we pad the input sentence [The `valid' convolution operation reduces the size of the feature matrix by k-L, where k is the size of the kernel vector and L is the stride length. For us, L = 1. To keep the size of the feature matrix uniform through the model, we padded a start token start at the beginning of the input sentence.] and pass it to an embedding layer. The embedding weights were initialized randomly and were trained along with other network parameters.This layer represents the sentence as a matrix of size (m+1) × d, where m is the actual sentence length and d is the embedding dimension. Next, we initialize a kernel of size 2 × d with stride length 1 and convolve it with the input sentence matrix. This results in a bigram embedding matrix B of size m × d using Eqn. <ref>.

    B_i,:= ∑^2_j=1I_i+j,:*K_j,:
  
where I is the input sentence matrix, K is the convolution kernel and n is the maximum sequence length for the current batch. Later, this bigram embedding is passed to a BLSTM layer to extract  more abstract features.



 §.§ Conditional Random Field

Each of the tasks that we are modelling requires a tag to be assigned to each word. In addition to using the current word to predict its tag, it is also possible to use the information about the neighboring words' tags. There are two main ways to do this. One way is to calculate the distribution of tags over each time step and then use a beam search-like algorithm, such as maximum entropy markov models <cit.> and maximum entropy classifiers <cit.>, to find the optimal sequence. Another way is to focus on the entire sentence rather than just the specific positions which leads to Conditional Random Fields (CRFs) <cit.>. CRFs have proven to give a higher tagging accuracy in cases where there are dependencies between the labels. Like the bidirectionality of BLSTM networks a CRF can provide tagging information by looking at its input features bidirectionally.

In our model we denote a generic input sequence as x= {x_i}_i=1^N, generic tag sequence as y= {y_i}_i=1^N, and set of possible tag sequences of x as F(x). Then we use CRF to calculate the conditional probability over all possible tag sequences y given x as 

    p(y|x;W,b) = ∏_i=1^nϕ_i(y_i-1,y_i;x)/∑_y'∈ F(x)^∏_i=1^nϕ_i(y'_i-1,y'_i;x)
  
where ϕ(.) is the score function for the transition between the tag pair (y',y) given x. We train this CRF model using maximum likelihood estimation (MLE) <cit.>. For a training pair (x_i, y_i) we maximize 

    L(W,b) = ∑_i^log p(y|x;W,b)
  
where W is the weight matrix and b is the bias term. While decoding, we search for the best tag ŷ with the highest conditional probability using the Viterbi algorithm <cit.>.

    ŷ = _y ∈ F(x)p(y|x;W,b)
  



 §.§ Morphology: Spelling and suffix features
 
For the morphological features, we have focused on spelling features and suffix features.

We extract the following 14 spelling features for a given word and store it as a binary vector SV_1 × 14.


  
  * whether it is composed only of alphabetics or not
  
  * whether it contains non-alphabetic characters except `.' or not
  
  * whether it starts with a capital letter or not
  
  * whether it is composed only of upper case letters or not
  
  * whether it is composed only of lower case letters or not
  
  * whether it is composed only of digits or not
  
  * whether it is composed of a mixture of alphabetics and numbers or not
  
  * whether it is the starting word in the sentence or not
  
  * whether it is the last word in the sentence or not
  
  * whether it is in the middle of the sentence or not
  
  * whether it ends with an apostrophe s ('s) or not
  
  * whether it has punctuation or not
  
  * whether it is the first word in the sentence and starts with a capital letter or not 
  
  * whether it is composed mostly of digits or not

Apart from extracting these features, we also replace all the numbers in the corpus with the numbertag. 


We have assembled a list of 137 suffixes from <https://www.learnthat.org/pages/view/suffix.html> and have used the ten that occur most often in our corpus for this study. Then for each of these suffixes, we have collected the words that end with that suffix and have recorded their POSs as well as the frequency. Next, we made an assumption that if a word w with POS x ends with a specific suffix s exceeds a frequency threshold in the training set, then s is the true suffix of word w. We record the pair as (w,s). Finally, we create a one hot vector SUV_1 × 10 for each word where a 1 at index k means the word has the kth suffix.



 §.§ BLSTM-CRF model

In this sub-section, we combine the BLSTM and CRF models with some feature connection techniques to form our final BLSTM-CRF model. We divided this final model into some modules and the description of each of these modules is as follows:

Module 1: Word Level Features This module starts with an embedding layer. In detail, we initialize the emebedding layer randomly as well as using pre-trained embeddings (GloVe / word2vec). Next we represent each sentence as a column vector I_m × 1 where each element of the vector is a unique index of the corresponding word. Then we pass this vector to an embedding layer which gives a matrix representation W_m × d. Here, d is the embedding dimension. 

Module 2: Character Embedding In this module, first we split a word into its characters and then transform it into a column vector C_k × 1, where k is the word length and each element of the vector is a unique index of the corresponding character. Next we initialize an embedding layer randomly and pass the character vector into it. This will change the representation to a matrix of size k × n where  n represents the embedding dimension. Then we use an LSTM on this matrix and store the last hidden state of this LSTM as the character level representation C_1 × n of the word. Finally, for a sentence with m words, it is stored as a matrix C_m × n.

Module 3: Selective Pickup from Char LSTM (SP-CLSTM) In this module, we introduce a new way of capturing the morphological features as well as the context features. The word embeddings from module 1 gives the contextual features in both directions and the character embeddings from module 2 gives the lexical information. We capture both sets of information by first representing each sentence in terms of its characters I_(k× m) × 1 and then turn this into a matrix of size k × m × d through a random embedding layer. Then we apply a BLSTM over this representation and finally we pick those indices from the output where each word ends. This selective pickup provides the morphological information of a word as well as information about the previous words in the sequence.

    C̃_m × d =  ((I_(k× m) × d))


Module 4: Sense Features This module calculates the sense level contextual features of a sentence. First, we initialize a sense embedding layer using the pre-trained sense embeddings from AdaGram. Then we tag each word in the input sentence using the module  from AdaGram (the word `apple' with sense 2 is tagged as `apple2'). This modified input sentence is then passed to the embedding layer initialized before and finally the resultant output is passed to a BLSTM layer. The output of this BLSTM layer gives the sense level contextual feature S_m × d.

Module 5: Bigram Features This module calculates the bigram embedding features B_m × d of a sentence as described in the Subsection <ref>.

Module 6: The Connection Technique In this module, we combine all the features and the modules using some novel connection techniques and build our final BLSTM-CRF model as shown in Figure <ref>. First we concatenate the word embedding from module 1 with the character embedding from module 2 and the suffix vector from Subsection <ref> as [W_m × d, C_m × n, SUV_m × 10 ]. Following this, we apply a BLSTM on this new embedding matrix, calling this output O^1_m × d. The outputs of modules 3, 4 and 5 are called O^2_m × d, O^3_m × d, and O^4_m × d, respectively. Then we initialize four scalar weights w_1, w_2, w_3 and w_4 with initial value 1.0 and add them as model parameters. We form a linear combination of the w_i weighted O^is to form the final output.


    O = ∑_i=1^4 O^i w_i


The final output (O_m × d) have pieces of information from all the features that we calculated above. We choose linear addition rather than concatenation of these output features, because concatenation will result in a very large feature matrix and the network have to tune each of the cell of this matrix during back-propagation. Following this, we initialize an LSTM layer where we pass the final output from Eqn. <ref> at each time step Õ^i_1 × z = (O^i_1 × d, h^i-1_1 × d) and store the outputs separately  Õ_m × z = [Õ^1, Õ^2, … , Õ^m]. This LSTM layer unfolds at each time step taking the hidden state of the previous time step to initialize the hidden state of the current time step. The previous hidden state has the information about the previous tag and initializing the current hidden state with the previous one explicitly gives this information. Next we pass the output from each time step to a  layer T_1 × d = (Õ^i), which squeezes the values between [-1,1]. Then we concatenate this  output T_1 × d with the spelling features SF_1 × 14 calculated in subsection <ref> and pass this to a fully connected (FC) layer. This FC layer maps the output to the number of tag classes Y_1 × c =  ([T_1 × d, SUV_1 × 14]), where c represents number of classes. We do this for each time step and concatenate the results to make a final tensor Y_m × c. Finally, we pass this tensor to the CRF layer and calculate the possible tag sequence for the given input sequence. 



§ EXPERIMENTAL SETUP

In  this  section,  we describe  the  detailed  experimental setup for the evaluation of our study. We first explain the dataset statistics for each tagging task. Following this, we explain the working environment details along with the hyper-parameter settings of our architecture.



 §.§ Dataset Description


We test our BLSTM-CRF model on three NLP tagging tasks: Penn TreeBank (PTB) POS tagging, CoNLL 2000 chunking, and CoNLL 2003 NER. Table <ref> shows the number of sentences in the training, validation and test sets respectively for each corpus. POS allocates each word with an unique label that shows its syntactic part. Chunking represents each word in terms of its phrase type. NER labels each word with one of four substance: Person, Location, Organization, or Miscellaneous.We utilize the BIO2 explanation standard for the chunking and NER tasks.




Table <ref> shows the detailed hyper-parameter settings of our model and some of the hyper-parameters for AdaGram (the remaining parameters are set to their default values <cit.>). We train our model on Nvidia GeForce GTX 1080 GPU with both the `Adam' and `SGD' optimizers. All of the results in the next section are reported using `SGD' as it was giving the best results. The `Learning rate decay' parameter was only used with the `SGD' optimizer. We used PyTorch 0.3.1 to implement our model and Julia 0.4.5 for running AdaGram under the Linux environment.



§ EXPERIMENTAL RESULTS




In this section, we describe in detail the results obtained with our proposed architecture. As the evaluation metrics, we use accuracy for the WSJ corpus and F1 score (micro averaged) for the CoNLL00 and CoNLL03 tasks. This section also contains the results of the top performing models for all three sequence labelling tasks. Additionally, we show the rate of convergence of our model compared to the state of the art one. Finally, we conclude this section by giving an ablation study by removing certain modules as well as features and mixing them in different combinations.


Table <ref> shows the performances of all of the chunking systems. An SVM based classifier <cit.> won the CoNLL 2000 challenge with an F1 score of 93.48%. However, later they improved their result up to 93.91% <cit.>. Recently, most of the models incorporate CRF in their architecture to capture the tag dependencies and achieve very good performance <cit.>. However, none of them surpass the performance of <cit.> which uses an HMM to capture the dependencies and a voting scheme to increase the confidence interval of the model. Our model outperforms all the existing models and achieves a state of the art F1 score of 96.76%.



Table <ref> shows the results of the existing models on the NER task. Huang et al. <cit.> did many experiments using random and pre-trained embeddings on their model. For random embeddings, they achieved a very low score of 84.26%. However, when they use pre-trained SENNA embeddings <cit.> along with a gazetteer feature, their F1-score jumped up to 90.10% surpassing the Conv-CRF model <cit.> which uses window and sequence approach networks to do the tagging. Our model achieves a state of the art result of 91.63%.





Table <ref> shows the performance of our architecture in comparison with some top performing ones for the POS tagging task. As can be seen, a number of models use Convolution or LSTM or BLSTM to get the contextual features and CRF to do the tagging. They achieve very good accuracies of 97.29% <cit.>,  97.51% <cit.> and 97.55% <cit.>. Some of the models use multitask learning, doing two or more tasks at the same time. They also achieve very good accuracies: 97.43% <cit.> and 97.59% <cit.>. However, the problem with these type of models is that it requires language modeling which itself is a very costly framework and the training time is very long.Our model achieves an accuracy of 97.58% which is higher than all of the existing models except LM-LSTM-CRF <cit.> which leverages a language model for the tagging tasks. LM-LSTM-CRF, however, has a mean accuracy of 97.53% (reported accuracy: 97.53±0.03) which is lower than the our model's mean accuracy (97.57±0.01). Also, as shown in Table <ref>, our model's training time is one quarter that of LM-LSTM-CRF with on par performance.

Table <ref> gives the ablation study of our model where we show how we apply different combinations of features in different parts of our BLSTM-CRF architecture to get an optimal configuration. With so many features and parameters, these sequence models are very much prone to overfit. But with careful tuning as well as with proper feature connections, it is possible to leverage those features. We extract a set of morphological as well as semantic features from our dataset such as spelling, suffix and char-level features. We experiment on applying various combinations of these features in different segments of our model. Our extensive experimentation shows that optimal results are achieved when these features are added in the model through residual connection, concatenation with word embeddings and concatenation with the second last output layer. Focusing on which segment to connect each feature, our experiments found that the spelling feature works best when concatenated with the second last output layer, and the suffix feature as well as the character embeddings work well when concatenated with the word embeddings. This configuration is what is kept in our final model. We further continue our experiments by turning on / off different modules such as word embedding, sense embedding, selective pickup from LSTM and bi-gram embedding. We found that the contribution of word embeddings, sense embeddings and selective pickup from LSTM are significant compared to the bigram module as shown by the weights at the 500th epoch in Table <ref>. The bigram module works better whenever we do not consider the previously generated part-of-speech. So we kept the first three modules and discarded the bigram module from our final model. Our best model as shown in the last row of Table <ref> gives state of the art results. 



§ CONCLUSION

In this paper, we propose an improved neural sequence labelling architecture by leveraging from additional linguistic information such as polysemy, bigrams, character level knowledge and morphological features. Benefitting from such adequately captured linguistic information, we can assemble a considerably more compact model, hence yielding much better training time without loss of effectiveness. To avoid feature collision we performed an extensive ablation study where we produced an optimal model structure along with an optimal set of features. Our best model achieved state of the art results on the POS tagging, NER and chunking benchmark datasets and at the same time remains four times faster to train than the best performing model currently available. Our experimental results show that multiple linguistic features and their proper inclusion significantly boosted our model performance.


IEEEtran


