\section{Introduction}
\input{sections/intro}

\section{Methodology}
\label{sec:method}
\input{sections/methodology}


\section{Experiment}
\label{sec:exp}
\input{sections/experiments}


\section{Conclusions}
\input{sections/conclusion}
In this paper, we have proposed Mask Augmented Named Entity Recognition (MANER) for NER in extreme low-resource language settings. 
MANER exploits the information encoded in pre-trained masked language models (inside \textit{<mask> } token specifically) and outperforms existing approaches for extreme low-resource languages with as few as only 100 training examples by up to $48\%$ and by $12\%$ on average on F1 score.
Analyses and ablation studies show that use of semantics encoded in \textit{<mask> } token is integral to MANER.
Future work will target using MANER for rapid prototyping in an active learning setup since it offers a cost-effective way to supplement human annotators. 







\subsection{Dataset, Models, and Baselines}
\textbf{Dataset:} 
We use the WikiANN multilingual NER dataset \citep{panx,wikiann}, which provides named entities for Wikipedia articles across 176 languages.
To the best of our knowledge, WikiANN is by far the most comprehensive dataset for multilingual NER.
There are other multilingual datasets (e.g., CoNLL \cite{CoNLL2002,CoNLL2003}), but they cover only four popular languages:  English, German, Dutch, and Spanish.

Our main emphasis in this paper is on the 100 languages in  WikiANN that each have only \textbf{100 samples} for train and test splits.
The NER labels in WikiANN are in IOB2 (Inside–outside–beginning) format \cite{iob} comprising PER (person), LOC (location), and ORG (organization) tags.
An instance of NER tagged sentence: \textit{UNICEF}(B-ORG) is a nonprofit  organization, founded by \textit{Ludwik}(B-PER) \textit{Rajchman}(I-PER) headquartered at  \textit{New}(B-LOC) \textit{York}(I-LOC), \textit{United}(B-LOC) \textit{States}(I-LOC).

\textbf{Model:} We use XLM-RoBERTa-large \cite{xlm_roberta} as the transformer backend for MANER. XLM-RoBERTa-large is a multilingual version of the RoBERTa \cite{roberta} transformer model, designed for NER experiments on WikiANN.
XLM-RoBERTa model has been pre-trained using the MLM objective on 2.5TB of filtered CommonCrawl data containing 100 languages \cite{cc100}.
We chose this model, since it is the largest publicly available multilingual transformer based model.

\textbf{First Baseline:} 
Similar to MANER  design, current NER systems built upon transformer models also simply add a NER classifier to top of a transformer model.
The classifier predicts the NER class of each token of an unmodified sentence $S$:
\begin{align*}
        \mc{M}_{\rm base}(T, \v{M}, S, i) =  \rm{softmax}\Bigl(\v{M}\bigl(\v{e}_i\bigl)\Bigl) = \v{p}_{i}, \,
    \\ 0 \leq i <n,
\end{align*}
where $\mc{M}_{\rm base}$ is an NER model built on a transformer model $T$ using classifier weight matrix $\v{M}$.
This baseline method remains the de-facto method for training NER models for most languages (especially low-resource languages) to the best of our knowledge, though specialized models have been built for popular languages like English.

\textit{Inference:}
Similar to training during inference, the NER class of each word in the sentence is the most probable NER tag assigned to the classified word embedding.

\textbf{Second Baseline:}
Our MANER methodology in Section \ref{sec:method} is one way to change the input phrase using the mask token.
In this second baseline, we introduce yet another way to repurpose the \textit{<mask> } token for NER that is inspired by the masked language modeling (MLM) framework that is used for pre-training transformer models.
In MLM, a word is predicted using the words surrounding it in the sentence.
Since the NER category of a word is also a semantic property of the word, we use the philosophy of MLM for NER fine-tuning.

In MLM pre-training, the dataset is prepared by \textit{masking} random words in a sentence with a \textit{<mask> } token with a fixed probability $p_{mlm}$.
Then, the masked words are predicted using the context information.

Analogous to MLM pre-training, for NER fine-tuning, we randomly replace words in sentence $S$ with the \textit{<mask> } token with the fixed probability $p_{ner}$.
However, instead of predicting the missing words, as with MLM, we predict the NER labels $L$ for each word $w$ in $S$ irrespective of whether the word was replaced by a \textit{<mask> } token or not.
In the case the word was replaced with the \textit{<mask> } token, the transformer outputs the \textit{<mask> } token embedding for that word.

Thus the modified input to the transformer is $S' = \rm{mask}(S)$, where 
\begin{equation}
    \rm{mask}(w_i) = 
    \begin{cases}

        \text{\textit{<mask> }},& \text{if } p_i \leq p_{ner},\\
        w_i,              & \text{otherwise}
    \end{cases}
\end{equation}
with $p_i$ a random number between $0$ and $1$ generated for $w_i$.
Then, we use the first baseline NER model design $\mc{M}_{base}$ for training, but now it is is fine-tuned on $S'$ and $L$ (note we predict the label of the \textit{<mask> } tokens as well).
Inference with this model remains same as the first baseline model.

\subsection{Quantitative results and comparison of different strategies to repurpose \textit{<mask> }}

We train an XLM-RoBERTa-large model for 100 languages for 30 epochs with a learning rate of $5\mathrm{e}^{-6}$ and the loss optimized using Adam \cite{adamw}.
We report the average of F1
score for 100 languages in WikiANN that have 100 training samples each in Table \ref{tab:ner_main}.

\begin{table}[h]
\resizebox{\columnwidth}{!}{%
\begin{tabular}{|c|c|c|c|}
\hline
\textbf{Metric} & \textbf{Baseline 1} & \textbf{Baseline 2} & \textbf{MANER } \\ \hline
F1 & 0.649 & 0.643 (-0.5\%) & 0.715 (12\%) \\ \hline
\end{tabular}
}
\caption{Average F1 scores for the 100 languages in the WikiANN dataset that have only 100 samples for the different strategies outlined in Figure \ref{fig:method}. In the low-resource language setting, MANER achieves a significant average improvement of \textbf{12\%}.}
\label{tab:ner_main}
\end{table}

\begin{figure*}[t]
    \centering
    \includegraphics[width=\linewidth]{images/ner_sample.png}
    \caption{Measure effect of training samples to performance in MANER. MANER  can give a boost in performance till 400 samples and then both MANER  and first baseline NER model perform similarly. This demonstrates that MANER  is best suited for extreme low resource languages and rapid prototyping since it is easy and cost-effective to obtain very few human annotations to achieve large performance improvements (just 100 annotations are required).}
    \label{fig:sample}
\end{figure*}

MANER provides a significant \textbf{12\%} average improvement in F1 score in the low-resource language setting. 
Moreover, MLM based baseline 2 performs similarly to baseline 1.

Our interpretation for why the second baseline does not improve over MANER  (even though both methods use the \textit{<mask> } token), is primarily because of two reasons,
First, there is a difference between the train and inference setting in the second baseline.
In the inference setting of second baseline, the \textit{<mask> } token is not used to classify the token, unlike in MANER .
\textit{In MANER , the model can learn to give more importance to the context in the case of out-of-distribution test labels using the \textit{<mask> } token during inference and vice versa, explaining its significant 12\% improvement as compared to both baselines.}

The second reason is because the \textit{<mask> } token introduces noise in the context during training of the second baseline model. 
Hence, it loses critical information that makes the NER task more difficult for the model.
However, in MANER , the \textit{<mask> } token is used in conjunction with the word to be labelled.

In Figure \ref{fig:100_strategy}, we plot the F1 score of all of the 100 low-resource languages comparing the first baseline against MANER . One can see that there are only a few languages (12 out of 100) in which the first baseline outperforms MANER.

\subsection{Importance of the \textit{<mask> } token}

We now conduct experiments to demonstrate the importance of using the \textit{<mask> } token in MANER.
Intuitively the \textit{<mask> } token can be helpful because it encodes the semantics of the context and, thereby, the word (by distributional hypothesis) that needs to be tagged.

\textbf{Control token:} In the first experiment to gauge the importance of the \textit{<mask> } token, we replace the token with a control/random token (\textit{<rand>}) in MANER .
Note that the \textit{<rand> } token is not learned during the XLM-RoBERTa model pre-training; thus it will not encode any contextual information.
As we see in Table \ref{tab:ner_control}, if we replace the \textit{<mask> } token with \textit{<rand> } in MANER, we achieve only a 6\% improvement in F1 performance over the first baseline.


\begin{table}[h]
\resizebox{\columnwidth}{!}{%
\begin{tabular}{|c|c|c|c|}
\hline
\textbf{Metric} & \textbf{Baseline 1} & \vtop{\hbox{\strut \textbf{MANER }}\hbox{\strut \textbf{(w/ \textit{<mask>})}}} & \vtop{\hbox{\strut \textbf{MANER }}\hbox{\strut \textbf{(w/ \textit{<rand>})}}} \\ \hline
F1 & 0.649 & 0.715 (12\%) & 0.679 (6\%) \\ \hline
\end{tabular}
}
\caption{Average F1 scores for 100 languages for MANER  using the \textit{<mask> } token and the \textit{<rand> } token.
}
\label{tab:ner_control}
\end{table}

\textit{Why does \textit{<rand> } token provide a 6\% gain in F1?} We believe that MANER  provides a significant 6\% boost in F1 score even without using the \textit{<mask> } token because the model can use the \textit{<rand> } token to predict how much weight to assign the context and the word immediately adjacent to it depending on the test sample.

\textbf{XLM-RoBERTa pre-training languages:}
In the second experiment to gauge the importance of the \textit{<mask> } token, we report the F1 score in Table \ref{tab:ner_pretrain} averaging only on those languages that the XLM-RoBERTa model was pre-trained on with at least 0.5GB of training data per language.
The rationale behind this experiment design is that the \textit{<mask> } token will encode the context semantics of a language only if the language was seen during the pre-training stage of XLM-RoBERTa model.

As we see in Table \ref{tab:ner_pretrain}, in this case, MANER  provides a whooping 18\% improvement in F1 score (as compared to the 12\% gain in Table \ref{tab:ner_main}) if the language was seen in the pre-training stage.
This experiment again highlights the importance of using \textit{<mask> } token in MANER.

\begin{table}[h]
\centering
\resizebox{0.75\columnwidth}{!}{%
\begin{tabular}{|c|c|c|c|}
\hline
\textbf{Metric} & \textbf{Baseline 1} & \textbf{MANER } \\ \hline
F1 & 0.603 & 0.705 (18\%) \\ \hline
\end{tabular}
}
\caption{
F1 scores comparing MANER  against the first baseline. F1 scores are averaged on languages that XLM-RoBERTa model was pre-trained on with at least 0.5GB of training data per language. The improvement over the baseline is 18\% compared to 12\% improvement in previous study that included all 100 languages.}
\label{tab:ner_pretrain}
\end{table}

\subsection{Effect of training samples}

In the previous subsections, we showed, by exploiting the context semantics encoded in the \textit{<mask> } token, MANER  can achieve significant gains over the baseline NER methodology for low-resource languages.
In this section, we measure the effectiveness of MANER in situations where more training data is available.
For this purpose, we select 4 languages from WikiANN dataset that have 1000 training data samples each.
From Figure \ref{fig:sample}, we see that MANER  boosts F1 performance over the the first baseline until about 400 samples and then both methods perform similarly. 
This demonstrates that MANER  is best suited for \textit{extreme low resource languages and rapid prototyping} since it is \textit{easy and cost-effective to obtain very few human annotations to achieve large performance improvements} (just 100 annotations are required).

The \textit{catastrophic forgetting} \cite{forget} phenomenon that masked language models undergo during any kind of fine-tuning is one of the reasons we think MANER does not provide gains when more training data is available (of course more training data also implies less reliance on specialized techniques like ours).
Catastrophic forgetting causes the loss of useful context semantics encoded in the \textit{<mask> } token during the fine-tuning stage that MANER heavily relies on.
Adding an additional masked language modeling loss to the NER loss during fine-tuning can help to circumvent catastrophic forgetting, but its investigation is left for future work.


\subsection{Contributions}
\textbf{Contributions:}
In this work, we propose Mask Augmented Named Entity Recognition (MANER), a new approach for NER of low-resource languages that does not rely on additional data or modify existing model architectures.
MANER exploits the semantic information encoded in a pre-trained masked language model (MLM). 
Specifically, we reformat the input to the MLM by prepending a mask token to every token in the text to be annotated with NER tags.
This reformatted input is then used to fine-tune the MLM with a randomly initialized NER prediction head on top of the prepended mask tokens.

Extensive experiments on 100 extremely low-resource languages (each with only 100 training examples) 
demonstrate that MANER improves over state-of-the-art approaches by up to $48\%$ and by $12\%$ on average on F1 score.
Detailed ablation and analyses of MANER demonstrate the importance of using the encoded semantic information (\textit{<mask> } token vs.\ new random token, languages seen during pre-training vs.\ languages seen only during fine-training), and performance with respect to the number of training examples).

\begin{figure*}[t]
    \centering
    \includegraphics[scale=0.4]{images/strategies.png}
    \caption{Proposed strategy to modify the input sentence using the \textit{<mask> } token which serves as input to our proposed 
   
    Mask Augmented Named Entity Recognition
    (MANER) model.}
    \label{fig:method}
\end{figure*}












