{'fraction_non_alphanumeric': 0.040899503056737334, 'fraction_numerical': 0.021879804082800044, 'mean_word_length': 4.576555023923445, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Optimizing Bi-Encoder for Named Entity Recognition via Contrastive Learning

We present an efficient bi-encoder framework for named entity recognition (NER), which applies contrastive learning to map candidate text spans and entity types into the same vector representation space. Prior work predominantly approaches NER as sequence labeling or span classification. We instead frame NER as a metric learning problem that maximizes the similarity between the vector representations of an entity mention and its type. This makes it easy to handle nested and flat NER alike, and can better leverage noisy self-supervision signals. A major challenge to this bi-encoder formulation for NER lies in separating non-entity spans from entity mentions. Instead of explicitly labeling all non-entity spans as the same class Outside (O) as in most prior methods, we introduce a novel dynamic thresholding loss, which is learned in conjunction with the standard contrastive loss. Experiments show that our method performs well in both supervised and distantly supervised settings, for nested and flat NER alike, establishing new state of the art across standard datasets in the general domain (e.g., ACE2004, ACE2005) and high-value verticals such as biomedicine (e.g., GENIA, NCBI, BC5CDR, JNLPBA).


Introduction
Named entity recognition (NER) is the task of identifying text spans associated with proper names and classifying them into a predefined set of semantic classes such as person, location, organization, etc. As a fundamental component in information extraction systems (Nadeau and Sekine, 2007), NER has been shown to be of benefit for a number of downstream tasks such as relation extraction (Mintz et al., 2009), coreference resolution (Chang et al., 2013), and fine-grained opinion mining (Choi et al., 2006). Inspired by its recent success of open-domain question answering (Karpukhin et al., 2020), and entity linking Zhang et al., 2021a), we propose an efficient BI-encoder for NameD Entity Rcognition (BINDER). Our model employs two encoders to separately map text and entity types into the same vector space, and it is able to reuse the vector representations of text for different entity types (or vice versa), resulting in a faster training and inference speed. Based on the vector outputs of bi-encoder, we propose to use span-based contrastive learning for NER, which differs us from existing work that mostly formulates NER as a classification problem (e.g., Chiu and Nichols, 2016;Ma and Hovy, 2016;Li et al., 2020;. 1 Through contrastive learning, we force the representation of entity type to be similar with the positive entity spans, and to be dissimilar with the negative text spans. On the other hand, existing work labels all non-entity tokens/spans as the same class Outside (O), which can introduce false negative noises when the training data is partially annotated (Das et al., 2022;Aly et al., 2021). Instead of explicitly labeling non-entity spans as O, we introduce a novel dynamic thresholding strategy in contrastive learning, which learn dynamic thresholds to distinguish entity spans from non-entity spans.
To the best of our knowledge, we are the first to optimize bi-encoder for NER via contrastive learning. We conduct extensive experiments to evaluate our method in both supervised and distantly supervised settings. The experimental results demonstrate that our method achieves the state of the art on a wide range of NER datasets, covering both the general domains and the biomedical domains. In supervised NER, compared to the previous state of the art, our method obtains a 1.4%-2.4% abso-lute improvement in F1 on standard nested NER benchmarks such as ACE2004 and ACE2005, and a 1.2%-1.9% absolute improvement on standard flat NER benchmarks such as BC5-chem, BC5disease, and NCBI. In distantly supervised NER, our method obtains a 1.5% absolute improvement in F1 on the BC5CDR dataset. We further study the impact of various choices of components in our method, and conduct breakdown analysis at entity type level and token level, which suggests further room for improvement.

Related Work
Supervised NER Early techniques for NER are based on hidden markov models (e.g., Zhou and Su, 2002) and conditional random fields (CRFs) (e.g., McDonald and Pereira 2005). However, due to the inability to handle nested named entities, techniques such as cascaded CRFs (Alex et al., 2007), adpated constituency parsing (Finkel and Manning, 2009), and hypergraphs (Lu and Roth, 2015) are developed for nested NER. More recently, with the advance in deep learning, a myriad of new techniques have been used in supervised NER. Depending on the way they formulate the task, these techniques can be categorized as NER as sequence labeling (Chiu and Nichols, 2016;Ma and Hovy, 2016;Katiyar and Cardie, 2018); NER as parsing (Lample et al., 2016;; NER as span classification (Sohrab and Miwa, 2018;Xia et al., 2019;Ouchi et al., 2020;; NER as a sequence-to-sequence problem (Straková et al., 2019;Yan et al., 2021); and NER as machine reading comprehension (Li et al., 2020;Mengge et al., 2020). Unlike previous work, we formulate NER as a metric learning problem. The span-based design of our bi-encoder and contrastive loss provides us with the flexibility to handle both nested and flat NER.
Distantly Supervised NER Distant supervision from external knowledge bases in conjunction with unlabeled text is generated by string matching (Giannakopoulos et al., 2017) or heuristic rules (Ren et al., 2015;Fries et al., 2017). Due to the limited coverage of external knowledge bases, distant supervision often has a high false negative rate. To alleviate this issue, Shang et al. (2018) design a new tagging scheme with an unknown tag specifically for false negatives; Mayhew et al. (2019) iteratively detect false negatives and downweigh them in training; Peng et al. (2019);Zhou et al. (2022) address overfitting to false negatives using Positive and Unlabeled (PU) learning; Zhang et al. (2021b) identify dictionary biases via a structural causal model, and de-bias them using causal interventions.  introduce a calibrated confidence estimation method and integrate it into a self-training framework. Without replying on sophisticated de-noising designs, our biencoder framework can be directly used in distant supervision. Experiments in §4.4 will show the robustness of our contrastive learning algorithm to the noise in distantly supervised NER.
Bi-Encoder The use of bi-encoder dates back to Bromley et al. (1993) for signature verification and Chopra et al. (2005) for face verification. These works and their descendants (e.g., Yih et al., 2011;Hu et al., 2014) refer to the architecture as siamese networks since two similar inputs are encoded separately by two copies of the same network (all parameters are shared). Wsabie (Weston et al., 2010) and StarSpace (Wu et al., 2018) subsequently employ bi-encoder to learn embeddings for different data types. Using deep pretrained transformers as encoders, Humeau et al. (2019) compare three different architectures, biencoder, poly-encoder, and cross-encoder. The biencoder architecture afterwards has been use in various tasks, e.g., information retrieval (Huang et al., 2013;Gillick et al., 2018), open-domain question answering (Karpukhin et al., 2020), and entity linking (Gillick et al., 2019;Zhang et al., 2021a). To our best knowledge, there is no previous work learning bi-encoder for NER.

Method
In this section, we present the design of BINDER, a novel architecture for NER tasks. As our model is built upon a bi-encoder framework, we first provide the necessary background for encoding both entity types and text using the Transformer-based (Vaswani et al., 2017) bi-encoder. Then, we discuss our ways of deriving entity type and individual mention span representations using embeddings output from the bi-encoder. Based on that, we introduce two types of contrastive learning objectives for NER using the token and span-level similarity respectively.

Bi-Encoder for NER
The overall architecture of BINDER is shown in Figure 1. Our model is built upon a bi-encoder  Figure 1: The overall architecture of BINDER. The entity type and text encoder are isomorphic and fully decoupled Transformer models. In the vector space, the anchor point ( ) represents the special token [CLS] from the entity type encoder. Through contrastive learning, we maximize the similarity between the anchor and the positive token ( Jim), and minimize the similarity between the anchor and negative tokens. The dotted gray circle (delimited by the similarity between the anchor and [CLS] from the text encoder) represents a threshold that separates entity tokens from non-entity tokens. To reduce clutter, we do not draw data points that represent possible spans from the input text.
architecture which has been mostly explored for dense retrieval (Karpukhin et al., 2020). Following the recent work, our bi-encoder also consists of two isomorphic and fully decoupled Transformer models (Vaswani et al., 2017), i.e. an entity type encoder and a text encoder. For NER tasks, we consider two types of inputs, entity type descriptions and text to detect named entities. At the high level, the entity type encoder produces type representations for each entity of interests (e.g. person in Figure 1) and the text encoder outputs representations for each input token in the given text where named entities are potentially mentioned (e.g. Jim in Figure 1). Then, we enumerate all span candidates based on corresponding token representations and match them with each entity type in the vector space. As shown in Figure 1, we maximize the similarity between the positive span and its corresponding entity type, and minimize the similarity of negative spans. We first formally discuss encoding both inputs using a pretrained Transformer model, BERT (Devlin et al., 2019). 2 Specifically, we use x 1 , . . . , x N to denote an input sequence of length N . When using BERT, there is a prepended to-ken [CLS] and an appended token Then the output is a sequence of hidden states h [CLS] , h 1 , . . . , h N , h [SEP] ∈ R d from the last BERT layer for each input token, where d is the hidden dimension. Note that as [CLS]is always in the beginning, h 0 and h [CLS] are interchangeable here. Based on this, we then discuss the way of computing entity type and text token embeddings, which are the basic building blocks for deriving our NER constrative learning objectives later.
Entity Type Embeddings The goal of entity type encoder is to produce entity type embeddings that serve as anchors in the vector space for contrastive learning. In this work, we focus on a predefined set of entity types E = {E 1 , . . . , E K }, where each entity type has one or multiple natural language descriptions. The natural language description can be formal textual definitions based on the dataset annotation guideline or Wikipedia, and prototypical instances where a target type of named entities are mentioned. For simplicity, the discussion proceeds with one description per type and we use E k to denote a sequence of tokens for the k-th entity type description. For a given entity type E k , we use BERT as the entity type encoder (BERT E ) and add an additional linear projection to compute corresponding entity type embeddings in the following way: where Linear is a learnable linear projection layer and e k ∈ R d is the final vector representation for entity type E k .
Text Token Embeddings Instead of using [CLS]embeddings as done in the recent biencoder work for entity retrieval , we consider using text token embeddings as the basic unit for computing similarity with entity span embeddings. As there are multiple potential named entities not known as a prior in the input text, naively using special markers  incurs huge computation overhead for NER tasks. Similar to the entity type embeddings, we again use BERT as the text encoder (BERT T ) and a linear layer for computing token representations 3 where Linear T is a learnable linear layer and p n ∈ R d is the vector representation for the nth token in the input sentence. Again, as [CLS]is always at the start, we use p 0 to denote p [CLS] for simplicity.

NER Contrastive Learning
Based on the entity type embeddings and text token embeddings discussed above, we then introduce two different contrastive learning objectives for NER in this part. Here, we assume a span (i, j) is a contiguous sequence of tokens in the input text with a start token in position i and an end token in position j. Throughout this work, we use the similarity function, sim(·, ·) = cos(·,·) τ , where τ is a scalar parameter.
As shown in Figure 1, the overall goal of NER constrastive learning is to push the entity mention span representations close to their corresponding entity type embeddings (positive) and far away from irrelevant types (negative) in vector space, e.g. Person closer to Jim but away from Acme. To achieve that, we propose a multi-objective formulation consisting of two objectives based on span and token embedding spaces respectively. 3 Here, we leave out special tokens for simplicity.
Span-based Objective Here, we consider the following way for deriving the vector representation for span (i, j) is a row from the span width embedding matrix D ∈ R L×m , and L is the maximum span width considered. Based on this, the span-based infoNCE (Oord et al., 2018) can be defined as , (6) where the span (i, j) belongs to entity type E k , S − k is the set of negative spans that are all possible spans from the input text, excluding entity spans from E k , and e k is the entity type embedding.
Position-based Objective One limitation of the span-based objective is that it penalizes negative spans in the same way, whether they are partially correct spans or spans that have no overlap with the gold entity span. Intuitively, it is more desirable to have partially correct spans, e.g., spans that have the same start or end token with the gold span. Therefore, we propose additional positionbased contrastive learning objectives. Specifically, we compute two additional entity type embeddings for E k by using additional linear layers where e B k , e Q k are the type embeddings for the start and end positions respectively, h E k [CLS] is from the entity type encoder (Equation 1). Accordingly, we can use two addtional linear layers to compute the corresponding token embeddings for the start and end tokens respectively Using e k B , e k Q as anchors, we then define two position-based objectives k are two sets of positions in the input text that do not belong to the start/end of any span of entity type k. Compared with Equation 6, the main difference of position-based objectives comes from the corresponding negative sets where start and end positions are independent of each other. In other words, the position-based objectives can potentially help the model to make better start and end position predictions.
Thresholding for Non-Entity Cases Although the contrastive learning objectives defined above can effectively push the positive spans close to their corresponding entity type in vector space, it might be problematic for our model at test time to decide how close a span should be before it can be considered as positive. In other words, the model is not able to separate entity spans from non-entity spans properly. To address this issue, we use the similarity between the special token [CLS] and the entity type as a dynamic threshold (as shown in Figure 1). Intuitively, the representation of [CLS] reads the entire input text and summarizes the contextual information, which could make it a good choice to compute the threshold to separate entity spans from non-entity spans. We compare it with several other thresholding choices in §4.5.
To learn thresholding, we extend the original contrastive learning objectives with extra adaptive learning objectives for non-entity cases. Specifically, for the start loss (Equation 11), the augmented start loss + start is .
(13) An augmented end loss + end can be defined in a similar fashion. For the span loss (Equation 6), we use the span embedding s 0,0 for deriving the augmented span loss + .
(14) Note that we use a single scalar hyperparameter β for balancing the adaptive thresholding learning and original contrastive learning for all three cases.
Training Finally, we consider a multi-task contrastive formulation by combing three augmented contrastive learning discussed above, leading to our overall training objective where α, γ, λ are all scalar hyperparameters.
Inference Strategy During inference, we enumerate all possible spans that are less than the length of L and compute three similarity scores based on the start/end/span cases for each entity type. We consider two prediction strategies, joint position-span and span-only predictions. In the joint position-span case, for entity type E k , we prune out spans (i, j) that have either start or end similar scores lower than the learned null . Then, only those spans with span similarity scores higher than the span null threshold are predicted as positive ones i.e. sim(s i,j , e k ) > sim(s 0,0 , e k ). For the spanonly strategy, we just rely on the span similarity score and keep all qualified spans as final predictions. The full inference algorithm is summarized in Algorithm 1. As we can see, the difference between joint position-span and span-only strategies is whether line 11 is used. As shown later in our experiments ( §4.5), we find the span-only inference is more effective, because the joint inference is more likely to be affected by annotation artifacts. Also, for flat NER datasets, we further carry out a post-processing step to remove overlapping predictions (line 22 in Algorithm 1). Here, the post-processing is carried out in a greedy fashion where higher scored span predictions with earlier start and end positions are preferred.

Experiments
We evaluate our method in both supervised and distantly supervised settings.

Evaluation Metrics
We follow the standard evaluation protocol and use micro F1 as the evaluation metric: a predicted entity span is considered correct if its span boundaries and the predicted entity type are both correct.

Implementation Details
We implement our models based on the Hugging-Face Transformers library (Wolf et al., 2020). The base encoders are initialized using PubMedBERTbase-uncased (Gu et al., 2021) or BioBERT  for biomedical NER datasets, and  BERT-base-uncased (Devlin et al., 2019) for other NER datasets. The linear layer output size is 128; the width embedding size is 128; the initial temperatures are 0.07. We train our models with the AdamW optimizer (Loshchilov and Hutter, 2017) of a linear scheduler and dropout of 0.1. The entity start/end/span contrastive loss (Equation 15) weights are set to α = 0.2, γ = 0.2, λ = 0.6, and the same loss weights are chosen for thresholding contrastive learning. The contrastive losses for thresholding and entity are weighted equally in the final loss. For all experiments, we ignore sentence boundaries, and tokenize and split text into sequences with the maximum token length of 128 and a stride of 16. We train our models for 20 epochs with a learning rate of 3e-5 and a batch size of 8 sequences. The maximum token length for entity spans is set to 30. We use early stop with a patience of 10 in the distantly supervised setting. Validation is done at every 50 steps of training, and we adopt the models that have the best performance on the development set. We report the median score of multiple runs.

Supervised NER Datasets
In the supervised setting, we evaluate our method on both nested and flat NER datasets. For nested NER, we consider ACE2004 4 , ACE2005 5 , and GENIA (Kim et al., 2003). ACE2004 and ACE2005 are collected from general domains (e.g., news and web). We follow Luan et al. (2018) to split ACE2004 into 5 folds, and ACE2005 into train, development and test sets. GENIA is collected from the biomedical domain. We use its v3.0.2 corpus 6 and follow Finkel and Manning (2009) and Lu and Roth (2015) to split it into 80%/10%/10% train/dev/test splits. For flat NER, we consider five biomedical NER datasets from the BLURB benchmark 7 : BC5-chem/disease (Li et al., 2016), NCBI (Dogan et al., 2014), BC2GM (Smith et al., 2008), and JNLPBA (Collier and Kim, 2004). Preprocessing and splits follow Gu et al. (2021). Table 2 shows the statistics of each dataset.  and GENIA. We report precision, recall, and F1. As is shown, our method achieves very strong performance on all three datasets. On ACE2004 and ACE2005, it outperforms all previous methods with 88.7% and 89.5% F1. Particularly, in comparison with the previous best method , our method significantly improves F1 by the absolute points of +1.4% and +2.4% respectively. On GENIA, our method is on par with the previous state of the art . Note that the previous methods are built on top of different encoders, e.g., LSTM, BERT-large, BARTlarge, and T5-base. Our method using a BERTbase encoder even outperforms the previous methods that use a larger encoder of BERT-large or BART-large. Overall, compared to the previous methods, our method has substantial gains on recall while maintaining a high precision. We conduct more detailed analysis in §4.5. Table 3 compares our method with all previous submissions on the BLURB benchmark. We only report F1 due to unavailability of precision and recall of these submissions. The major difference among these submissions are encoders. A direct comparison can be made between our method and Gu et al. (2021) which formulates NER as a sequential labeling task and fine-tunes a standard LSTM+CRF classifier on top of the pretrained transformer encoder. While both use PubMed-BERT as the encoder, our method significantly outperforms Gu et al. (2021) across the board. Compared to the recent submissions, Kanakarajan et al. (2021); Yasunaga et al. (2022), our method also show substantial gains on BC5-chem (+1.2%), BC5-disease (+1.9%), and NCBI (1.5%).

Distantly Supervised NER
Dataset In the distantly supervised setting, we consider BC5CDR (Li et al., 2016) from the biomedical domain. It consists of 1,500 articles annotated with 15,935 chemical and 12,852 disease mentions. We use the standard train, development, and test splits. In train and development, we discard all human annotations and only keep the unlabeled articles. Their distant labels are generated using exact string matching against a dictionary released by Shang et al. (2018). 8 On the training set, the distant labels have high precision (97.99% for chemicals, and 98.34% for diseases), but low recall (63.14% for chemicals, and 46.73%  Table 3: Test F1 scores on five flat NER datasets from the BLURB benchmark (aka.ms/blurb). Bold and underline indicate the best and the second best respectively. All encoders use their base version.
Results Table 4 compares test scores of our method and previous methods on BC5CDR. It presents a clear advantage of our method over all previous methods in the distantly supervised setting. The F1 score is improved by +1.5% over the previous best method (Zhou et al., 2022), which adapts positive and unlabeled (PU) learning to obtain a high recall yet at the loss of precision. In contrast, our method maintains a reasonable recall (comparable to Shang et al., 2018;Peng et al., 2019) and substantially improves the precision. For reference,  (Shang et al., 2018) 82.6 77.5 80.0 BNPU (Peng et al., 2019) 48.1 77.1 59.2 BERT-ES (Liang et al., 2020) 80.4 67.9 73.7 Conf-MPU (Zhou et al., 2022)

Analysis
Ablation Study We compare several variants of our model and report their test scores on ACE2005 in Table 5. We observe performance degradation in all these variants. Shared linear layers uses the same linear layer for span and token embeddings, and the same linear layer for entity type embeddings, in the hope of projecting them into the same vector space and sharing their semantics. It leads to a slightly better precision but a drop of recall. Similar changes are observed in Joint positionspan inference, which adopts a more strict strategy to prune out spans -only keep spans whose start, end, and span scores are all above the thresholds. In contrast, No position-based objectives only optimizes the span-based objective, which penalizes partially corrected spans in the same way as other spans. It results in a marginal improvement of recall but a significant loss of precision.  Thresholding Strategies Our method uses dynamic similarity thresholds to distinguish entity spans from non-entity spans. We compare the impact of dynamic thresholds in our method with two straightforward variants: (1) Learned global thresholds replaces dynamic thresholds with global thresholds, one for each entity type. Specifically, we consider the global similarity thresholds as scalar parameters (initialized as 0). During training, we replace the similarity function outputs sim(p [CLS] , e k ) in Equation 13 and sim(s 0,0 , e k ) in Equation 14 with the correspond-     Table 7, the majority of entity spans and non-entity spans are separable regardless of the thresholding strategy. This is true even when no thresholds are used during training and instead we tune global thresholds on dev. Dynamic thresholds (shown in green and best viewed in an extreme zoom-in) are advantageous among a small set of entity spans which can not be separated from non-entity spans using a global threshold. We also observe that the learned global thresholds tend to yield a smaller gap between en- tity spans and non-entity spans, and a denser similarity distribution of non-entity spans.
Performance Breakdown Table 6 shows the test F1 scores on each entity type of ACE2005 and GENIA. We report four types of F1 scores: S-F1 span is the strict F1 based on the exact match of entity spans; S-F1 start is the strict F1 based on the exact match of entity start words; S-F1 end is the strict F1 based on the exact match of entity end words; L-F1 span is the loose F1 allowing the partial match of entity spans. We notice that S-F1 end is often substantially better than S-F1 span and S-F1 start . To explain this difference, we go through these partially corrected predictions and summarize the common errors in Table 8. The most common one is the inclusion or exclusion of modifiers. This could be due to annotation disagreement: in ACE2005, sometimes generic spans are preferred (e.g., "tomcats" vs. "f-14 tomcats"), while in some cases specific spans are preferred (e.g., "cruise ship" vs. "ship"). This issue is more common in the biomedical dataset GENIA, e.g., "human GR" is considered as wrong while "human lymphocytes" are correct, which explains the higher scores of S-F1 end than S-F1 start . Missing genitives for person names are another common errors in ACE2005. We also discover some annotation errors, where names of a single person, protein, or cell line are sometimes broken into two less meaningful spans, e.g., "Dr. Germ" is annotated as two person mentions "Dr" and "Germ", and "EBVtransformed human B cell line SKW6.4" is annotated as two separate cell lines "EBV-transformed human B cell line" and "SKW6.4".
Error Type Ent. Type Predicted ↔ Gold Modifier Error

Conclusions
We present a bi-encoder framework for NER, which separately maps text and entity types into the same vector space. By formulating NER as a metric learning problem, we propose to use a novel contrastive loss to simultaneously learn entity span identification and classification. Experiments in both supervised and distantly supervised setting demonstrate the effectiveness and robustness of our method. We conduct extensive analysis to explain the success of our method and reveal the room for further improvement. Future directions include applying our method in self-supervised or zero-shot settings.