<!DOCTYPE html><html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>[2311.08526] GLiNER: Generalist Model for Named Entity Recognition using Bidirectional Transformer</title><meta property="og:description" content="Named Entity Recognition (NER) is essential in various Natural Language Processing (NLP) applications. Traditional NER models are effective but limited to a set of predefined entity types. In contrast, Large Language M…">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="GLiNER: Generalist Model for Named Entity Recognition using Bidirectional Transformer">
<meta name="twitter:image:src" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta name="twitter:image:alt" content="ar5iv logo">
<meta property="og:title" content="GLiNER: Generalist Model for Named Entity Recognition using Bidirectional Transformer">
<meta property="og:site_name" content="ar5iv">
<meta property="og:image" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta property="og:type" content="article">
<meta property="og:url" content="https://ar5iv.labs.arxiv.org/html/2311.08526">

<!--Generated on Tue Feb 27 19:11:26 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.-->
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

<script>
  function detectColorScheme(){
    var theme="light";
    var current_theme = localStorage.getItem("ar5iv_theme");
    if(current_theme){
      if(current_theme == "dark"){
        theme = "dark";
      } }
    else if(!window.matchMedia) { return false; }
    else if(window.matchMedia("(prefers-color-scheme: dark)").matches) {
      theme = "dark"; }
    if (theme=="dark") {
      document.documentElement.setAttribute("data-theme", "dark");
    } else {
      document.documentElement.setAttribute("data-theme", "light"); } }

  detectColorScheme();

  function toggleColorScheme(){
    var current_theme = localStorage.getItem("ar5iv_theme");
    if (current_theme) {
      if (current_theme == "light") {
        localStorage.setItem("ar5iv_theme", "dark"); }
      else {
        localStorage.setItem("ar5iv_theme", "light"); } }
    else {
        localStorage.setItem("ar5iv_theme", "dark"); }
    detectColorScheme(); }
</script>
<link media="all" rel="stylesheet" href="/assets/ar5iv-fonts.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv-site.0.2.2.css">
</head>
<body>
<div class="ltx_page_main">
<div class="ltx_page_content">
<article class="ltx_document ltx_authors_1line">
<h1 class="ltx_title ltx_title_document">GLiNER: Generalist Model for Named Entity Recognition using Bidirectional Transformer</h1>
<div class="ltx_authors">
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Urchade Zaratiana<sup id="id7.7.id1" class="ltx_sup"><span id="id7.7.id1.1" class="ltx_text ltx_font_italic">1,2</span></sup>, Nadi Tomeh<sup id="id8.8.id2" class="ltx_sup">2</sup>, Pierre Holat<sup id="id9.9.id3" class="ltx_sup"><span id="id9.9.id3.1" class="ltx_text ltx_font_italic">1,2</span></sup>, Thierry Charnois<sup id="id10.10.id4" class="ltx_sup">2</sup> 
<br class="ltx_break"><sup id="id11.11.id5" class="ltx_sup">1</sup> FI Group,
<sup id="id12.12.id6" class="ltx_sup">2</sup> LIPN, CNRS UMR 7030, France 
<br class="ltx_break"><span id="id13.13.id7" class="ltx_text ltx_font_typewriter">zaratiana@lipn.fr</span> 
<br class="ltx_break"><span id="id14.14.id8" class="ltx_text ltx_font_typewriter">https://github.com/urchade/GLiNER</span>
</span></span>
</div>

<div class="ltx_abstract">
<h6 class="ltx_title ltx_title_abstract">Abstract</h6>
<p id="id15.id1" class="ltx_p">Named Entity Recognition (NER) is essential in various Natural Language Processing (NLP) applications. Traditional NER models are effective but limited to a set of predefined entity types. In contrast, Large Language Models (LLMs) can extract arbitrary entities through natural language instructions, offering greater flexibility. However, their size and cost, particularly for those accessed via APIs like ChatGPT, make them impractical in resource-limited scenarios. In this paper, we introduce a compact NER model trained to identify any type of entity. Leveraging a bidirectional transformer encoder, our model, GLiNER, facilitates parallel entity extraction, an advantage over the slow sequential token generation of LLMs. Through comprehensive testing, GLiNER demonstrate strong performance, outperforming both ChatGPT and fine-tuned LLMs in zero-shot evaluations on various NER benchmarks.</p>
</div>
<section id="S1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">1 </span>Introduction</h2>

<div id="S1.p1" class="ltx_para">
<p id="S1.p1.1" class="ltx_p">Named Entity Recognition plays a crucial role in various real-world applications, such as constructing knowledge graphs. Traditional NER models are limited to a predefined set of entity types. Expanding the number of entity types can be beneficial for many applications but may involve labeling additional datasets. The emergence of Large Language Models, like GPT-3 <cite class="ltx_cite ltx_citemacro_citep">(Brown et al., <a href="#bib.bib4" title="" class="ltx_ref">2020</a>)</cite>, has introduced a new era for open-type NER by enabling the identification of any types of entity types only by natural language instruction. This shift signifies a significant departure from the inflexibility observed in traditional models. However, powerful LLMs typically consist of billions of parameters and thus require substantial computing resources. Although it is possible to access some LLMs via APIs <cite class="ltx_cite ltx_citemacro_citep">(OpenAI, <a href="#bib.bib32" title="" class="ltx_ref">2023</a>)</cite>, using them at scale can incur high costs.</p>
</div>
<figure id="S1.F1" class="ltx_figure"><img src="/html/2311.08526/assets/x1.png" id="S1.F1.g1" class="ltx_graphics ltx_centering ltx_img_square" width="461" height="510" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 1: </span><span id="S1.F1.2.1" class="ltx_text ltx_font_bold">BiLM for Open NER.</span> Previous models, such as UniNER <cite class="ltx_cite ltx_citemacro_citep">(Zhou et al., <a href="#bib.bib50" title="" class="ltx_ref">2023</a>)</cite> (Fig. a), approach the task of open type NER by prompting LLMs with natural language instructions (using a multi-turn dialogue style). Our proposed GLiNER utilizes small bidirectional LMs and treats the NER task as matching entity types with textual spans in a latent space.</figcaption>
</figure>
<figure id="S1.F2" class="ltx_figure"><img src="/html/2311.08526/assets/x2.png" id="S1.F2.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="259" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 2: </span><span id="S1.F2.3.1" class="ltx_text ltx_font_bold">Model architecture</span>. GLiNER employs a BiLM and takes as input entity type prompts and a sentence/text. Each entity is separated by a learned token <span id="S1.F2.4.2" class="ltx_text ltx_font_typewriter">[ENT]</span>. The BiLM outputs representations for each token. Entity embeddings are passed into a FeedForward Network, while input word representations are passed into a span representation layer to compute embeddings for each span. Finally, we compute a matching score between entity representations and span representations (using dot product and sigmoid activation). For instance, in the figure, the span representation of (0, 1), corresponding to "Alain Farley," has a high matching score with the entity embeddings of "Person".</figcaption>
</figure>
<div id="S1.p2" class="ltx_para">
<p id="S1.p2.1" class="ltx_p">Recently, researchers have explored the fine-tuning of open source language models such as LLaMa <cite class="ltx_cite ltx_citemacro_citep">(Touvron et al., <a href="#bib.bib39" title="" class="ltx_ref">2023</a>)</cite> for named entity recognition tasks. <cite class="ltx_cite ltx_citemacro_citet">Wang et al. (<a href="#bib.bib40" title="" class="ltx_ref">2023</a>)</cite>, for example, introduced InstructUIE, a fine-tuned FlanT5-11B <cite class="ltx_cite ltx_citemacro_citep">(Raffel et al., <a href="#bib.bib34" title="" class="ltx_ref">2019</a>; Chung et al., <a href="#bib.bib6" title="" class="ltx_ref">2022</a>)</cite> model on existing information extraction datasets, achieving excellent performance in zero-shot settings. Additionally, GoLLIE <cite class="ltx_cite ltx_citemacro_citep">(Sainz et al., <a href="#bib.bib37" title="" class="ltx_ref">2023</a>)</cite> was introduced as an extension of InstructUIE work by by finetuning a CodeLLaMa <cite class="ltx_cite ltx_citemacro_citep">(Rozière et al., <a href="#bib.bib36" title="" class="ltx_ref">2023</a>)</cite> using detailed annotation guidelines, resulting in significant performance improvements. Another recent proposal by <cite class="ltx_cite ltx_citemacro_citet">Zhou et al. (<a href="#bib.bib50" title="" class="ltx_ref">2023</a>)</cite>, called UniversalNER, involves the fine-tuning of LLMs using diverse data from various domains annotated with ChatGPT instead of relying on standard NER datasets. Their approach enables the replication and even surpassing of the original capability of ChatGPT when evaluated in zero-shot on several NER datasets. While these works have achieved remarkable results, they present certain limitations we seek to address: They use autoregressive language models, which can be slow due to token-by-token generation; Moreover, they employ large models with several billion parameters, limiting their deployment in compute-limited scenarios. Furthurmore, as NER is treated as a text generation problem, the generation of entities is done in several decoding steps, and there is no way to perform the prediction of multiple entity types in parallel.</p>
</div>
<div id="S1.p3" class="ltx_para">
<p id="S1.p3.1" class="ltx_p">In our work, we propose a model that addresses the above-mentioned problems. Instead of relying on large autoregressive models, we utilize smaller-scale Bidirectional Language Models (BiLM), such as BERT <cite class="ltx_cite ltx_citemacro_citep">(Devlin et al., <a href="#bib.bib10" title="" class="ltx_ref">2019</a>)</cite> or deBERTa <cite class="ltx_cite ltx_citemacro_citep">(He et al., <a href="#bib.bib13" title="" class="ltx_ref">2021</a>)</cite>. The core concept of our model involves treating the task of Open NER as matching entity type embeddings to textual span representations in latent space, rather than as a generation task. This approach naturally solves the scalability issues of autoregressive models and allows for bidirectional context processing, which enables richer representations. When trained on the dataset released by <cite class="ltx_cite ltx_citemacro_citep">(Zhou et al., <a href="#bib.bib50" title="" class="ltx_ref">2023</a>)</cite>, which comprises texts from numerous domains and thousands of entity types, our model demonstrates impressive zero-shot performance. More specifilcally, it outperforms both ChatGPT and fine-tuned LLMs on zero-shot NER datasets (Table <a href="#S2.T1" title="Table 1 ‣ Entity Type and Span Matching ‣ 2.1 Architecture ‣ 2 Method ‣ GLiNER: Generalist Model for Named Entity Recognition using Bidirectional Transformer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>). Our model’s robustness is further evident in its ability to handle languages not even encountered during training. Specifically, our model surpasses ChatGPT in 8 of 10 languages (Table <a href="#S4.T3" title="Table 3 ‣ 20 NER Benchmark ‣ 4.1 Zero-shot on English datasets ‣ 4 Results ‣ GLiNER: Generalist Model for Named Entity Recognition using Bidirectional Transformer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>) that were not included in its training data.</p>
</div>
</section>
<section id="S2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">2 </span>Method</h2>

<div id="S2.p1" class="ltx_para">
<p id="S2.p1.1" class="ltx_p">This section presents our model, GLiNER, which is train to extract any types of entity using a Bidirectional Language Models. Our model has three main components: i) a pre-trained textual encoder (a BiLM such as BERT), ii) a span representation module which computes span embeddings from token embeddings, iii) an entity representation module which computes entity embeddings that the model seeks to extract. The goal is to have entity and span embeddings in the same latent space to assess their compatibility (degree of matching). The overall architecture of our model is depicted in Figure <a href="#S1.F2" title="Figure 2 ‣ 1 Introduction ‣ GLiNER: Generalist Model for Named Entity Recognition using Bidirectional Transformer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>.</p>
</div>
<section id="S2.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.1 </span>Architecture</h3>

<section id="S2.SS1.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Input format</h4>

<div id="S2.SS1.SSS0.Px1.p1" class="ltx_para">
<p id="S2.SS1.SSS0.Px1.p1.1" class="ltx_p">The input to our model comprises a unified sequence combining entity types (expressed in natural language) and the input text from which entities are to be extracted. The input format is as follows:</p>
</div>
<figure id="S2.SS1.SSS0.Px1.1" class="ltx_figure"><img src="/html/2311.08526/assets/x3.png" id="S2.SS1.SSS0.Px1.1.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="35" alt="[Uncaptioned image]">
</figure>
<div id="S2.SS1.SSS0.Px1.p2" class="ltx_para">
<p id="S2.SS1.SSS0.Px1.p2.1" class="ltx_p"><span id="S2.SS1.SSS0.Px1.p2.1.1" class="ltx_text ltx_font_typewriter">[ENT]</span> token represents a special token placed before each entity type and the <span id="S2.SS1.SSS0.Px1.p2.1.2" class="ltx_text ltx_font_typewriter">[SEP]</span> token functions as a delimiter, separating the sequence of entity types from the input text. They are initialized randomly at the start of training.</p>
</div>
</section>
<section id="S2.SS1.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Token representation</h4>

<div id="S2.SS1.SSS0.Px2.p1" class="ltx_para">
<p id="S2.SS1.SSS0.Px2.p1.2" class="ltx_p">The token encoder processes the unified input to compute interactions between all tokens (both entity types and input text), producing contextualized representations. Let <math id="S2.SS1.SSS0.Px2.p1.1.m1.1" class="ltx_Math" alttext="{\bm{p}}=\{{\bm{p}}_{i}\}_{0}^{M-1}\in\mathbb{R}^{M\times D}" display="inline"><semantics id="S2.SS1.SSS0.Px2.p1.1.m1.1a"><mrow id="S2.SS1.SSS0.Px2.p1.1.m1.1.1" xref="S2.SS1.SSS0.Px2.p1.1.m1.1.1.cmml"><mi id="S2.SS1.SSS0.Px2.p1.1.m1.1.1.3" xref="S2.SS1.SSS0.Px2.p1.1.m1.1.1.3.cmml">𝒑</mi><mo id="S2.SS1.SSS0.Px2.p1.1.m1.1.1.4" xref="S2.SS1.SSS0.Px2.p1.1.m1.1.1.4.cmml">=</mo><msubsup id="S2.SS1.SSS0.Px2.p1.1.m1.1.1.1" xref="S2.SS1.SSS0.Px2.p1.1.m1.1.1.1.cmml"><mrow id="S2.SS1.SSS0.Px2.p1.1.m1.1.1.1.1.1.1" xref="S2.SS1.SSS0.Px2.p1.1.m1.1.1.1.1.1.2.cmml"><mo stretchy="false" id="S2.SS1.SSS0.Px2.p1.1.m1.1.1.1.1.1.1.2" xref="S2.SS1.SSS0.Px2.p1.1.m1.1.1.1.1.1.2.cmml">{</mo><msub id="S2.SS1.SSS0.Px2.p1.1.m1.1.1.1.1.1.1.1" xref="S2.SS1.SSS0.Px2.p1.1.m1.1.1.1.1.1.1.1.cmml"><mi id="S2.SS1.SSS0.Px2.p1.1.m1.1.1.1.1.1.1.1.2" xref="S2.SS1.SSS0.Px2.p1.1.m1.1.1.1.1.1.1.1.2.cmml">𝒑</mi><mi id="S2.SS1.SSS0.Px2.p1.1.m1.1.1.1.1.1.1.1.3" xref="S2.SS1.SSS0.Px2.p1.1.m1.1.1.1.1.1.1.1.3.cmml">i</mi></msub><mo stretchy="false" id="S2.SS1.SSS0.Px2.p1.1.m1.1.1.1.1.1.1.3" xref="S2.SS1.SSS0.Px2.p1.1.m1.1.1.1.1.1.2.cmml">}</mo></mrow><mn id="S2.SS1.SSS0.Px2.p1.1.m1.1.1.1.1.3" xref="S2.SS1.SSS0.Px2.p1.1.m1.1.1.1.1.3.cmml">0</mn><mrow id="S2.SS1.SSS0.Px2.p1.1.m1.1.1.1.3" xref="S2.SS1.SSS0.Px2.p1.1.m1.1.1.1.3.cmml"><mi id="S2.SS1.SSS0.Px2.p1.1.m1.1.1.1.3.2" xref="S2.SS1.SSS0.Px2.p1.1.m1.1.1.1.3.2.cmml">M</mi><mo id="S2.SS1.SSS0.Px2.p1.1.m1.1.1.1.3.1" xref="S2.SS1.SSS0.Px2.p1.1.m1.1.1.1.3.1.cmml">−</mo><mn id="S2.SS1.SSS0.Px2.p1.1.m1.1.1.1.3.3" xref="S2.SS1.SSS0.Px2.p1.1.m1.1.1.1.3.3.cmml">1</mn></mrow></msubsup><mo id="S2.SS1.SSS0.Px2.p1.1.m1.1.1.5" xref="S2.SS1.SSS0.Px2.p1.1.m1.1.1.5.cmml">∈</mo><msup id="S2.SS1.SSS0.Px2.p1.1.m1.1.1.6" xref="S2.SS1.SSS0.Px2.p1.1.m1.1.1.6.cmml"><mi id="S2.SS1.SSS0.Px2.p1.1.m1.1.1.6.2" xref="S2.SS1.SSS0.Px2.p1.1.m1.1.1.6.2.cmml">ℝ</mi><mrow id="S2.SS1.SSS0.Px2.p1.1.m1.1.1.6.3" xref="S2.SS1.SSS0.Px2.p1.1.m1.1.1.6.3.cmml"><mi id="S2.SS1.SSS0.Px2.p1.1.m1.1.1.6.3.2" xref="S2.SS1.SSS0.Px2.p1.1.m1.1.1.6.3.2.cmml">M</mi><mo lspace="0.222em" rspace="0.222em" id="S2.SS1.SSS0.Px2.p1.1.m1.1.1.6.3.1" xref="S2.SS1.SSS0.Px2.p1.1.m1.1.1.6.3.1.cmml">×</mo><mi id="S2.SS1.SSS0.Px2.p1.1.m1.1.1.6.3.3" xref="S2.SS1.SSS0.Px2.p1.1.m1.1.1.6.3.3.cmml">D</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S2.SS1.SSS0.Px2.p1.1.m1.1b"><apply id="S2.SS1.SSS0.Px2.p1.1.m1.1.1.cmml" xref="S2.SS1.SSS0.Px2.p1.1.m1.1.1"><and id="S2.SS1.SSS0.Px2.p1.1.m1.1.1a.cmml" xref="S2.SS1.SSS0.Px2.p1.1.m1.1.1"></and><apply id="S2.SS1.SSS0.Px2.p1.1.m1.1.1b.cmml" xref="S2.SS1.SSS0.Px2.p1.1.m1.1.1"><eq id="S2.SS1.SSS0.Px2.p1.1.m1.1.1.4.cmml" xref="S2.SS1.SSS0.Px2.p1.1.m1.1.1.4"></eq><ci id="S2.SS1.SSS0.Px2.p1.1.m1.1.1.3.cmml" xref="S2.SS1.SSS0.Px2.p1.1.m1.1.1.3">𝒑</ci><apply id="S2.SS1.SSS0.Px2.p1.1.m1.1.1.1.cmml" xref="S2.SS1.SSS0.Px2.p1.1.m1.1.1.1"><csymbol cd="ambiguous" id="S2.SS1.SSS0.Px2.p1.1.m1.1.1.1.2.cmml" xref="S2.SS1.SSS0.Px2.p1.1.m1.1.1.1">superscript</csymbol><apply id="S2.SS1.SSS0.Px2.p1.1.m1.1.1.1.1.cmml" xref="S2.SS1.SSS0.Px2.p1.1.m1.1.1.1"><csymbol cd="ambiguous" id="S2.SS1.SSS0.Px2.p1.1.m1.1.1.1.1.2.cmml" xref="S2.SS1.SSS0.Px2.p1.1.m1.1.1.1">subscript</csymbol><set id="S2.SS1.SSS0.Px2.p1.1.m1.1.1.1.1.1.2.cmml" xref="S2.SS1.SSS0.Px2.p1.1.m1.1.1.1.1.1.1"><apply id="S2.SS1.SSS0.Px2.p1.1.m1.1.1.1.1.1.1.1.cmml" xref="S2.SS1.SSS0.Px2.p1.1.m1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS1.SSS0.Px2.p1.1.m1.1.1.1.1.1.1.1.1.cmml" xref="S2.SS1.SSS0.Px2.p1.1.m1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S2.SS1.SSS0.Px2.p1.1.m1.1.1.1.1.1.1.1.2.cmml" xref="S2.SS1.SSS0.Px2.p1.1.m1.1.1.1.1.1.1.1.2">𝒑</ci><ci id="S2.SS1.SSS0.Px2.p1.1.m1.1.1.1.1.1.1.1.3.cmml" xref="S2.SS1.SSS0.Px2.p1.1.m1.1.1.1.1.1.1.1.3">𝑖</ci></apply></set><cn type="integer" id="S2.SS1.SSS0.Px2.p1.1.m1.1.1.1.1.3.cmml" xref="S2.SS1.SSS0.Px2.p1.1.m1.1.1.1.1.3">0</cn></apply><apply id="S2.SS1.SSS0.Px2.p1.1.m1.1.1.1.3.cmml" xref="S2.SS1.SSS0.Px2.p1.1.m1.1.1.1.3"><minus id="S2.SS1.SSS0.Px2.p1.1.m1.1.1.1.3.1.cmml" xref="S2.SS1.SSS0.Px2.p1.1.m1.1.1.1.3.1"></minus><ci id="S2.SS1.SSS0.Px2.p1.1.m1.1.1.1.3.2.cmml" xref="S2.SS1.SSS0.Px2.p1.1.m1.1.1.1.3.2">𝑀</ci><cn type="integer" id="S2.SS1.SSS0.Px2.p1.1.m1.1.1.1.3.3.cmml" xref="S2.SS1.SSS0.Px2.p1.1.m1.1.1.1.3.3">1</cn></apply></apply></apply><apply id="S2.SS1.SSS0.Px2.p1.1.m1.1.1c.cmml" xref="S2.SS1.SSS0.Px2.p1.1.m1.1.1"><in id="S2.SS1.SSS0.Px2.p1.1.m1.1.1.5.cmml" xref="S2.SS1.SSS0.Px2.p1.1.m1.1.1.5"></in><share href="#S2.SS1.SSS0.Px2.p1.1.m1.1.1.1.cmml" id="S2.SS1.SSS0.Px2.p1.1.m1.1.1d.cmml" xref="S2.SS1.SSS0.Px2.p1.1.m1.1.1"></share><apply id="S2.SS1.SSS0.Px2.p1.1.m1.1.1.6.cmml" xref="S2.SS1.SSS0.Px2.p1.1.m1.1.1.6"><csymbol cd="ambiguous" id="S2.SS1.SSS0.Px2.p1.1.m1.1.1.6.1.cmml" xref="S2.SS1.SSS0.Px2.p1.1.m1.1.1.6">superscript</csymbol><ci id="S2.SS1.SSS0.Px2.p1.1.m1.1.1.6.2.cmml" xref="S2.SS1.SSS0.Px2.p1.1.m1.1.1.6.2">ℝ</ci><apply id="S2.SS1.SSS0.Px2.p1.1.m1.1.1.6.3.cmml" xref="S2.SS1.SSS0.Px2.p1.1.m1.1.1.6.3"><times id="S2.SS1.SSS0.Px2.p1.1.m1.1.1.6.3.1.cmml" xref="S2.SS1.SSS0.Px2.p1.1.m1.1.1.6.3.1"></times><ci id="S2.SS1.SSS0.Px2.p1.1.m1.1.1.6.3.2.cmml" xref="S2.SS1.SSS0.Px2.p1.1.m1.1.1.6.3.2">𝑀</ci><ci id="S2.SS1.SSS0.Px2.p1.1.m1.1.1.6.3.3.cmml" xref="S2.SS1.SSS0.Px2.p1.1.m1.1.1.6.3.3">𝐷</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.SSS0.Px2.p1.1.m1.1c">{\bm{p}}=\{{\bm{p}}_{i}\}_{0}^{M-1}\in\mathbb{R}^{M\times D}</annotation></semantics></math> represent the encoder’s output for each entity type, corresponding to all the <span id="S2.SS1.SSS0.Px2.p1.2.1" class="ltx_text ltx_font_typewriter">[ENT]</span> token representations. Similarly, <math id="S2.SS1.SSS0.Px2.p1.2.m2.1" class="ltx_Math" alttext="{\bm{h}}=\{{\bm{h}}_{i}\}_{0}^{N-1}\in\mathbb{R}^{N\times D}" display="inline"><semantics id="S2.SS1.SSS0.Px2.p1.2.m2.1a"><mrow id="S2.SS1.SSS0.Px2.p1.2.m2.1.1" xref="S2.SS1.SSS0.Px2.p1.2.m2.1.1.cmml"><mi id="S2.SS1.SSS0.Px2.p1.2.m2.1.1.3" xref="S2.SS1.SSS0.Px2.p1.2.m2.1.1.3.cmml">𝒉</mi><mo id="S2.SS1.SSS0.Px2.p1.2.m2.1.1.4" xref="S2.SS1.SSS0.Px2.p1.2.m2.1.1.4.cmml">=</mo><msubsup id="S2.SS1.SSS0.Px2.p1.2.m2.1.1.1" xref="S2.SS1.SSS0.Px2.p1.2.m2.1.1.1.cmml"><mrow id="S2.SS1.SSS0.Px2.p1.2.m2.1.1.1.1.1.1" xref="S2.SS1.SSS0.Px2.p1.2.m2.1.1.1.1.1.2.cmml"><mo stretchy="false" id="S2.SS1.SSS0.Px2.p1.2.m2.1.1.1.1.1.1.2" xref="S2.SS1.SSS0.Px2.p1.2.m2.1.1.1.1.1.2.cmml">{</mo><msub id="S2.SS1.SSS0.Px2.p1.2.m2.1.1.1.1.1.1.1" xref="S2.SS1.SSS0.Px2.p1.2.m2.1.1.1.1.1.1.1.cmml"><mi id="S2.SS1.SSS0.Px2.p1.2.m2.1.1.1.1.1.1.1.2" xref="S2.SS1.SSS0.Px2.p1.2.m2.1.1.1.1.1.1.1.2.cmml">𝒉</mi><mi id="S2.SS1.SSS0.Px2.p1.2.m2.1.1.1.1.1.1.1.3" xref="S2.SS1.SSS0.Px2.p1.2.m2.1.1.1.1.1.1.1.3.cmml">i</mi></msub><mo stretchy="false" id="S2.SS1.SSS0.Px2.p1.2.m2.1.1.1.1.1.1.3" xref="S2.SS1.SSS0.Px2.p1.2.m2.1.1.1.1.1.2.cmml">}</mo></mrow><mn id="S2.SS1.SSS0.Px2.p1.2.m2.1.1.1.1.3" xref="S2.SS1.SSS0.Px2.p1.2.m2.1.1.1.1.3.cmml">0</mn><mrow id="S2.SS1.SSS0.Px2.p1.2.m2.1.1.1.3" xref="S2.SS1.SSS0.Px2.p1.2.m2.1.1.1.3.cmml"><mi id="S2.SS1.SSS0.Px2.p1.2.m2.1.1.1.3.2" xref="S2.SS1.SSS0.Px2.p1.2.m2.1.1.1.3.2.cmml">N</mi><mo id="S2.SS1.SSS0.Px2.p1.2.m2.1.1.1.3.1" xref="S2.SS1.SSS0.Px2.p1.2.m2.1.1.1.3.1.cmml">−</mo><mn id="S2.SS1.SSS0.Px2.p1.2.m2.1.1.1.3.3" xref="S2.SS1.SSS0.Px2.p1.2.m2.1.1.1.3.3.cmml">1</mn></mrow></msubsup><mo id="S2.SS1.SSS0.Px2.p1.2.m2.1.1.5" xref="S2.SS1.SSS0.Px2.p1.2.m2.1.1.5.cmml">∈</mo><msup id="S2.SS1.SSS0.Px2.p1.2.m2.1.1.6" xref="S2.SS1.SSS0.Px2.p1.2.m2.1.1.6.cmml"><mi id="S2.SS1.SSS0.Px2.p1.2.m2.1.1.6.2" xref="S2.SS1.SSS0.Px2.p1.2.m2.1.1.6.2.cmml">ℝ</mi><mrow id="S2.SS1.SSS0.Px2.p1.2.m2.1.1.6.3" xref="S2.SS1.SSS0.Px2.p1.2.m2.1.1.6.3.cmml"><mi id="S2.SS1.SSS0.Px2.p1.2.m2.1.1.6.3.2" xref="S2.SS1.SSS0.Px2.p1.2.m2.1.1.6.3.2.cmml">N</mi><mo lspace="0.222em" rspace="0.222em" id="S2.SS1.SSS0.Px2.p1.2.m2.1.1.6.3.1" xref="S2.SS1.SSS0.Px2.p1.2.m2.1.1.6.3.1.cmml">×</mo><mi id="S2.SS1.SSS0.Px2.p1.2.m2.1.1.6.3.3" xref="S2.SS1.SSS0.Px2.p1.2.m2.1.1.6.3.3.cmml">D</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S2.SS1.SSS0.Px2.p1.2.m2.1b"><apply id="S2.SS1.SSS0.Px2.p1.2.m2.1.1.cmml" xref="S2.SS1.SSS0.Px2.p1.2.m2.1.1"><and id="S2.SS1.SSS0.Px2.p1.2.m2.1.1a.cmml" xref="S2.SS1.SSS0.Px2.p1.2.m2.1.1"></and><apply id="S2.SS1.SSS0.Px2.p1.2.m2.1.1b.cmml" xref="S2.SS1.SSS0.Px2.p1.2.m2.1.1"><eq id="S2.SS1.SSS0.Px2.p1.2.m2.1.1.4.cmml" xref="S2.SS1.SSS0.Px2.p1.2.m2.1.1.4"></eq><ci id="S2.SS1.SSS0.Px2.p1.2.m2.1.1.3.cmml" xref="S2.SS1.SSS0.Px2.p1.2.m2.1.1.3">𝒉</ci><apply id="S2.SS1.SSS0.Px2.p1.2.m2.1.1.1.cmml" xref="S2.SS1.SSS0.Px2.p1.2.m2.1.1.1"><csymbol cd="ambiguous" id="S2.SS1.SSS0.Px2.p1.2.m2.1.1.1.2.cmml" xref="S2.SS1.SSS0.Px2.p1.2.m2.1.1.1">superscript</csymbol><apply id="S2.SS1.SSS0.Px2.p1.2.m2.1.1.1.1.cmml" xref="S2.SS1.SSS0.Px2.p1.2.m2.1.1.1"><csymbol cd="ambiguous" id="S2.SS1.SSS0.Px2.p1.2.m2.1.1.1.1.2.cmml" xref="S2.SS1.SSS0.Px2.p1.2.m2.1.1.1">subscript</csymbol><set id="S2.SS1.SSS0.Px2.p1.2.m2.1.1.1.1.1.2.cmml" xref="S2.SS1.SSS0.Px2.p1.2.m2.1.1.1.1.1.1"><apply id="S2.SS1.SSS0.Px2.p1.2.m2.1.1.1.1.1.1.1.cmml" xref="S2.SS1.SSS0.Px2.p1.2.m2.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS1.SSS0.Px2.p1.2.m2.1.1.1.1.1.1.1.1.cmml" xref="S2.SS1.SSS0.Px2.p1.2.m2.1.1.1.1.1.1.1">subscript</csymbol><ci id="S2.SS1.SSS0.Px2.p1.2.m2.1.1.1.1.1.1.1.2.cmml" xref="S2.SS1.SSS0.Px2.p1.2.m2.1.1.1.1.1.1.1.2">𝒉</ci><ci id="S2.SS1.SSS0.Px2.p1.2.m2.1.1.1.1.1.1.1.3.cmml" xref="S2.SS1.SSS0.Px2.p1.2.m2.1.1.1.1.1.1.1.3">𝑖</ci></apply></set><cn type="integer" id="S2.SS1.SSS0.Px2.p1.2.m2.1.1.1.1.3.cmml" xref="S2.SS1.SSS0.Px2.p1.2.m2.1.1.1.1.3">0</cn></apply><apply id="S2.SS1.SSS0.Px2.p1.2.m2.1.1.1.3.cmml" xref="S2.SS1.SSS0.Px2.p1.2.m2.1.1.1.3"><minus id="S2.SS1.SSS0.Px2.p1.2.m2.1.1.1.3.1.cmml" xref="S2.SS1.SSS0.Px2.p1.2.m2.1.1.1.3.1"></minus><ci id="S2.SS1.SSS0.Px2.p1.2.m2.1.1.1.3.2.cmml" xref="S2.SS1.SSS0.Px2.p1.2.m2.1.1.1.3.2">𝑁</ci><cn type="integer" id="S2.SS1.SSS0.Px2.p1.2.m2.1.1.1.3.3.cmml" xref="S2.SS1.SSS0.Px2.p1.2.m2.1.1.1.3.3">1</cn></apply></apply></apply><apply id="S2.SS1.SSS0.Px2.p1.2.m2.1.1c.cmml" xref="S2.SS1.SSS0.Px2.p1.2.m2.1.1"><in id="S2.SS1.SSS0.Px2.p1.2.m2.1.1.5.cmml" xref="S2.SS1.SSS0.Px2.p1.2.m2.1.1.5"></in><share href="#S2.SS1.SSS0.Px2.p1.2.m2.1.1.1.cmml" id="S2.SS1.SSS0.Px2.p1.2.m2.1.1d.cmml" xref="S2.SS1.SSS0.Px2.p1.2.m2.1.1"></share><apply id="S2.SS1.SSS0.Px2.p1.2.m2.1.1.6.cmml" xref="S2.SS1.SSS0.Px2.p1.2.m2.1.1.6"><csymbol cd="ambiguous" id="S2.SS1.SSS0.Px2.p1.2.m2.1.1.6.1.cmml" xref="S2.SS1.SSS0.Px2.p1.2.m2.1.1.6">superscript</csymbol><ci id="S2.SS1.SSS0.Px2.p1.2.m2.1.1.6.2.cmml" xref="S2.SS1.SSS0.Px2.p1.2.m2.1.1.6.2">ℝ</ci><apply id="S2.SS1.SSS0.Px2.p1.2.m2.1.1.6.3.cmml" xref="S2.SS1.SSS0.Px2.p1.2.m2.1.1.6.3"><times id="S2.SS1.SSS0.Px2.p1.2.m2.1.1.6.3.1.cmml" xref="S2.SS1.SSS0.Px2.p1.2.m2.1.1.6.3.1"></times><ci id="S2.SS1.SSS0.Px2.p1.2.m2.1.1.6.3.2.cmml" xref="S2.SS1.SSS0.Px2.p1.2.m2.1.1.6.3.2">𝑁</ci><ci id="S2.SS1.SSS0.Px2.p1.2.m2.1.1.6.3.3.cmml" xref="S2.SS1.SSS0.Px2.p1.2.m2.1.1.6.3.3">𝐷</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.SSS0.Px2.p1.2.m2.1c">{\bm{h}}=\{{\bm{h}}_{i}\}_{0}^{N-1}\in\mathbb{R}^{N\times D}</annotation></semantics></math> denotes the representation of each word in the input text. For words tokenized into multiple subwords, we use the representation of the first subword, which is a standard choice in the NER literature <cite class="ltx_cite ltx_citemacro_citep">(Zaratiana et al., <a href="#bib.bib49" title="" class="ltx_ref">2022</a>)</cite>.</p>
</div>
</section>
<section id="S2.SS1.SSS0.Px3" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Entity and Span Representation</h4>

<div id="S2.SS1.SSS0.Px3.p1" class="ltx_para">
<p id="S2.SS1.SSS0.Px3.p1.5" class="ltx_p">In our model, we aim to encode entity types and span embeddings into a unified latent space. The entity representation is computed by refining the initial representation <math id="S2.SS1.SSS0.Px3.p1.1.m1.1" class="ltx_Math" alttext="{\bm{p}}" display="inline"><semantics id="S2.SS1.SSS0.Px3.p1.1.m1.1a"><mi id="S2.SS1.SSS0.Px3.p1.1.m1.1.1" xref="S2.SS1.SSS0.Px3.p1.1.m1.1.1.cmml">𝒑</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.SSS0.Px3.p1.1.m1.1b"><ci id="S2.SS1.SSS0.Px3.p1.1.m1.1.1.cmml" xref="S2.SS1.SSS0.Px3.p1.1.m1.1.1">𝒑</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.SSS0.Px3.p1.1.m1.1c">{\bm{p}}</annotation></semantics></math> using a two-layer feedforward network, resulting in <math id="S2.SS1.SSS0.Px3.p1.2.m2.1" class="ltx_Math" alttext="{\bm{q}}=\{{\bm{q}}_{i}\}_{0}^{M-1}\in\mathbb{R}^{M\times D}" display="inline"><semantics id="S2.SS1.SSS0.Px3.p1.2.m2.1a"><mrow id="S2.SS1.SSS0.Px3.p1.2.m2.1.1" xref="S2.SS1.SSS0.Px3.p1.2.m2.1.1.cmml"><mi id="S2.SS1.SSS0.Px3.p1.2.m2.1.1.3" xref="S2.SS1.SSS0.Px3.p1.2.m2.1.1.3.cmml">𝒒</mi><mo id="S2.SS1.SSS0.Px3.p1.2.m2.1.1.4" xref="S2.SS1.SSS0.Px3.p1.2.m2.1.1.4.cmml">=</mo><msubsup id="S2.SS1.SSS0.Px3.p1.2.m2.1.1.1" xref="S2.SS1.SSS0.Px3.p1.2.m2.1.1.1.cmml"><mrow id="S2.SS1.SSS0.Px3.p1.2.m2.1.1.1.1.1.1" xref="S2.SS1.SSS0.Px3.p1.2.m2.1.1.1.1.1.2.cmml"><mo stretchy="false" id="S2.SS1.SSS0.Px3.p1.2.m2.1.1.1.1.1.1.2" xref="S2.SS1.SSS0.Px3.p1.2.m2.1.1.1.1.1.2.cmml">{</mo><msub id="S2.SS1.SSS0.Px3.p1.2.m2.1.1.1.1.1.1.1" xref="S2.SS1.SSS0.Px3.p1.2.m2.1.1.1.1.1.1.1.cmml"><mi id="S2.SS1.SSS0.Px3.p1.2.m2.1.1.1.1.1.1.1.2" xref="S2.SS1.SSS0.Px3.p1.2.m2.1.1.1.1.1.1.1.2.cmml">𝒒</mi><mi id="S2.SS1.SSS0.Px3.p1.2.m2.1.1.1.1.1.1.1.3" xref="S2.SS1.SSS0.Px3.p1.2.m2.1.1.1.1.1.1.1.3.cmml">i</mi></msub><mo stretchy="false" id="S2.SS1.SSS0.Px3.p1.2.m2.1.1.1.1.1.1.3" xref="S2.SS1.SSS0.Px3.p1.2.m2.1.1.1.1.1.2.cmml">}</mo></mrow><mn id="S2.SS1.SSS0.Px3.p1.2.m2.1.1.1.1.3" xref="S2.SS1.SSS0.Px3.p1.2.m2.1.1.1.1.3.cmml">0</mn><mrow id="S2.SS1.SSS0.Px3.p1.2.m2.1.1.1.3" xref="S2.SS1.SSS0.Px3.p1.2.m2.1.1.1.3.cmml"><mi id="S2.SS1.SSS0.Px3.p1.2.m2.1.1.1.3.2" xref="S2.SS1.SSS0.Px3.p1.2.m2.1.1.1.3.2.cmml">M</mi><mo id="S2.SS1.SSS0.Px3.p1.2.m2.1.1.1.3.1" xref="S2.SS1.SSS0.Px3.p1.2.m2.1.1.1.3.1.cmml">−</mo><mn id="S2.SS1.SSS0.Px3.p1.2.m2.1.1.1.3.3" xref="S2.SS1.SSS0.Px3.p1.2.m2.1.1.1.3.3.cmml">1</mn></mrow></msubsup><mo id="S2.SS1.SSS0.Px3.p1.2.m2.1.1.5" xref="S2.SS1.SSS0.Px3.p1.2.m2.1.1.5.cmml">∈</mo><msup id="S2.SS1.SSS0.Px3.p1.2.m2.1.1.6" xref="S2.SS1.SSS0.Px3.p1.2.m2.1.1.6.cmml"><mi id="S2.SS1.SSS0.Px3.p1.2.m2.1.1.6.2" xref="S2.SS1.SSS0.Px3.p1.2.m2.1.1.6.2.cmml">ℝ</mi><mrow id="S2.SS1.SSS0.Px3.p1.2.m2.1.1.6.3" xref="S2.SS1.SSS0.Px3.p1.2.m2.1.1.6.3.cmml"><mi id="S2.SS1.SSS0.Px3.p1.2.m2.1.1.6.3.2" xref="S2.SS1.SSS0.Px3.p1.2.m2.1.1.6.3.2.cmml">M</mi><mo lspace="0.222em" rspace="0.222em" id="S2.SS1.SSS0.Px3.p1.2.m2.1.1.6.3.1" xref="S2.SS1.SSS0.Px3.p1.2.m2.1.1.6.3.1.cmml">×</mo><mi id="S2.SS1.SSS0.Px3.p1.2.m2.1.1.6.3.3" xref="S2.SS1.SSS0.Px3.p1.2.m2.1.1.6.3.3.cmml">D</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S2.SS1.SSS0.Px3.p1.2.m2.1b"><apply id="S2.SS1.SSS0.Px3.p1.2.m2.1.1.cmml" xref="S2.SS1.SSS0.Px3.p1.2.m2.1.1"><and id="S2.SS1.SSS0.Px3.p1.2.m2.1.1a.cmml" xref="S2.SS1.SSS0.Px3.p1.2.m2.1.1"></and><apply id="S2.SS1.SSS0.Px3.p1.2.m2.1.1b.cmml" xref="S2.SS1.SSS0.Px3.p1.2.m2.1.1"><eq id="S2.SS1.SSS0.Px3.p1.2.m2.1.1.4.cmml" xref="S2.SS1.SSS0.Px3.p1.2.m2.1.1.4"></eq><ci id="S2.SS1.SSS0.Px3.p1.2.m2.1.1.3.cmml" xref="S2.SS1.SSS0.Px3.p1.2.m2.1.1.3">𝒒</ci><apply id="S2.SS1.SSS0.Px3.p1.2.m2.1.1.1.cmml" xref="S2.SS1.SSS0.Px3.p1.2.m2.1.1.1"><csymbol cd="ambiguous" id="S2.SS1.SSS0.Px3.p1.2.m2.1.1.1.2.cmml" xref="S2.SS1.SSS0.Px3.p1.2.m2.1.1.1">superscript</csymbol><apply id="S2.SS1.SSS0.Px3.p1.2.m2.1.1.1.1.cmml" xref="S2.SS1.SSS0.Px3.p1.2.m2.1.1.1"><csymbol cd="ambiguous" id="S2.SS1.SSS0.Px3.p1.2.m2.1.1.1.1.2.cmml" xref="S2.SS1.SSS0.Px3.p1.2.m2.1.1.1">subscript</csymbol><set id="S2.SS1.SSS0.Px3.p1.2.m2.1.1.1.1.1.2.cmml" xref="S2.SS1.SSS0.Px3.p1.2.m2.1.1.1.1.1.1"><apply id="S2.SS1.SSS0.Px3.p1.2.m2.1.1.1.1.1.1.1.cmml" xref="S2.SS1.SSS0.Px3.p1.2.m2.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS1.SSS0.Px3.p1.2.m2.1.1.1.1.1.1.1.1.cmml" xref="S2.SS1.SSS0.Px3.p1.2.m2.1.1.1.1.1.1.1">subscript</csymbol><ci id="S2.SS1.SSS0.Px3.p1.2.m2.1.1.1.1.1.1.1.2.cmml" xref="S2.SS1.SSS0.Px3.p1.2.m2.1.1.1.1.1.1.1.2">𝒒</ci><ci id="S2.SS1.SSS0.Px3.p1.2.m2.1.1.1.1.1.1.1.3.cmml" xref="S2.SS1.SSS0.Px3.p1.2.m2.1.1.1.1.1.1.1.3">𝑖</ci></apply></set><cn type="integer" id="S2.SS1.SSS0.Px3.p1.2.m2.1.1.1.1.3.cmml" xref="S2.SS1.SSS0.Px3.p1.2.m2.1.1.1.1.3">0</cn></apply><apply id="S2.SS1.SSS0.Px3.p1.2.m2.1.1.1.3.cmml" xref="S2.SS1.SSS0.Px3.p1.2.m2.1.1.1.3"><minus id="S2.SS1.SSS0.Px3.p1.2.m2.1.1.1.3.1.cmml" xref="S2.SS1.SSS0.Px3.p1.2.m2.1.1.1.3.1"></minus><ci id="S2.SS1.SSS0.Px3.p1.2.m2.1.1.1.3.2.cmml" xref="S2.SS1.SSS0.Px3.p1.2.m2.1.1.1.3.2">𝑀</ci><cn type="integer" id="S2.SS1.SSS0.Px3.p1.2.m2.1.1.1.3.3.cmml" xref="S2.SS1.SSS0.Px3.p1.2.m2.1.1.1.3.3">1</cn></apply></apply></apply><apply id="S2.SS1.SSS0.Px3.p1.2.m2.1.1c.cmml" xref="S2.SS1.SSS0.Px3.p1.2.m2.1.1"><in id="S2.SS1.SSS0.Px3.p1.2.m2.1.1.5.cmml" xref="S2.SS1.SSS0.Px3.p1.2.m2.1.1.5"></in><share href="#S2.SS1.SSS0.Px3.p1.2.m2.1.1.1.cmml" id="S2.SS1.SSS0.Px3.p1.2.m2.1.1d.cmml" xref="S2.SS1.SSS0.Px3.p1.2.m2.1.1"></share><apply id="S2.SS1.SSS0.Px3.p1.2.m2.1.1.6.cmml" xref="S2.SS1.SSS0.Px3.p1.2.m2.1.1.6"><csymbol cd="ambiguous" id="S2.SS1.SSS0.Px3.p1.2.m2.1.1.6.1.cmml" xref="S2.SS1.SSS0.Px3.p1.2.m2.1.1.6">superscript</csymbol><ci id="S2.SS1.SSS0.Px3.p1.2.m2.1.1.6.2.cmml" xref="S2.SS1.SSS0.Px3.p1.2.m2.1.1.6.2">ℝ</ci><apply id="S2.SS1.SSS0.Px3.p1.2.m2.1.1.6.3.cmml" xref="S2.SS1.SSS0.Px3.p1.2.m2.1.1.6.3"><times id="S2.SS1.SSS0.Px3.p1.2.m2.1.1.6.3.1.cmml" xref="S2.SS1.SSS0.Px3.p1.2.m2.1.1.6.3.1"></times><ci id="S2.SS1.SSS0.Px3.p1.2.m2.1.1.6.3.2.cmml" xref="S2.SS1.SSS0.Px3.p1.2.m2.1.1.6.3.2">𝑀</ci><ci id="S2.SS1.SSS0.Px3.p1.2.m2.1.1.6.3.3.cmml" xref="S2.SS1.SSS0.Px3.p1.2.m2.1.1.6.3.3">𝐷</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.SSS0.Px3.p1.2.m2.1c">{\bm{q}}=\{{\bm{q}}_{i}\}_{0}^{M-1}\in\mathbb{R}^{M\times D}</annotation></semantics></math>. The representation of a span starting at position <math id="S2.SS1.SSS0.Px3.p1.3.m3.1" class="ltx_Math" alttext="i" display="inline"><semantics id="S2.SS1.SSS0.Px3.p1.3.m3.1a"><mi id="S2.SS1.SSS0.Px3.p1.3.m3.1.1" xref="S2.SS1.SSS0.Px3.p1.3.m3.1.1.cmml">i</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.SSS0.Px3.p1.3.m3.1b"><ci id="S2.SS1.SSS0.Px3.p1.3.m3.1.1.cmml" xref="S2.SS1.SSS0.Px3.p1.3.m3.1.1">𝑖</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.SSS0.Px3.p1.3.m3.1c">i</annotation></semantics></math> and ending at position <math id="S2.SS1.SSS0.Px3.p1.4.m4.1" class="ltx_Math" alttext="j" display="inline"><semantics id="S2.SS1.SSS0.Px3.p1.4.m4.1a"><mi id="S2.SS1.SSS0.Px3.p1.4.m4.1.1" xref="S2.SS1.SSS0.Px3.p1.4.m4.1.1.cmml">j</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.SSS0.Px3.p1.4.m4.1b"><ci id="S2.SS1.SSS0.Px3.p1.4.m4.1.1.cmml" xref="S2.SS1.SSS0.Px3.p1.4.m4.1.1">𝑗</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.SSS0.Px3.p1.4.m4.1c">j</annotation></semantics></math> in the input text, <math id="S2.SS1.SSS0.Px3.p1.5.m5.1" class="ltx_Math" alttext="{\bm{S}}_{ij}\in\mathbb{R}^{D}" display="inline"><semantics id="S2.SS1.SSS0.Px3.p1.5.m5.1a"><mrow id="S2.SS1.SSS0.Px3.p1.5.m5.1.1" xref="S2.SS1.SSS0.Px3.p1.5.m5.1.1.cmml"><msub id="S2.SS1.SSS0.Px3.p1.5.m5.1.1.2" xref="S2.SS1.SSS0.Px3.p1.5.m5.1.1.2.cmml"><mi id="S2.SS1.SSS0.Px3.p1.5.m5.1.1.2.2" xref="S2.SS1.SSS0.Px3.p1.5.m5.1.1.2.2.cmml">𝑺</mi><mrow id="S2.SS1.SSS0.Px3.p1.5.m5.1.1.2.3" xref="S2.SS1.SSS0.Px3.p1.5.m5.1.1.2.3.cmml"><mi id="S2.SS1.SSS0.Px3.p1.5.m5.1.1.2.3.2" xref="S2.SS1.SSS0.Px3.p1.5.m5.1.1.2.3.2.cmml">i</mi><mo lspace="0em" rspace="0em" id="S2.SS1.SSS0.Px3.p1.5.m5.1.1.2.3.1" xref="S2.SS1.SSS0.Px3.p1.5.m5.1.1.2.3.1.cmml">​</mo><mi id="S2.SS1.SSS0.Px3.p1.5.m5.1.1.2.3.3" xref="S2.SS1.SSS0.Px3.p1.5.m5.1.1.2.3.3.cmml">j</mi></mrow></msub><mo id="S2.SS1.SSS0.Px3.p1.5.m5.1.1.1" xref="S2.SS1.SSS0.Px3.p1.5.m5.1.1.1.cmml">∈</mo><msup id="S2.SS1.SSS0.Px3.p1.5.m5.1.1.3" xref="S2.SS1.SSS0.Px3.p1.5.m5.1.1.3.cmml"><mi id="S2.SS1.SSS0.Px3.p1.5.m5.1.1.3.2" xref="S2.SS1.SSS0.Px3.p1.5.m5.1.1.3.2.cmml">ℝ</mi><mi id="S2.SS1.SSS0.Px3.p1.5.m5.1.1.3.3" xref="S2.SS1.SSS0.Px3.p1.5.m5.1.1.3.3.cmml">D</mi></msup></mrow><annotation-xml encoding="MathML-Content" id="S2.SS1.SSS0.Px3.p1.5.m5.1b"><apply id="S2.SS1.SSS0.Px3.p1.5.m5.1.1.cmml" xref="S2.SS1.SSS0.Px3.p1.5.m5.1.1"><in id="S2.SS1.SSS0.Px3.p1.5.m5.1.1.1.cmml" xref="S2.SS1.SSS0.Px3.p1.5.m5.1.1.1"></in><apply id="S2.SS1.SSS0.Px3.p1.5.m5.1.1.2.cmml" xref="S2.SS1.SSS0.Px3.p1.5.m5.1.1.2"><csymbol cd="ambiguous" id="S2.SS1.SSS0.Px3.p1.5.m5.1.1.2.1.cmml" xref="S2.SS1.SSS0.Px3.p1.5.m5.1.1.2">subscript</csymbol><ci id="S2.SS1.SSS0.Px3.p1.5.m5.1.1.2.2.cmml" xref="S2.SS1.SSS0.Px3.p1.5.m5.1.1.2.2">𝑺</ci><apply id="S2.SS1.SSS0.Px3.p1.5.m5.1.1.2.3.cmml" xref="S2.SS1.SSS0.Px3.p1.5.m5.1.1.2.3"><times id="S2.SS1.SSS0.Px3.p1.5.m5.1.1.2.3.1.cmml" xref="S2.SS1.SSS0.Px3.p1.5.m5.1.1.2.3.1"></times><ci id="S2.SS1.SSS0.Px3.p1.5.m5.1.1.2.3.2.cmml" xref="S2.SS1.SSS0.Px3.p1.5.m5.1.1.2.3.2">𝑖</ci><ci id="S2.SS1.SSS0.Px3.p1.5.m5.1.1.2.3.3.cmml" xref="S2.SS1.SSS0.Px3.p1.5.m5.1.1.2.3.3">𝑗</ci></apply></apply><apply id="S2.SS1.SSS0.Px3.p1.5.m5.1.1.3.cmml" xref="S2.SS1.SSS0.Px3.p1.5.m5.1.1.3"><csymbol cd="ambiguous" id="S2.SS1.SSS0.Px3.p1.5.m5.1.1.3.1.cmml" xref="S2.SS1.SSS0.Px3.p1.5.m5.1.1.3">superscript</csymbol><ci id="S2.SS1.SSS0.Px3.p1.5.m5.1.1.3.2.cmml" xref="S2.SS1.SSS0.Px3.p1.5.m5.1.1.3.2">ℝ</ci><ci id="S2.SS1.SSS0.Px3.p1.5.m5.1.1.3.3.cmml" xref="S2.SS1.SSS0.Px3.p1.5.m5.1.1.3.3">𝐷</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.SSS0.Px3.p1.5.m5.1c">{\bm{S}}_{ij}\in\mathbb{R}^{D}</annotation></semantics></math>, is computed as:</p>
<table id="S2.E1" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S2.E1.m1.1" class="ltx_Math" alttext="{\bm{S}}_{ij}=\texttt{FFN}({\bm{h}}_{i}\otimes{\bm{h}}_{j})" display="block"><semantics id="S2.E1.m1.1a"><mrow id="S2.E1.m1.1.1" xref="S2.E1.m1.1.1.cmml"><msub id="S2.E1.m1.1.1.3" xref="S2.E1.m1.1.1.3.cmml"><mi id="S2.E1.m1.1.1.3.2" xref="S2.E1.m1.1.1.3.2.cmml">𝑺</mi><mrow id="S2.E1.m1.1.1.3.3" xref="S2.E1.m1.1.1.3.3.cmml"><mi id="S2.E1.m1.1.1.3.3.2" xref="S2.E1.m1.1.1.3.3.2.cmml">i</mi><mo lspace="0em" rspace="0em" id="S2.E1.m1.1.1.3.3.1" xref="S2.E1.m1.1.1.3.3.1.cmml">​</mo><mi id="S2.E1.m1.1.1.3.3.3" xref="S2.E1.m1.1.1.3.3.3.cmml">j</mi></mrow></msub><mo id="S2.E1.m1.1.1.2" xref="S2.E1.m1.1.1.2.cmml">=</mo><mrow id="S2.E1.m1.1.1.1" xref="S2.E1.m1.1.1.1.cmml"><mtext class="ltx_mathvariant_monospace" id="S2.E1.m1.1.1.1.3" xref="S2.E1.m1.1.1.1.3a.cmml">FFN</mtext><mo lspace="0em" rspace="0em" id="S2.E1.m1.1.1.1.2" xref="S2.E1.m1.1.1.1.2.cmml">​</mo><mrow id="S2.E1.m1.1.1.1.1.1" xref="S2.E1.m1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.E1.m1.1.1.1.1.1.2" xref="S2.E1.m1.1.1.1.1.1.1.cmml">(</mo><mrow id="S2.E1.m1.1.1.1.1.1.1" xref="S2.E1.m1.1.1.1.1.1.1.cmml"><msub id="S2.E1.m1.1.1.1.1.1.1.2" xref="S2.E1.m1.1.1.1.1.1.1.2.cmml"><mi id="S2.E1.m1.1.1.1.1.1.1.2.2" xref="S2.E1.m1.1.1.1.1.1.1.2.2.cmml">𝒉</mi><mi id="S2.E1.m1.1.1.1.1.1.1.2.3" xref="S2.E1.m1.1.1.1.1.1.1.2.3.cmml">i</mi></msub><mo lspace="0.222em" rspace="0.222em" id="S2.E1.m1.1.1.1.1.1.1.1" xref="S2.E1.m1.1.1.1.1.1.1.1.cmml">⊗</mo><msub id="S2.E1.m1.1.1.1.1.1.1.3" xref="S2.E1.m1.1.1.1.1.1.1.3.cmml"><mi id="S2.E1.m1.1.1.1.1.1.1.3.2" xref="S2.E1.m1.1.1.1.1.1.1.3.2.cmml">𝒉</mi><mi id="S2.E1.m1.1.1.1.1.1.1.3.3" xref="S2.E1.m1.1.1.1.1.1.1.3.3.cmml">j</mi></msub></mrow><mo stretchy="false" id="S2.E1.m1.1.1.1.1.1.3" xref="S2.E1.m1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.E1.m1.1b"><apply id="S2.E1.m1.1.1.cmml" xref="S2.E1.m1.1.1"><eq id="S2.E1.m1.1.1.2.cmml" xref="S2.E1.m1.1.1.2"></eq><apply id="S2.E1.m1.1.1.3.cmml" xref="S2.E1.m1.1.1.3"><csymbol cd="ambiguous" id="S2.E1.m1.1.1.3.1.cmml" xref="S2.E1.m1.1.1.3">subscript</csymbol><ci id="S2.E1.m1.1.1.3.2.cmml" xref="S2.E1.m1.1.1.3.2">𝑺</ci><apply id="S2.E1.m1.1.1.3.3.cmml" xref="S2.E1.m1.1.1.3.3"><times id="S2.E1.m1.1.1.3.3.1.cmml" xref="S2.E1.m1.1.1.3.3.1"></times><ci id="S2.E1.m1.1.1.3.3.2.cmml" xref="S2.E1.m1.1.1.3.3.2">𝑖</ci><ci id="S2.E1.m1.1.1.3.3.3.cmml" xref="S2.E1.m1.1.1.3.3.3">𝑗</ci></apply></apply><apply id="S2.E1.m1.1.1.1.cmml" xref="S2.E1.m1.1.1.1"><times id="S2.E1.m1.1.1.1.2.cmml" xref="S2.E1.m1.1.1.1.2"></times><ci id="S2.E1.m1.1.1.1.3a.cmml" xref="S2.E1.m1.1.1.1.3"><mtext class="ltx_mathvariant_monospace" id="S2.E1.m1.1.1.1.3.cmml" xref="S2.E1.m1.1.1.1.3">FFN</mtext></ci><apply id="S2.E1.m1.1.1.1.1.1.1.cmml" xref="S2.E1.m1.1.1.1.1.1"><csymbol cd="latexml" id="S2.E1.m1.1.1.1.1.1.1.1.cmml" xref="S2.E1.m1.1.1.1.1.1.1.1">tensor-product</csymbol><apply id="S2.E1.m1.1.1.1.1.1.1.2.cmml" xref="S2.E1.m1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S2.E1.m1.1.1.1.1.1.1.2.1.cmml" xref="S2.E1.m1.1.1.1.1.1.1.2">subscript</csymbol><ci id="S2.E1.m1.1.1.1.1.1.1.2.2.cmml" xref="S2.E1.m1.1.1.1.1.1.1.2.2">𝒉</ci><ci id="S2.E1.m1.1.1.1.1.1.1.2.3.cmml" xref="S2.E1.m1.1.1.1.1.1.1.2.3">𝑖</ci></apply><apply id="S2.E1.m1.1.1.1.1.1.1.3.cmml" xref="S2.E1.m1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.E1.m1.1.1.1.1.1.1.3.1.cmml" xref="S2.E1.m1.1.1.1.1.1.1.3">subscript</csymbol><ci id="S2.E1.m1.1.1.1.1.1.1.3.2.cmml" xref="S2.E1.m1.1.1.1.1.1.1.3.2">𝒉</ci><ci id="S2.E1.m1.1.1.1.1.1.1.3.3.cmml" xref="S2.E1.m1.1.1.1.1.1.1.3.3">𝑗</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E1.m1.1c">{\bm{S}}_{ij}=\texttt{FFN}({\bm{h}}_{i}\otimes{\bm{h}}_{j})</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(1)</span></td>
</tr></tbody>
</table>
</div>
<div id="S2.SS1.SSS0.Px3.p2" class="ltx_para">
<p id="S2.SS1.SSS0.Px3.p2.1" class="ltx_p">Here, <span id="S2.SS1.SSS0.Px3.p2.1.1" class="ltx_text ltx_font_typewriter">FFN</span> denotes a two-layer feedforward network, and <math id="S2.SS1.SSS0.Px3.p2.1.m1.1" class="ltx_Math" alttext="\otimes" display="inline"><semantics id="S2.SS1.SSS0.Px3.p2.1.m1.1a"><mo id="S2.SS1.SSS0.Px3.p2.1.m1.1.1" xref="S2.SS1.SSS0.Px3.p2.1.m1.1.1.cmml">⊗</mo><annotation-xml encoding="MathML-Content" id="S2.SS1.SSS0.Px3.p2.1.m1.1b"><csymbol cd="latexml" id="S2.SS1.SSS0.Px3.p2.1.m1.1.1.cmml" xref="S2.SS1.SSS0.Px3.p2.1.m1.1.1">tensor-product</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.SSS0.Px3.p2.1.m1.1c">\otimes</annotation></semantics></math> represents the concatenation operation. In practice, The computation of all span representations can be easily parallelized. Moreover, we set an upper bound to the length (K=12) of the span in order to keep linear complexity, without harming recall.</p>
</div>
</section>
<section id="S2.SS1.SSS0.Px4" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Entity Type and Span Matching</h4>

<div id="S2.SS1.SSS0.Px4.p1" class="ltx_para">
<p id="S2.SS1.SSS0.Px4.p1.2" class="ltx_p">To evaluate whether a span <math id="S2.SS1.SSS0.Px4.p1.1.m1.2" class="ltx_Math" alttext="(i,j)" display="inline"><semantics id="S2.SS1.SSS0.Px4.p1.1.m1.2a"><mrow id="S2.SS1.SSS0.Px4.p1.1.m1.2.3.2" xref="S2.SS1.SSS0.Px4.p1.1.m1.2.3.1.cmml"><mo stretchy="false" id="S2.SS1.SSS0.Px4.p1.1.m1.2.3.2.1" xref="S2.SS1.SSS0.Px4.p1.1.m1.2.3.1.cmml">(</mo><mi id="S2.SS1.SSS0.Px4.p1.1.m1.1.1" xref="S2.SS1.SSS0.Px4.p1.1.m1.1.1.cmml">i</mi><mo id="S2.SS1.SSS0.Px4.p1.1.m1.2.3.2.2" xref="S2.SS1.SSS0.Px4.p1.1.m1.2.3.1.cmml">,</mo><mi id="S2.SS1.SSS0.Px4.p1.1.m1.2.2" xref="S2.SS1.SSS0.Px4.p1.1.m1.2.2.cmml">j</mi><mo stretchy="false" id="S2.SS1.SSS0.Px4.p1.1.m1.2.3.2.3" xref="S2.SS1.SSS0.Px4.p1.1.m1.2.3.1.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.SS1.SSS0.Px4.p1.1.m1.2b"><interval closure="open" id="S2.SS1.SSS0.Px4.p1.1.m1.2.3.1.cmml" xref="S2.SS1.SSS0.Px4.p1.1.m1.2.3.2"><ci id="S2.SS1.SSS0.Px4.p1.1.m1.1.1.cmml" xref="S2.SS1.SSS0.Px4.p1.1.m1.1.1">𝑖</ci><ci id="S2.SS1.SSS0.Px4.p1.1.m1.2.2.cmml" xref="S2.SS1.SSS0.Px4.p1.1.m1.2.2">𝑗</ci></interval></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.SSS0.Px4.p1.1.m1.2c">(i,j)</annotation></semantics></math> corresponds to entity type <math id="S2.SS1.SSS0.Px4.p1.2.m2.1" class="ltx_Math" alttext="t" display="inline"><semantics id="S2.SS1.SSS0.Px4.p1.2.m2.1a"><mi id="S2.SS1.SSS0.Px4.p1.2.m2.1.1" xref="S2.SS1.SSS0.Px4.p1.2.m2.1.1.cmml">t</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.SSS0.Px4.p1.2.m2.1b"><ci id="S2.SS1.SSS0.Px4.p1.2.m2.1.1.cmml" xref="S2.SS1.SSS0.Px4.p1.2.m2.1.1">𝑡</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.SSS0.Px4.p1.2.m2.1c">t</annotation></semantics></math>, we calculate the following matching score:</p>
</div>
<div id="S2.SS1.SSS0.Px4.p2" class="ltx_para">
<table id="S2.E2" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S2.E2.m1.4" class="ltx_Math" alttext="\phi(i,j,t)=\sigma({\bm{S}}_{ij}^{T}{\bm{q}}_{t})\in\mathbb{R}" display="block"><semantics id="S2.E2.m1.4a"><mrow id="S2.E2.m1.4.4" xref="S2.E2.m1.4.4.cmml"><mrow id="S2.E2.m1.4.4.3" xref="S2.E2.m1.4.4.3.cmml"><mi id="S2.E2.m1.4.4.3.2" xref="S2.E2.m1.4.4.3.2.cmml">ϕ</mi><mo lspace="0em" rspace="0em" id="S2.E2.m1.4.4.3.1" xref="S2.E2.m1.4.4.3.1.cmml">​</mo><mrow id="S2.E2.m1.4.4.3.3.2" xref="S2.E2.m1.4.4.3.3.1.cmml"><mo stretchy="false" id="S2.E2.m1.4.4.3.3.2.1" xref="S2.E2.m1.4.4.3.3.1.cmml">(</mo><mi id="S2.E2.m1.1.1" xref="S2.E2.m1.1.1.cmml">i</mi><mo id="S2.E2.m1.4.4.3.3.2.2" xref="S2.E2.m1.4.4.3.3.1.cmml">,</mo><mi id="S2.E2.m1.2.2" xref="S2.E2.m1.2.2.cmml">j</mi><mo id="S2.E2.m1.4.4.3.3.2.3" xref="S2.E2.m1.4.4.3.3.1.cmml">,</mo><mi id="S2.E2.m1.3.3" xref="S2.E2.m1.3.3.cmml">t</mi><mo stretchy="false" id="S2.E2.m1.4.4.3.3.2.4" xref="S2.E2.m1.4.4.3.3.1.cmml">)</mo></mrow></mrow><mo id="S2.E2.m1.4.4.4" xref="S2.E2.m1.4.4.4.cmml">=</mo><mrow id="S2.E2.m1.4.4.1" xref="S2.E2.m1.4.4.1.cmml"><mi id="S2.E2.m1.4.4.1.3" xref="S2.E2.m1.4.4.1.3.cmml">σ</mi><mo lspace="0em" rspace="0em" id="S2.E2.m1.4.4.1.2" xref="S2.E2.m1.4.4.1.2.cmml">​</mo><mrow id="S2.E2.m1.4.4.1.1.1" xref="S2.E2.m1.4.4.1.1.1.1.cmml"><mo stretchy="false" id="S2.E2.m1.4.4.1.1.1.2" xref="S2.E2.m1.4.4.1.1.1.1.cmml">(</mo><mrow id="S2.E2.m1.4.4.1.1.1.1" xref="S2.E2.m1.4.4.1.1.1.1.cmml"><msubsup id="S2.E2.m1.4.4.1.1.1.1.2" xref="S2.E2.m1.4.4.1.1.1.1.2.cmml"><mi id="S2.E2.m1.4.4.1.1.1.1.2.2.2" xref="S2.E2.m1.4.4.1.1.1.1.2.2.2.cmml">𝑺</mi><mrow id="S2.E2.m1.4.4.1.1.1.1.2.2.3" xref="S2.E2.m1.4.4.1.1.1.1.2.2.3.cmml"><mi id="S2.E2.m1.4.4.1.1.1.1.2.2.3.2" xref="S2.E2.m1.4.4.1.1.1.1.2.2.3.2.cmml">i</mi><mo lspace="0em" rspace="0em" id="S2.E2.m1.4.4.1.1.1.1.2.2.3.1" xref="S2.E2.m1.4.4.1.1.1.1.2.2.3.1.cmml">​</mo><mi id="S2.E2.m1.4.4.1.1.1.1.2.2.3.3" xref="S2.E2.m1.4.4.1.1.1.1.2.2.3.3.cmml">j</mi></mrow><mi id="S2.E2.m1.4.4.1.1.1.1.2.3" xref="S2.E2.m1.4.4.1.1.1.1.2.3.cmml">T</mi></msubsup><mo lspace="0em" rspace="0em" id="S2.E2.m1.4.4.1.1.1.1.1" xref="S2.E2.m1.4.4.1.1.1.1.1.cmml">​</mo><msub id="S2.E2.m1.4.4.1.1.1.1.3" xref="S2.E2.m1.4.4.1.1.1.1.3.cmml"><mi id="S2.E2.m1.4.4.1.1.1.1.3.2" xref="S2.E2.m1.4.4.1.1.1.1.3.2.cmml">𝒒</mi><mi id="S2.E2.m1.4.4.1.1.1.1.3.3" xref="S2.E2.m1.4.4.1.1.1.1.3.3.cmml">t</mi></msub></mrow><mo stretchy="false" id="S2.E2.m1.4.4.1.1.1.3" xref="S2.E2.m1.4.4.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S2.E2.m1.4.4.5" xref="S2.E2.m1.4.4.5.cmml">∈</mo><mi id="S2.E2.m1.4.4.6" xref="S2.E2.m1.4.4.6.cmml">ℝ</mi></mrow><annotation-xml encoding="MathML-Content" id="S2.E2.m1.4b"><apply id="S2.E2.m1.4.4.cmml" xref="S2.E2.m1.4.4"><and id="S2.E2.m1.4.4a.cmml" xref="S2.E2.m1.4.4"></and><apply id="S2.E2.m1.4.4b.cmml" xref="S2.E2.m1.4.4"><eq id="S2.E2.m1.4.4.4.cmml" xref="S2.E2.m1.4.4.4"></eq><apply id="S2.E2.m1.4.4.3.cmml" xref="S2.E2.m1.4.4.3"><times id="S2.E2.m1.4.4.3.1.cmml" xref="S2.E2.m1.4.4.3.1"></times><ci id="S2.E2.m1.4.4.3.2.cmml" xref="S2.E2.m1.4.4.3.2">italic-ϕ</ci><vector id="S2.E2.m1.4.4.3.3.1.cmml" xref="S2.E2.m1.4.4.3.3.2"><ci id="S2.E2.m1.1.1.cmml" xref="S2.E2.m1.1.1">𝑖</ci><ci id="S2.E2.m1.2.2.cmml" xref="S2.E2.m1.2.2">𝑗</ci><ci id="S2.E2.m1.3.3.cmml" xref="S2.E2.m1.3.3">𝑡</ci></vector></apply><apply id="S2.E2.m1.4.4.1.cmml" xref="S2.E2.m1.4.4.1"><times id="S2.E2.m1.4.4.1.2.cmml" xref="S2.E2.m1.4.4.1.2"></times><ci id="S2.E2.m1.4.4.1.3.cmml" xref="S2.E2.m1.4.4.1.3">𝜎</ci><apply id="S2.E2.m1.4.4.1.1.1.1.cmml" xref="S2.E2.m1.4.4.1.1.1"><times id="S2.E2.m1.4.4.1.1.1.1.1.cmml" xref="S2.E2.m1.4.4.1.1.1.1.1"></times><apply id="S2.E2.m1.4.4.1.1.1.1.2.cmml" xref="S2.E2.m1.4.4.1.1.1.1.2"><csymbol cd="ambiguous" id="S2.E2.m1.4.4.1.1.1.1.2.1.cmml" xref="S2.E2.m1.4.4.1.1.1.1.2">superscript</csymbol><apply id="S2.E2.m1.4.4.1.1.1.1.2.2.cmml" xref="S2.E2.m1.4.4.1.1.1.1.2"><csymbol cd="ambiguous" id="S2.E2.m1.4.4.1.1.1.1.2.2.1.cmml" xref="S2.E2.m1.4.4.1.1.1.1.2">subscript</csymbol><ci id="S2.E2.m1.4.4.1.1.1.1.2.2.2.cmml" xref="S2.E2.m1.4.4.1.1.1.1.2.2.2">𝑺</ci><apply id="S2.E2.m1.4.4.1.1.1.1.2.2.3.cmml" xref="S2.E2.m1.4.4.1.1.1.1.2.2.3"><times id="S2.E2.m1.4.4.1.1.1.1.2.2.3.1.cmml" xref="S2.E2.m1.4.4.1.1.1.1.2.2.3.1"></times><ci id="S2.E2.m1.4.4.1.1.1.1.2.2.3.2.cmml" xref="S2.E2.m1.4.4.1.1.1.1.2.2.3.2">𝑖</ci><ci id="S2.E2.m1.4.4.1.1.1.1.2.2.3.3.cmml" xref="S2.E2.m1.4.4.1.1.1.1.2.2.3.3">𝑗</ci></apply></apply><ci id="S2.E2.m1.4.4.1.1.1.1.2.3.cmml" xref="S2.E2.m1.4.4.1.1.1.1.2.3">𝑇</ci></apply><apply id="S2.E2.m1.4.4.1.1.1.1.3.cmml" xref="S2.E2.m1.4.4.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.E2.m1.4.4.1.1.1.1.3.1.cmml" xref="S2.E2.m1.4.4.1.1.1.1.3">subscript</csymbol><ci id="S2.E2.m1.4.4.1.1.1.1.3.2.cmml" xref="S2.E2.m1.4.4.1.1.1.1.3.2">𝒒</ci><ci id="S2.E2.m1.4.4.1.1.1.1.3.3.cmml" xref="S2.E2.m1.4.4.1.1.1.1.3.3">𝑡</ci></apply></apply></apply></apply><apply id="S2.E2.m1.4.4c.cmml" xref="S2.E2.m1.4.4"><in id="S2.E2.m1.4.4.5.cmml" xref="S2.E2.m1.4.4.5"></in><share href="#S2.E2.m1.4.4.1.cmml" id="S2.E2.m1.4.4d.cmml" xref="S2.E2.m1.4.4"></share><ci id="S2.E2.m1.4.4.6.cmml" xref="S2.E2.m1.4.4.6">ℝ</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E2.m1.4c">\phi(i,j,t)=\sigma({\bm{S}}_{ij}^{T}{\bm{q}}_{t})\in\mathbb{R}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(2)</span></td>
</tr></tbody>
</table>
</div>
<div id="S2.SS1.SSS0.Px4.p3" class="ltx_para">
<p id="S2.SS1.SSS0.Px4.p3.4" class="ltx_p">In this equation, <math id="S2.SS1.SSS0.Px4.p3.1.m1.1" class="ltx_Math" alttext="\sigma" display="inline"><semantics id="S2.SS1.SSS0.Px4.p3.1.m1.1a"><mi id="S2.SS1.SSS0.Px4.p3.1.m1.1.1" xref="S2.SS1.SSS0.Px4.p3.1.m1.1.1.cmml">σ</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.SSS0.Px4.p3.1.m1.1b"><ci id="S2.SS1.SSS0.Px4.p3.1.m1.1.1.cmml" xref="S2.SS1.SSS0.Px4.p3.1.m1.1.1">𝜎</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.SSS0.Px4.p3.1.m1.1c">\sigma</annotation></semantics></math> denotes a sigmoid activation function. As we train with binary cross-entropy loss (see next sec. <a href="#S2.SS2" title="2.2 Training ‣ 2 Method ‣ GLiNER: Generalist Model for Named Entity Recognition using Bidirectional Transformer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2.2</span></a>), <math id="S2.SS1.SSS0.Px4.p3.2.m2.3" class="ltx_Math" alttext="\phi(i,j,t)" display="inline"><semantics id="S2.SS1.SSS0.Px4.p3.2.m2.3a"><mrow id="S2.SS1.SSS0.Px4.p3.2.m2.3.4" xref="S2.SS1.SSS0.Px4.p3.2.m2.3.4.cmml"><mi id="S2.SS1.SSS0.Px4.p3.2.m2.3.4.2" xref="S2.SS1.SSS0.Px4.p3.2.m2.3.4.2.cmml">ϕ</mi><mo lspace="0em" rspace="0em" id="S2.SS1.SSS0.Px4.p3.2.m2.3.4.1" xref="S2.SS1.SSS0.Px4.p3.2.m2.3.4.1.cmml">​</mo><mrow id="S2.SS1.SSS0.Px4.p3.2.m2.3.4.3.2" xref="S2.SS1.SSS0.Px4.p3.2.m2.3.4.3.1.cmml"><mo stretchy="false" id="S2.SS1.SSS0.Px4.p3.2.m2.3.4.3.2.1" xref="S2.SS1.SSS0.Px4.p3.2.m2.3.4.3.1.cmml">(</mo><mi id="S2.SS1.SSS0.Px4.p3.2.m2.1.1" xref="S2.SS1.SSS0.Px4.p3.2.m2.1.1.cmml">i</mi><mo id="S2.SS1.SSS0.Px4.p3.2.m2.3.4.3.2.2" xref="S2.SS1.SSS0.Px4.p3.2.m2.3.4.3.1.cmml">,</mo><mi id="S2.SS1.SSS0.Px4.p3.2.m2.2.2" xref="S2.SS1.SSS0.Px4.p3.2.m2.2.2.cmml">j</mi><mo id="S2.SS1.SSS0.Px4.p3.2.m2.3.4.3.2.3" xref="S2.SS1.SSS0.Px4.p3.2.m2.3.4.3.1.cmml">,</mo><mi id="S2.SS1.SSS0.Px4.p3.2.m2.3.3" xref="S2.SS1.SSS0.Px4.p3.2.m2.3.3.cmml">t</mi><mo stretchy="false" id="S2.SS1.SSS0.Px4.p3.2.m2.3.4.3.2.4" xref="S2.SS1.SSS0.Px4.p3.2.m2.3.4.3.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS1.SSS0.Px4.p3.2.m2.3b"><apply id="S2.SS1.SSS0.Px4.p3.2.m2.3.4.cmml" xref="S2.SS1.SSS0.Px4.p3.2.m2.3.4"><times id="S2.SS1.SSS0.Px4.p3.2.m2.3.4.1.cmml" xref="S2.SS1.SSS0.Px4.p3.2.m2.3.4.1"></times><ci id="S2.SS1.SSS0.Px4.p3.2.m2.3.4.2.cmml" xref="S2.SS1.SSS0.Px4.p3.2.m2.3.4.2">italic-ϕ</ci><vector id="S2.SS1.SSS0.Px4.p3.2.m2.3.4.3.1.cmml" xref="S2.SS1.SSS0.Px4.p3.2.m2.3.4.3.2"><ci id="S2.SS1.SSS0.Px4.p3.2.m2.1.1.cmml" xref="S2.SS1.SSS0.Px4.p3.2.m2.1.1">𝑖</ci><ci id="S2.SS1.SSS0.Px4.p3.2.m2.2.2.cmml" xref="S2.SS1.SSS0.Px4.p3.2.m2.2.2">𝑗</ci><ci id="S2.SS1.SSS0.Px4.p3.2.m2.3.3.cmml" xref="S2.SS1.SSS0.Px4.p3.2.m2.3.3">𝑡</ci></vector></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.SSS0.Px4.p3.2.m2.3c">\phi(i,j,t)</annotation></semantics></math> can be interpreted as the probability of the span <math id="S2.SS1.SSS0.Px4.p3.3.m3.2" class="ltx_Math" alttext="(i,j)" display="inline"><semantics id="S2.SS1.SSS0.Px4.p3.3.m3.2a"><mrow id="S2.SS1.SSS0.Px4.p3.3.m3.2.3.2" xref="S2.SS1.SSS0.Px4.p3.3.m3.2.3.1.cmml"><mo stretchy="false" id="S2.SS1.SSS0.Px4.p3.3.m3.2.3.2.1" xref="S2.SS1.SSS0.Px4.p3.3.m3.2.3.1.cmml">(</mo><mi id="S2.SS1.SSS0.Px4.p3.3.m3.1.1" xref="S2.SS1.SSS0.Px4.p3.3.m3.1.1.cmml">i</mi><mo id="S2.SS1.SSS0.Px4.p3.3.m3.2.3.2.2" xref="S2.SS1.SSS0.Px4.p3.3.m3.2.3.1.cmml">,</mo><mi id="S2.SS1.SSS0.Px4.p3.3.m3.2.2" xref="S2.SS1.SSS0.Px4.p3.3.m3.2.2.cmml">j</mi><mo stretchy="false" id="S2.SS1.SSS0.Px4.p3.3.m3.2.3.2.3" xref="S2.SS1.SSS0.Px4.p3.3.m3.2.3.1.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.SS1.SSS0.Px4.p3.3.m3.2b"><interval closure="open" id="S2.SS1.SSS0.Px4.p3.3.m3.2.3.1.cmml" xref="S2.SS1.SSS0.Px4.p3.3.m3.2.3.2"><ci id="S2.SS1.SSS0.Px4.p3.3.m3.1.1.cmml" xref="S2.SS1.SSS0.Px4.p3.3.m3.1.1">𝑖</ci><ci id="S2.SS1.SSS0.Px4.p3.3.m3.2.2.cmml" xref="S2.SS1.SSS0.Px4.p3.3.m3.2.2">𝑗</ci></interval></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.SSS0.Px4.p3.3.m3.2c">(i,j)</annotation></semantics></math> being of type <math id="S2.SS1.SSS0.Px4.p3.4.m4.1" class="ltx_Math" alttext="t" display="inline"><semantics id="S2.SS1.SSS0.Px4.p3.4.m4.1a"><mi id="S2.SS1.SSS0.Px4.p3.4.m4.1.1" xref="S2.SS1.SSS0.Px4.p3.4.m4.1.1.cmml">t</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.SSS0.Px4.p3.4.m4.1b"><ci id="S2.SS1.SSS0.Px4.p3.4.m4.1.1.cmml" xref="S2.SS1.SSS0.Px4.p3.4.m4.1.1">𝑡</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.SSS0.Px4.p3.4.m4.1c">t</annotation></semantics></math>.</p>
</div>
<figure id="S2.T1" class="ltx_table">
<table id="S2.T1.1" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="S2.T1.1.1.1" class="ltx_tr">
<th id="S2.T1.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r ltx_border_tt">Model</th>
<th id="S2.T1.1.1.1.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r ltx_border_tt">Params</th>
<td id="S2.T1.1.1.1.3" class="ltx_td ltx_align_center ltx_border_tt">Movie</td>
<td id="S2.T1.1.1.1.4" class="ltx_td ltx_align_center ltx_border_tt">Restaurant</td>
<td id="S2.T1.1.1.1.5" class="ltx_td ltx_align_center ltx_border_tt">AI</td>
<td id="S2.T1.1.1.1.6" class="ltx_td ltx_align_center ltx_border_tt">Literature</td>
<td id="S2.T1.1.1.1.7" class="ltx_td ltx_align_center ltx_border_tt">Music</td>
<td id="S2.T1.1.1.1.8" class="ltx_td ltx_align_center ltx_border_tt">Politics</td>
<td id="S2.T1.1.1.1.9" class="ltx_td ltx_align_center ltx_border_tt">Science</td>
<td id="S2.T1.1.1.1.10" class="ltx_td ltx_align_center ltx_border_tt">Average</td>
</tr>
<tr id="S2.T1.1.2.2" class="ltx_tr">
<th id="S2.T1.1.2.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r ltx_border_t">Vicuna-7B</th>
<th id="S2.T1.1.2.2.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r ltx_border_t">7B</th>
<td id="S2.T1.1.2.2.3" class="ltx_td ltx_align_center ltx_border_t">6.0</td>
<td id="S2.T1.1.2.2.4" class="ltx_td ltx_align_center ltx_border_t">5.3</td>
<td id="S2.T1.1.2.2.5" class="ltx_td ltx_align_center ltx_border_t">12.8</td>
<td id="S2.T1.1.2.2.6" class="ltx_td ltx_align_center ltx_border_t">16.1</td>
<td id="S2.T1.1.2.2.7" class="ltx_td ltx_align_center ltx_border_t">17.0</td>
<td id="S2.T1.1.2.2.8" class="ltx_td ltx_align_center ltx_border_t">20.5</td>
<td id="S2.T1.1.2.2.9" class="ltx_td ltx_align_center ltx_border_t">13.0</td>
<td id="S2.T1.1.2.2.10" class="ltx_td ltx_align_center ltx_border_t">13.0</td>
</tr>
<tr id="S2.T1.1.3.3" class="ltx_tr">
<th id="S2.T1.1.3.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">Vicuna-13B</th>
<th id="S2.T1.1.3.3.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">13B</th>
<td id="S2.T1.1.3.3.3" class="ltx_td ltx_align_center">0.9</td>
<td id="S2.T1.1.3.3.4" class="ltx_td ltx_align_center">0.4</td>
<td id="S2.T1.1.3.3.5" class="ltx_td ltx_align_center">22.7</td>
<td id="S2.T1.1.3.3.6" class="ltx_td ltx_align_center">22.7</td>
<td id="S2.T1.1.3.3.7" class="ltx_td ltx_align_center">26.6</td>
<td id="S2.T1.1.3.3.8" class="ltx_td ltx_align_center">27.0</td>
<td id="S2.T1.1.3.3.9" class="ltx_td ltx_align_center">22.0</td>
<td id="S2.T1.1.3.3.10" class="ltx_td ltx_align_center">17.5</td>
</tr>
<tr id="S2.T1.1.4.4" class="ltx_tr">
<th id="S2.T1.1.4.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">USM</th>
<th id="S2.T1.1.4.4.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">0.3B</th>
<td id="S2.T1.1.4.4.3" class="ltx_td ltx_align_center">37.7</td>
<td id="S2.T1.1.4.4.4" class="ltx_td ltx_align_center">17.7</td>
<td id="S2.T1.1.4.4.5" class="ltx_td ltx_align_center">28.2</td>
<td id="S2.T1.1.4.4.6" class="ltx_td ltx_align_center">56.0</td>
<td id="S2.T1.1.4.4.7" class="ltx_td ltx_align_center">44.9</td>
<td id="S2.T1.1.4.4.8" class="ltx_td ltx_align_center">36.1</td>
<td id="S2.T1.1.4.4.9" class="ltx_td ltx_align_center">44.0</td>
<td id="S2.T1.1.4.4.10" class="ltx_td ltx_align_center">37.8</td>
</tr>
<tr id="S2.T1.1.5.5" class="ltx_tr">
<th id="S2.T1.1.5.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">ChatGPT</th>
<th id="S2.T1.1.5.5.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">–</th>
<td id="S2.T1.1.5.5.3" class="ltx_td ltx_align_center">5.3</td>
<td id="S2.T1.1.5.5.4" class="ltx_td ltx_align_center">32.8</td>
<td id="S2.T1.1.5.5.5" class="ltx_td ltx_align_center">52.4</td>
<td id="S2.T1.1.5.5.6" class="ltx_td ltx_align_center">39.8</td>
<td id="S2.T1.1.5.5.7" class="ltx_td ltx_align_center">66.6</td>
<td id="S2.T1.1.5.5.8" class="ltx_td ltx_align_center">68.5</td>
<td id="S2.T1.1.5.5.9" class="ltx_td ltx_align_center"><span id="S2.T1.1.5.5.9.1" class="ltx_text ltx_font_bold">67.0</span></td>
<td id="S2.T1.1.5.5.10" class="ltx_td ltx_align_center">47.5</td>
</tr>
<tr id="S2.T1.1.6.6" class="ltx_tr">
<th id="S2.T1.1.6.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">InstructUIE</th>
<th id="S2.T1.1.6.6.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">11B</th>
<td id="S2.T1.1.6.6.3" class="ltx_td ltx_align_center"><span id="S2.T1.1.6.6.3.1" class="ltx_text ltx_font_bold">63.0</span></td>
<td id="S2.T1.1.6.6.4" class="ltx_td ltx_align_center">21.0</td>
<td id="S2.T1.1.6.6.5" class="ltx_td ltx_align_center">49.0</td>
<td id="S2.T1.1.6.6.6" class="ltx_td ltx_align_center">47.2</td>
<td id="S2.T1.1.6.6.7" class="ltx_td ltx_align_center">53.2</td>
<td id="S2.T1.1.6.6.8" class="ltx_td ltx_align_center">48.1</td>
<td id="S2.T1.1.6.6.9" class="ltx_td ltx_align_center">49.2</td>
<td id="S2.T1.1.6.6.10" class="ltx_td ltx_align_center">47.2</td>
</tr>
<tr id="S2.T1.1.7.7" class="ltx_tr">
<th id="S2.T1.1.7.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">UniNER-7B</th>
<th id="S2.T1.1.7.7.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">7B</th>
<td id="S2.T1.1.7.7.3" class="ltx_td ltx_align_center">42.4</td>
<td id="S2.T1.1.7.7.4" class="ltx_td ltx_align_center">31.7</td>
<td id="S2.T1.1.7.7.5" class="ltx_td ltx_align_center">53.6</td>
<td id="S2.T1.1.7.7.6" class="ltx_td ltx_align_center">59.3</td>
<td id="S2.T1.1.7.7.7" class="ltx_td ltx_align_center">67.0</td>
<td id="S2.T1.1.7.7.8" class="ltx_td ltx_align_center">60.9</td>
<td id="S2.T1.1.7.7.9" class="ltx_td ltx_align_center">61.1</td>
<td id="S2.T1.1.7.7.10" class="ltx_td ltx_align_center">53.7</td>
</tr>
<tr id="S2.T1.1.8.8" class="ltx_tr">
<th id="S2.T1.1.8.8.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">UniNER-13B</th>
<th id="S2.T1.1.8.8.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">13B</th>
<td id="S2.T1.1.8.8.3" class="ltx_td ltx_align_center">48.7</td>
<td id="S2.T1.1.8.8.4" class="ltx_td ltx_align_center">36.2</td>
<td id="S2.T1.1.8.8.5" class="ltx_td ltx_align_center">54.2</td>
<td id="S2.T1.1.8.8.6" class="ltx_td ltx_align_center">60.9</td>
<td id="S2.T1.1.8.8.7" class="ltx_td ltx_align_center">64.5</td>
<td id="S2.T1.1.8.8.8" class="ltx_td ltx_align_center">61.4</td>
<td id="S2.T1.1.8.8.9" class="ltx_td ltx_align_center">63.5</td>
<td id="S2.T1.1.8.8.10" class="ltx_td ltx_align_center">55.6</td>
</tr>
<tr id="S2.T1.1.9.9" class="ltx_tr">
<th id="S2.T1.1.9.9.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">GoLLIE</th>
<th id="S2.T1.1.9.9.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">7B</th>
<td id="S2.T1.1.9.9.3" class="ltx_td ltx_align_center"><span id="S2.T1.1.9.9.3.1" class="ltx_text ltx_font_bold">63.0</span></td>
<td id="S2.T1.1.9.9.4" class="ltx_td ltx_align_center"><span id="S2.T1.1.9.9.4.1" class="ltx_text ltx_font_bold">43.4</span></td>
<td id="S2.T1.1.9.9.5" class="ltx_td ltx_align_center"><span id="S2.T1.1.9.9.5.1" class="ltx_text ltx_font_bold">59.1</span></td>
<td id="S2.T1.1.9.9.6" class="ltx_td ltx_align_center">62.7</td>
<td id="S2.T1.1.9.9.7" class="ltx_td ltx_align_center">67.8</td>
<td id="S2.T1.1.9.9.8" class="ltx_td ltx_align_center">57.2</td>
<td id="S2.T1.1.9.9.9" class="ltx_td ltx_align_center">55.5</td>
<td id="S2.T1.1.9.9.10" class="ltx_td ltx_align_center">58.0</td>
</tr>
<tr id="S2.T1.1.10.10" class="ltx_tr">
<th id="S2.T1.1.10.10.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r ltx_border_t">GLiNER-S</th>
<th id="S2.T1.1.10.10.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r ltx_border_t">50M</th>
<td id="S2.T1.1.10.10.3" class="ltx_td ltx_align_center ltx_border_t">46.9</td>
<td id="S2.T1.1.10.10.4" class="ltx_td ltx_align_center ltx_border_t">33.3</td>
<td id="S2.T1.1.10.10.5" class="ltx_td ltx_align_center ltx_border_t">50.7</td>
<td id="S2.T1.1.10.10.6" class="ltx_td ltx_align_center ltx_border_t">60.0</td>
<td id="S2.T1.1.10.10.7" class="ltx_td ltx_align_center ltx_border_t">60.9</td>
<td id="S2.T1.1.10.10.8" class="ltx_td ltx_align_center ltx_border_t">61.5</td>
<td id="S2.T1.1.10.10.9" class="ltx_td ltx_align_center ltx_border_t">55.6</td>
<td id="S2.T1.1.10.10.10" class="ltx_td ltx_align_center ltx_border_t">52.7</td>
</tr>
<tr id="S2.T1.1.11.11" class="ltx_tr">
<th id="S2.T1.1.11.11.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">GLiNER-M</th>
<th id="S2.T1.1.11.11.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">90M</th>
<td id="S2.T1.1.11.11.3" class="ltx_td ltx_align_center">42.9</td>
<td id="S2.T1.1.11.11.4" class="ltx_td ltx_align_center">37.3</td>
<td id="S2.T1.1.11.11.5" class="ltx_td ltx_align_center">51.8</td>
<td id="S2.T1.1.11.11.6" class="ltx_td ltx_align_center">59.7</td>
<td id="S2.T1.1.11.11.7" class="ltx_td ltx_align_center">69.4</td>
<td id="S2.T1.1.11.11.8" class="ltx_td ltx_align_center">68.6</td>
<td id="S2.T1.1.11.11.9" class="ltx_td ltx_align_center">58.1</td>
<td id="S2.T1.1.11.11.10" class="ltx_td ltx_align_center">55.4</td>
</tr>
<tr id="S2.T1.1.12.12" class="ltx_tr">
<th id="S2.T1.1.12.12.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb ltx_border_r">GLiNER-L</th>
<th id="S2.T1.1.12.12.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb ltx_border_r">0.3B</th>
<td id="S2.T1.1.12.12.3" class="ltx_td ltx_align_center ltx_border_bb">57.2</td>
<td id="S2.T1.1.12.12.4" class="ltx_td ltx_align_center ltx_border_bb">42.9</td>
<td id="S2.T1.1.12.12.5" class="ltx_td ltx_align_center ltx_border_bb">57.2</td>
<td id="S2.T1.1.12.12.6" class="ltx_td ltx_align_center ltx_border_bb"><span id="S2.T1.1.12.12.6.1" class="ltx_text ltx_font_bold">64.4</span></td>
<td id="S2.T1.1.12.12.7" class="ltx_td ltx_align_center ltx_border_bb"><span id="S2.T1.1.12.12.7.1" class="ltx_text ltx_font_bold">69.6</span></td>
<td id="S2.T1.1.12.12.8" class="ltx_td ltx_align_center ltx_border_bb"><span id="S2.T1.1.12.12.8.1" class="ltx_text ltx_font_bold">72.6</span></td>
<td id="S2.T1.1.12.12.9" class="ltx_td ltx_align_center ltx_border_bb">62.6</td>
<td id="S2.T1.1.12.12.10" class="ltx_td ltx_align_center ltx_border_bb"><span id="S2.T1.1.12.12.10.1" class="ltx_text ltx_font_bold">60.9</span></td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 1: </span><span id="S2.T1.3.1" class="ltx_text ltx_font_bold">Zero-Shot Scores on Out-of-Domain NER Benchmark.</span> We report the performance of GLiNER with various DeBERTa-v3 <cite class="ltx_cite ltx_citemacro_citep">(He et al., <a href="#bib.bib13" title="" class="ltx_ref">2021</a>)</cite> model sizes. Results for Vicuna, ChatGPT, and UniNER are from <cite class="ltx_cite ltx_citemacro_citet">Zhou et al. (<a href="#bib.bib50" title="" class="ltx_ref">2023</a>)</cite>; USM and InstructUIE are from <cite class="ltx_cite ltx_citemacro_citet">Wang et al. (<a href="#bib.bib40" title="" class="ltx_ref">2023</a>)</cite>; and GoLLIE is from <cite class="ltx_cite ltx_citemacro_citet">Sainz et al. (<a href="#bib.bib37" title="" class="ltx_ref">2023</a>)</cite>.</figcaption>
</figure>
</section>
</section>
<section id="S2.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.2 </span>Training </h3>

<div id="S2.SS2.p1" class="ltx_para">
<p id="S2.SS2.p1.7" class="ltx_p">During training, our objective is to optimize model parameters to enhance the matching score for correct span-type pairs (positive pairs) and reduce it for incorrect pairs (negative pairs). A span <math id="S2.SS2.p1.1.m1.2" class="ltx_Math" alttext="(i,j)" display="inline"><semantics id="S2.SS2.p1.1.m1.2a"><mrow id="S2.SS2.p1.1.m1.2.3.2" xref="S2.SS2.p1.1.m1.2.3.1.cmml"><mo stretchy="false" id="S2.SS2.p1.1.m1.2.3.2.1" xref="S2.SS2.p1.1.m1.2.3.1.cmml">(</mo><mi id="S2.SS2.p1.1.m1.1.1" xref="S2.SS2.p1.1.m1.1.1.cmml">i</mi><mo id="S2.SS2.p1.1.m1.2.3.2.2" xref="S2.SS2.p1.1.m1.2.3.1.cmml">,</mo><mi id="S2.SS2.p1.1.m1.2.2" xref="S2.SS2.p1.1.m1.2.2.cmml">j</mi><mo stretchy="false" id="S2.SS2.p1.1.m1.2.3.2.3" xref="S2.SS2.p1.1.m1.2.3.1.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.1.m1.2b"><interval closure="open" id="S2.SS2.p1.1.m1.2.3.1.cmml" xref="S2.SS2.p1.1.m1.2.3.2"><ci id="S2.SS2.p1.1.m1.1.1.cmml" xref="S2.SS2.p1.1.m1.1.1">𝑖</ci><ci id="S2.SS2.p1.1.m1.2.2.cmml" xref="S2.SS2.p1.1.m1.2.2">𝑗</ci></interval></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.1.m1.2c">(i,j)</annotation></semantics></math> paired with an entity type <math id="S2.SS2.p1.2.m2.1" class="ltx_Math" alttext="t" display="inline"><semantics id="S2.SS2.p1.2.m2.1a"><mi id="S2.SS2.p1.2.m2.1.1" xref="S2.SS2.p1.2.m2.1.1.cmml">t</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.2.m2.1b"><ci id="S2.SS2.p1.2.m2.1.1.cmml" xref="S2.SS2.p1.2.m2.1.1">𝑡</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.2.m2.1c">t</annotation></semantics></math> forms a positive pair (<math id="S2.SS2.p1.3.m3.1" class="ltx_Math" alttext="s\in\mathcal{P}" display="inline"><semantics id="S2.SS2.p1.3.m3.1a"><mrow id="S2.SS2.p1.3.m3.1.1" xref="S2.SS2.p1.3.m3.1.1.cmml"><mi id="S2.SS2.p1.3.m3.1.1.2" xref="S2.SS2.p1.3.m3.1.1.2.cmml">s</mi><mo id="S2.SS2.p1.3.m3.1.1.1" xref="S2.SS2.p1.3.m3.1.1.1.cmml">∈</mo><mi class="ltx_font_mathcaligraphic" id="S2.SS2.p1.3.m3.1.1.3" xref="S2.SS2.p1.3.m3.1.1.3.cmml">𝒫</mi></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.3.m3.1b"><apply id="S2.SS2.p1.3.m3.1.1.cmml" xref="S2.SS2.p1.3.m3.1.1"><in id="S2.SS2.p1.3.m3.1.1.1.cmml" xref="S2.SS2.p1.3.m3.1.1.1"></in><ci id="S2.SS2.p1.3.m3.1.1.2.cmml" xref="S2.SS2.p1.3.m3.1.1.2">𝑠</ci><ci id="S2.SS2.p1.3.m3.1.1.3.cmml" xref="S2.SS2.p1.3.m3.1.1.3">𝒫</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.3.m3.1c">s\in\mathcal{P}</annotation></semantics></math>) if the span is labeled with type <math id="S2.SS2.p1.4.m4.1" class="ltx_Math" alttext="t" display="inline"><semantics id="S2.SS2.p1.4.m4.1a"><mi id="S2.SS2.p1.4.m4.1.1" xref="S2.SS2.p1.4.m4.1.1.cmml">t</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.4.m4.1b"><ci id="S2.SS2.p1.4.m4.1.1.cmml" xref="S2.SS2.p1.4.m4.1.1">𝑡</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.4.m4.1c">t</annotation></semantics></math> in the training data. Otherwise, it is a negative pair (<math id="S2.SS2.p1.5.m5.1" class="ltx_Math" alttext="s\in\mathcal{N}" display="inline"><semantics id="S2.SS2.p1.5.m5.1a"><mrow id="S2.SS2.p1.5.m5.1.1" xref="S2.SS2.p1.5.m5.1.1.cmml"><mi id="S2.SS2.p1.5.m5.1.1.2" xref="S2.SS2.p1.5.m5.1.1.2.cmml">s</mi><mo id="S2.SS2.p1.5.m5.1.1.1" xref="S2.SS2.p1.5.m5.1.1.1.cmml">∈</mo><mi class="ltx_font_mathcaligraphic" id="S2.SS2.p1.5.m5.1.1.3" xref="S2.SS2.p1.5.m5.1.1.3.cmml">𝒩</mi></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.5.m5.1b"><apply id="S2.SS2.p1.5.m5.1.1.cmml" xref="S2.SS2.p1.5.m5.1.1"><in id="S2.SS2.p1.5.m5.1.1.1.cmml" xref="S2.SS2.p1.5.m5.1.1.1"></in><ci id="S2.SS2.p1.5.m5.1.1.2.cmml" xref="S2.SS2.p1.5.m5.1.1.2">𝑠</ci><ci id="S2.SS2.p1.5.m5.1.1.3.cmml" xref="S2.SS2.p1.5.m5.1.1.3">𝒩</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.5.m5.1c">s\in\mathcal{N}</annotation></semantics></math>). The training loss for an individual example, comprising spans <math id="S2.SS2.p1.6.m6.1" class="ltx_Math" alttext="\mathcal{S}" display="inline"><semantics id="S2.SS2.p1.6.m6.1a"><mi class="ltx_font_mathcaligraphic" id="S2.SS2.p1.6.m6.1.1" xref="S2.SS2.p1.6.m6.1.1.cmml">𝒮</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.6.m6.1b"><ci id="S2.SS2.p1.6.m6.1.1.cmml" xref="S2.SS2.p1.6.m6.1.1">𝒮</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.6.m6.1c">\mathcal{S}</annotation></semantics></math> and entity types <math id="S2.SS2.p1.7.m7.1" class="ltx_Math" alttext="\mathcal{T}" display="inline"><semantics id="S2.SS2.p1.7.m7.1a"><mi class="ltx_font_mathcaligraphic" id="S2.SS2.p1.7.m7.1.1" xref="S2.SS2.p1.7.m7.1.1.cmml">𝒯</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.7.m7.1b"><ci id="S2.SS2.p1.7.m7.1.1.cmml" xref="S2.SS2.p1.7.m7.1.1">𝒯</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.7.m7.1c">\mathcal{T}</annotation></semantics></math>, is defined as:</p>
</div>
<div id="S2.SS2.p2" class="ltx_para">
<table id="S2.E3" class="ltx_equationgroup ltx_eqn_table">
<tbody>
<tr id="S2.E3X" class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S2.E3X.2.1.1.m1.1" class="ltx_Math" alttext="\displaystyle\mathcal{L}_{\text{BCE}}=-\sum_{s\in\mathcal{S}\times\mathcal{T}}" display="inline"><semantics id="S2.E3X.2.1.1.m1.1a"><mrow id="S2.E3X.2.1.1.m1.1.1" xref="S2.E3X.2.1.1.m1.1.1.cmml"><msub id="S2.E3X.2.1.1.m1.1.1.2" xref="S2.E3X.2.1.1.m1.1.1.2.cmml"><mi class="ltx_font_mathcaligraphic" id="S2.E3X.2.1.1.m1.1.1.2.2" xref="S2.E3X.2.1.1.m1.1.1.2.2.cmml">ℒ</mi><mtext id="S2.E3X.2.1.1.m1.1.1.2.3" xref="S2.E3X.2.1.1.m1.1.1.2.3a.cmml">BCE</mtext></msub><mo id="S2.E3X.2.1.1.m1.1.1.1" xref="S2.E3X.2.1.1.m1.1.1.1.cmml">=</mo><mrow id="S2.E3X.2.1.1.m1.1.1.3" xref="S2.E3X.2.1.1.m1.1.1.3.cmml"><mo id="S2.E3X.2.1.1.m1.1.1.3a" xref="S2.E3X.2.1.1.m1.1.1.3.cmml">−</mo><mstyle displaystyle="true" id="S2.E3X.2.1.1.m1.1.1.3.2" xref="S2.E3X.2.1.1.m1.1.1.3.2.cmml"><munder id="S2.E3X.2.1.1.m1.1.1.3.2a" xref="S2.E3X.2.1.1.m1.1.1.3.2.cmml"><mo movablelimits="false" id="S2.E3X.2.1.1.m1.1.1.3.2.2" xref="S2.E3X.2.1.1.m1.1.1.3.2.2.cmml">∑</mo><mrow id="S2.E3X.2.1.1.m1.1.1.3.2.3" xref="S2.E3X.2.1.1.m1.1.1.3.2.3.cmml"><mi id="S2.E3X.2.1.1.m1.1.1.3.2.3.2" xref="S2.E3X.2.1.1.m1.1.1.3.2.3.2.cmml">s</mi><mo id="S2.E3X.2.1.1.m1.1.1.3.2.3.1" xref="S2.E3X.2.1.1.m1.1.1.3.2.3.1.cmml">∈</mo><mrow id="S2.E3X.2.1.1.m1.1.1.3.2.3.3" xref="S2.E3X.2.1.1.m1.1.1.3.2.3.3.cmml"><mi class="ltx_font_mathcaligraphic" id="S2.E3X.2.1.1.m1.1.1.3.2.3.3.2" xref="S2.E3X.2.1.1.m1.1.1.3.2.3.3.2.cmml">𝒮</mi><mo lspace="0.222em" rspace="0.222em" id="S2.E3X.2.1.1.m1.1.1.3.2.3.3.1" xref="S2.E3X.2.1.1.m1.1.1.3.2.3.3.1.cmml">×</mo><mi class="ltx_font_mathcaligraphic" id="S2.E3X.2.1.1.m1.1.1.3.2.3.3.3" xref="S2.E3X.2.1.1.m1.1.1.3.2.3.3.3.cmml">𝒯</mi></mrow></mrow></munder></mstyle></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.E3X.2.1.1.m1.1b"><apply id="S2.E3X.2.1.1.m1.1.1.cmml" xref="S2.E3X.2.1.1.m1.1.1"><eq id="S2.E3X.2.1.1.m1.1.1.1.cmml" xref="S2.E3X.2.1.1.m1.1.1.1"></eq><apply id="S2.E3X.2.1.1.m1.1.1.2.cmml" xref="S2.E3X.2.1.1.m1.1.1.2"><csymbol cd="ambiguous" id="S2.E3X.2.1.1.m1.1.1.2.1.cmml" xref="S2.E3X.2.1.1.m1.1.1.2">subscript</csymbol><ci id="S2.E3X.2.1.1.m1.1.1.2.2.cmml" xref="S2.E3X.2.1.1.m1.1.1.2.2">ℒ</ci><ci id="S2.E3X.2.1.1.m1.1.1.2.3a.cmml" xref="S2.E3X.2.1.1.m1.1.1.2.3"><mtext mathsize="70%" id="S2.E3X.2.1.1.m1.1.1.2.3.cmml" xref="S2.E3X.2.1.1.m1.1.1.2.3">BCE</mtext></ci></apply><apply id="S2.E3X.2.1.1.m1.1.1.3.cmml" xref="S2.E3X.2.1.1.m1.1.1.3"><minus id="S2.E3X.2.1.1.m1.1.1.3.1.cmml" xref="S2.E3X.2.1.1.m1.1.1.3"></minus><apply id="S2.E3X.2.1.1.m1.1.1.3.2.cmml" xref="S2.E3X.2.1.1.m1.1.1.3.2"><csymbol cd="ambiguous" id="S2.E3X.2.1.1.m1.1.1.3.2.1.cmml" xref="S2.E3X.2.1.1.m1.1.1.3.2">subscript</csymbol><sum id="S2.E3X.2.1.1.m1.1.1.3.2.2.cmml" xref="S2.E3X.2.1.1.m1.1.1.3.2.2"></sum><apply id="S2.E3X.2.1.1.m1.1.1.3.2.3.cmml" xref="S2.E3X.2.1.1.m1.1.1.3.2.3"><in id="S2.E3X.2.1.1.m1.1.1.3.2.3.1.cmml" xref="S2.E3X.2.1.1.m1.1.1.3.2.3.1"></in><ci id="S2.E3X.2.1.1.m1.1.1.3.2.3.2.cmml" xref="S2.E3X.2.1.1.m1.1.1.3.2.3.2">𝑠</ci><apply id="S2.E3X.2.1.1.m1.1.1.3.2.3.3.cmml" xref="S2.E3X.2.1.1.m1.1.1.3.2.3.3"><times id="S2.E3X.2.1.1.m1.1.1.3.2.3.3.1.cmml" xref="S2.E3X.2.1.1.m1.1.1.3.2.3.3.1"></times><ci id="S2.E3X.2.1.1.m1.1.1.3.2.3.3.2.cmml" xref="S2.E3X.2.1.1.m1.1.1.3.2.3.3.2">𝒮</ci><ci id="S2.E3X.2.1.1.m1.1.1.3.2.3.3.3.cmml" xref="S2.E3X.2.1.1.m1.1.1.3.2.3.3.3">𝒯</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E3X.2.1.1.m1.1c">\displaystyle\mathcal{L}_{\text{BCE}}=-\sum_{s\in\mathcal{S}\times\mathcal{T}}</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S2.E3X.3.2.2.m1.1" class="ltx_Math" alttext="\displaystyle\mathbb{I}_{s\in\mathcal{P}}\log\phi(s)+" display="inline"><semantics id="S2.E3X.3.2.2.m1.1a"><mrow id="S2.E3X.3.2.2.m1.1.2" xref="S2.E3X.3.2.2.m1.1.2.cmml"><mrow id="S2.E3X.3.2.2.m1.1.2.2" xref="S2.E3X.3.2.2.m1.1.2.2.cmml"><msub id="S2.E3X.3.2.2.m1.1.2.2.2" xref="S2.E3X.3.2.2.m1.1.2.2.2.cmml"><mi id="S2.E3X.3.2.2.m1.1.2.2.2.2" xref="S2.E3X.3.2.2.m1.1.2.2.2.2.cmml">𝕀</mi><mrow id="S2.E3X.3.2.2.m1.1.2.2.2.3" xref="S2.E3X.3.2.2.m1.1.2.2.2.3.cmml"><mi id="S2.E3X.3.2.2.m1.1.2.2.2.3.2" xref="S2.E3X.3.2.2.m1.1.2.2.2.3.2.cmml">s</mi><mo id="S2.E3X.3.2.2.m1.1.2.2.2.3.1" xref="S2.E3X.3.2.2.m1.1.2.2.2.3.1.cmml">∈</mo><mi class="ltx_font_mathcaligraphic" id="S2.E3X.3.2.2.m1.1.2.2.2.3.3" xref="S2.E3X.3.2.2.m1.1.2.2.2.3.3.cmml">𝒫</mi></mrow></msub><mo lspace="0.167em" rspace="0em" id="S2.E3X.3.2.2.m1.1.2.2.1" xref="S2.E3X.3.2.2.m1.1.2.2.1.cmml">​</mo><mrow id="S2.E3X.3.2.2.m1.1.2.2.3" xref="S2.E3X.3.2.2.m1.1.2.2.3.cmml"><mi id="S2.E3X.3.2.2.m1.1.2.2.3.1" xref="S2.E3X.3.2.2.m1.1.2.2.3.1.cmml">log</mi><mo lspace="0.167em" id="S2.E3X.3.2.2.m1.1.2.2.3a" xref="S2.E3X.3.2.2.m1.1.2.2.3.cmml">⁡</mo><mi id="S2.E3X.3.2.2.m1.1.2.2.3.2" xref="S2.E3X.3.2.2.m1.1.2.2.3.2.cmml">ϕ</mi></mrow><mo lspace="0em" rspace="0em" id="S2.E3X.3.2.2.m1.1.2.2.1a" xref="S2.E3X.3.2.2.m1.1.2.2.1.cmml">​</mo><mrow id="S2.E3X.3.2.2.m1.1.2.2.4.2" xref="S2.E3X.3.2.2.m1.1.2.2.cmml"><mo stretchy="false" id="S2.E3X.3.2.2.m1.1.2.2.4.2.1" xref="S2.E3X.3.2.2.m1.1.2.2.cmml">(</mo><mi id="S2.E3X.3.2.2.m1.1.1" xref="S2.E3X.3.2.2.m1.1.1.cmml">s</mi><mo stretchy="false" id="S2.E3X.3.2.2.m1.1.2.2.4.2.2" xref="S2.E3X.3.2.2.m1.1.2.2.cmml">)</mo></mrow></mrow><mo id="S2.E3X.3.2.2.m1.1.2.3" xref="S2.E3X.3.2.2.m1.1.2.3.cmml">+</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.E3X.3.2.2.m1.1b"><apply id="S2.E3X.3.2.2.m1.1.2.cmml" xref="S2.E3X.3.2.2.m1.1.2"><csymbol cd="latexml" id="S2.E3X.3.2.2.m1.1.2.1.cmml" xref="S2.E3X.3.2.2.m1.1.2">limit-from</csymbol><apply id="S2.E3X.3.2.2.m1.1.2.2.cmml" xref="S2.E3X.3.2.2.m1.1.2.2"><times id="S2.E3X.3.2.2.m1.1.2.2.1.cmml" xref="S2.E3X.3.2.2.m1.1.2.2.1"></times><apply id="S2.E3X.3.2.2.m1.1.2.2.2.cmml" xref="S2.E3X.3.2.2.m1.1.2.2.2"><csymbol cd="ambiguous" id="S2.E3X.3.2.2.m1.1.2.2.2.1.cmml" xref="S2.E3X.3.2.2.m1.1.2.2.2">subscript</csymbol><ci id="S2.E3X.3.2.2.m1.1.2.2.2.2.cmml" xref="S2.E3X.3.2.2.m1.1.2.2.2.2">𝕀</ci><apply id="S2.E3X.3.2.2.m1.1.2.2.2.3.cmml" xref="S2.E3X.3.2.2.m1.1.2.2.2.3"><in id="S2.E3X.3.2.2.m1.1.2.2.2.3.1.cmml" xref="S2.E3X.3.2.2.m1.1.2.2.2.3.1"></in><ci id="S2.E3X.3.2.2.m1.1.2.2.2.3.2.cmml" xref="S2.E3X.3.2.2.m1.1.2.2.2.3.2">𝑠</ci><ci id="S2.E3X.3.2.2.m1.1.2.2.2.3.3.cmml" xref="S2.E3X.3.2.2.m1.1.2.2.2.3.3">𝒫</ci></apply></apply><apply id="S2.E3X.3.2.2.m1.1.2.2.3.cmml" xref="S2.E3X.3.2.2.m1.1.2.2.3"><log id="S2.E3X.3.2.2.m1.1.2.2.3.1.cmml" xref="S2.E3X.3.2.2.m1.1.2.2.3.1"></log><ci id="S2.E3X.3.2.2.m1.1.2.2.3.2.cmml" xref="S2.E3X.3.2.2.m1.1.2.2.3.2">italic-ϕ</ci></apply><ci id="S2.E3X.3.2.2.m1.1.1.cmml" xref="S2.E3X.3.2.2.m1.1.1">𝑠</ci></apply><plus id="S2.E3X.3.2.2.m1.1.2.3.cmml" xref="S2.E3X.3.2.2.m1.1.2.3"></plus></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E3X.3.2.2.m1.1c">\displaystyle\mathbb{I}_{s\in\mathcal{P}}\log\phi(s)+</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="2" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equationgroup ltx_align_right">(3)</span></td>
</tr>
<tr id="S2.E3Xa" class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_eqn_cell"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S2.E3Xa.2.1.1.m1.3" class="ltx_Math" alttext="\displaystyle\mathbb{I}_{s\in\mathcal{N}}\log\left(1-\phi(s)\right)" display="inline"><semantics id="S2.E3Xa.2.1.1.m1.3a"><mrow id="S2.E3Xa.2.1.1.m1.3.3" xref="S2.E3Xa.2.1.1.m1.3.3.cmml"><msub id="S2.E3Xa.2.1.1.m1.3.3.3" xref="S2.E3Xa.2.1.1.m1.3.3.3.cmml"><mi id="S2.E3Xa.2.1.1.m1.3.3.3.2" xref="S2.E3Xa.2.1.1.m1.3.3.3.2.cmml">𝕀</mi><mrow id="S2.E3Xa.2.1.1.m1.3.3.3.3" xref="S2.E3Xa.2.1.1.m1.3.3.3.3.cmml"><mi id="S2.E3Xa.2.1.1.m1.3.3.3.3.2" xref="S2.E3Xa.2.1.1.m1.3.3.3.3.2.cmml">s</mi><mo id="S2.E3Xa.2.1.1.m1.3.3.3.3.1" xref="S2.E3Xa.2.1.1.m1.3.3.3.3.1.cmml">∈</mo><mi class="ltx_font_mathcaligraphic" id="S2.E3Xa.2.1.1.m1.3.3.3.3.3" xref="S2.E3Xa.2.1.1.m1.3.3.3.3.3.cmml">𝒩</mi></mrow></msub><mo lspace="0.167em" rspace="0em" id="S2.E3Xa.2.1.1.m1.3.3.2" xref="S2.E3Xa.2.1.1.m1.3.3.2.cmml">​</mo><mrow id="S2.E3Xa.2.1.1.m1.3.3.1.1" xref="S2.E3Xa.2.1.1.m1.3.3.1.2.cmml"><mi id="S2.E3Xa.2.1.1.m1.2.2" xref="S2.E3Xa.2.1.1.m1.2.2.cmml">log</mi><mo id="S2.E3Xa.2.1.1.m1.3.3.1.1a" xref="S2.E3Xa.2.1.1.m1.3.3.1.2.cmml">⁡</mo><mrow id="S2.E3Xa.2.1.1.m1.3.3.1.1.1" xref="S2.E3Xa.2.1.1.m1.3.3.1.2.cmml"><mo id="S2.E3Xa.2.1.1.m1.3.3.1.1.1.2" xref="S2.E3Xa.2.1.1.m1.3.3.1.2.cmml">(</mo><mrow id="S2.E3Xa.2.1.1.m1.3.3.1.1.1.1" xref="S2.E3Xa.2.1.1.m1.3.3.1.1.1.1.cmml"><mn id="S2.E3Xa.2.1.1.m1.3.3.1.1.1.1.2" xref="S2.E3Xa.2.1.1.m1.3.3.1.1.1.1.2.cmml">1</mn><mo id="S2.E3Xa.2.1.1.m1.3.3.1.1.1.1.1" xref="S2.E3Xa.2.1.1.m1.3.3.1.1.1.1.1.cmml">−</mo><mrow id="S2.E3Xa.2.1.1.m1.3.3.1.1.1.1.3" xref="S2.E3Xa.2.1.1.m1.3.3.1.1.1.1.3.cmml"><mi id="S2.E3Xa.2.1.1.m1.3.3.1.1.1.1.3.2" xref="S2.E3Xa.2.1.1.m1.3.3.1.1.1.1.3.2.cmml">ϕ</mi><mo lspace="0em" rspace="0em" id="S2.E3Xa.2.1.1.m1.3.3.1.1.1.1.3.1" xref="S2.E3Xa.2.1.1.m1.3.3.1.1.1.1.3.1.cmml">​</mo><mrow id="S2.E3Xa.2.1.1.m1.3.3.1.1.1.1.3.3.2" xref="S2.E3Xa.2.1.1.m1.3.3.1.1.1.1.3.cmml"><mo stretchy="false" id="S2.E3Xa.2.1.1.m1.3.3.1.1.1.1.3.3.2.1" xref="S2.E3Xa.2.1.1.m1.3.3.1.1.1.1.3.cmml">(</mo><mi id="S2.E3Xa.2.1.1.m1.1.1" xref="S2.E3Xa.2.1.1.m1.1.1.cmml">s</mi><mo stretchy="false" id="S2.E3Xa.2.1.1.m1.3.3.1.1.1.1.3.3.2.2" xref="S2.E3Xa.2.1.1.m1.3.3.1.1.1.1.3.cmml">)</mo></mrow></mrow></mrow><mo id="S2.E3Xa.2.1.1.m1.3.3.1.1.1.3" xref="S2.E3Xa.2.1.1.m1.3.3.1.2.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.E3Xa.2.1.1.m1.3b"><apply id="S2.E3Xa.2.1.1.m1.3.3.cmml" xref="S2.E3Xa.2.1.1.m1.3.3"><times id="S2.E3Xa.2.1.1.m1.3.3.2.cmml" xref="S2.E3Xa.2.1.1.m1.3.3.2"></times><apply id="S2.E3Xa.2.1.1.m1.3.3.3.cmml" xref="S2.E3Xa.2.1.1.m1.3.3.3"><csymbol cd="ambiguous" id="S2.E3Xa.2.1.1.m1.3.3.3.1.cmml" xref="S2.E3Xa.2.1.1.m1.3.3.3">subscript</csymbol><ci id="S2.E3Xa.2.1.1.m1.3.3.3.2.cmml" xref="S2.E3Xa.2.1.1.m1.3.3.3.2">𝕀</ci><apply id="S2.E3Xa.2.1.1.m1.3.3.3.3.cmml" xref="S2.E3Xa.2.1.1.m1.3.3.3.3"><in id="S2.E3Xa.2.1.1.m1.3.3.3.3.1.cmml" xref="S2.E3Xa.2.1.1.m1.3.3.3.3.1"></in><ci id="S2.E3Xa.2.1.1.m1.3.3.3.3.2.cmml" xref="S2.E3Xa.2.1.1.m1.3.3.3.3.2">𝑠</ci><ci id="S2.E3Xa.2.1.1.m1.3.3.3.3.3.cmml" xref="S2.E3Xa.2.1.1.m1.3.3.3.3.3">𝒩</ci></apply></apply><apply id="S2.E3Xa.2.1.1.m1.3.3.1.2.cmml" xref="S2.E3Xa.2.1.1.m1.3.3.1.1"><log id="S2.E3Xa.2.1.1.m1.2.2.cmml" xref="S2.E3Xa.2.1.1.m1.2.2"></log><apply id="S2.E3Xa.2.1.1.m1.3.3.1.1.1.1.cmml" xref="S2.E3Xa.2.1.1.m1.3.3.1.1.1.1"><minus id="S2.E3Xa.2.1.1.m1.3.3.1.1.1.1.1.cmml" xref="S2.E3Xa.2.1.1.m1.3.3.1.1.1.1.1"></minus><cn type="integer" id="S2.E3Xa.2.1.1.m1.3.3.1.1.1.1.2.cmml" xref="S2.E3Xa.2.1.1.m1.3.3.1.1.1.1.2">1</cn><apply id="S2.E3Xa.2.1.1.m1.3.3.1.1.1.1.3.cmml" xref="S2.E3Xa.2.1.1.m1.3.3.1.1.1.1.3"><times id="S2.E3Xa.2.1.1.m1.3.3.1.1.1.1.3.1.cmml" xref="S2.E3Xa.2.1.1.m1.3.3.1.1.1.1.3.1"></times><ci id="S2.E3Xa.2.1.1.m1.3.3.1.1.1.1.3.2.cmml" xref="S2.E3Xa.2.1.1.m1.3.3.1.1.1.1.3.2">italic-ϕ</ci><ci id="S2.E3Xa.2.1.1.m1.1.1.cmml" xref="S2.E3Xa.2.1.1.m1.1.1">𝑠</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E3Xa.2.1.1.m1.3c">\displaystyle\mathbb{I}_{s\in\mathcal{N}}\log\left(1-\phi(s)\right)</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr>
</tbody>
</table>
</div>
<div id="S2.SS2.p3" class="ltx_para">
<p id="S2.SS2.p3.2" class="ltx_p">The variable <math id="S2.SS2.p3.1.m1.1" class="ltx_Math" alttext="s" display="inline"><semantics id="S2.SS2.p3.1.m1.1a"><mi id="S2.SS2.p3.1.m1.1.1" xref="S2.SS2.p3.1.m1.1.1.cmml">s</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p3.1.m1.1b"><ci id="S2.SS2.p3.1.m1.1.1.cmml" xref="S2.SS2.p3.1.m1.1.1">𝑠</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p3.1.m1.1c">s</annotation></semantics></math> represents a pair of span/entity type and <math id="S2.SS2.p3.2.m2.1" class="ltx_Math" alttext="\mathbb{I}" display="inline"><semantics id="S2.SS2.p3.2.m2.1a"><mi id="S2.SS2.p3.2.m2.1.1" xref="S2.SS2.p3.2.m2.1.1.cmml">𝕀</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p3.2.m2.1b"><ci id="S2.SS2.p3.2.m2.1.1.cmml" xref="S2.SS2.p3.2.m2.1.1">𝕀</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p3.2.m2.1c">\mathbb{I}</annotation></semantics></math> is an indicator function, which returns 1 when the specified condition is true and 0 otherwise. This loss function corresponds to binary cross-entropy.</p>
</div>
</section>
<section id="S2.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.3 </span>Decoding algorithm</h3>

<div id="S2.SS3.p1" class="ltx_para">
<p id="S2.SS3.p1.2" class="ltx_p">In the decoding phase, we employ a greedy span section <cite class="ltx_cite ltx_citemacro_citep">(Zaratiana et al., <a href="#bib.bib49" title="" class="ltx_ref">2022</a>)</cite> that selects entity spans based on matching scores, to ensure task/dataset specific constraints. This strategy is applied independently to each sentence. Only, spans <math id="S2.SS3.p1.1.m1.2" class="ltx_Math" alttext="(i,j)" display="inline"><semantics id="S2.SS3.p1.1.m1.2a"><mrow id="S2.SS3.p1.1.m1.2.3.2" xref="S2.SS3.p1.1.m1.2.3.1.cmml"><mo stretchy="false" id="S2.SS3.p1.1.m1.2.3.2.1" xref="S2.SS3.p1.1.m1.2.3.1.cmml">(</mo><mi id="S2.SS3.p1.1.m1.1.1" xref="S2.SS3.p1.1.m1.1.1.cmml">i</mi><mo id="S2.SS3.p1.1.m1.2.3.2.2" xref="S2.SS3.p1.1.m1.2.3.1.cmml">,</mo><mi id="S2.SS3.p1.1.m1.2.2" xref="S2.SS3.p1.1.m1.2.2.cmml">j</mi><mo stretchy="false" id="S2.SS3.p1.1.m1.2.3.2.3" xref="S2.SS3.p1.1.m1.2.3.1.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.SS3.p1.1.m1.2b"><interval closure="open" id="S2.SS3.p1.1.m1.2.3.1.cmml" xref="S2.SS3.p1.1.m1.2.3.2"><ci id="S2.SS3.p1.1.m1.1.1.cmml" xref="S2.SS3.p1.1.m1.1.1">𝑖</ci><ci id="S2.SS3.p1.1.m1.2.2.cmml" xref="S2.SS3.p1.1.m1.2.2">𝑗</ci></interval></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p1.1.m1.2c">(i,j)</annotation></semantics></math> with matching scores <math id="S2.SS3.p1.2.m2.3" class="ltx_Math" alttext="\phi(i,j,c)&gt;0.5" display="inline"><semantics id="S2.SS3.p1.2.m2.3a"><mrow id="S2.SS3.p1.2.m2.3.4" xref="S2.SS3.p1.2.m2.3.4.cmml"><mrow id="S2.SS3.p1.2.m2.3.4.2" xref="S2.SS3.p1.2.m2.3.4.2.cmml"><mi id="S2.SS3.p1.2.m2.3.4.2.2" xref="S2.SS3.p1.2.m2.3.4.2.2.cmml">ϕ</mi><mo lspace="0em" rspace="0em" id="S2.SS3.p1.2.m2.3.4.2.1" xref="S2.SS3.p1.2.m2.3.4.2.1.cmml">​</mo><mrow id="S2.SS3.p1.2.m2.3.4.2.3.2" xref="S2.SS3.p1.2.m2.3.4.2.3.1.cmml"><mo stretchy="false" id="S2.SS3.p1.2.m2.3.4.2.3.2.1" xref="S2.SS3.p1.2.m2.3.4.2.3.1.cmml">(</mo><mi id="S2.SS3.p1.2.m2.1.1" xref="S2.SS3.p1.2.m2.1.1.cmml">i</mi><mo id="S2.SS3.p1.2.m2.3.4.2.3.2.2" xref="S2.SS3.p1.2.m2.3.4.2.3.1.cmml">,</mo><mi id="S2.SS3.p1.2.m2.2.2" xref="S2.SS3.p1.2.m2.2.2.cmml">j</mi><mo id="S2.SS3.p1.2.m2.3.4.2.3.2.3" xref="S2.SS3.p1.2.m2.3.4.2.3.1.cmml">,</mo><mi id="S2.SS3.p1.2.m2.3.3" xref="S2.SS3.p1.2.m2.3.3.cmml">c</mi><mo stretchy="false" id="S2.SS3.p1.2.m2.3.4.2.3.2.4" xref="S2.SS3.p1.2.m2.3.4.2.3.1.cmml">)</mo></mrow></mrow><mo id="S2.SS3.p1.2.m2.3.4.1" xref="S2.SS3.p1.2.m2.3.4.1.cmml">&gt;</mo><mn id="S2.SS3.p1.2.m2.3.4.3" xref="S2.SS3.p1.2.m2.3.4.3.cmml">0.5</mn></mrow><annotation-xml encoding="MathML-Content" id="S2.SS3.p1.2.m2.3b"><apply id="S2.SS3.p1.2.m2.3.4.cmml" xref="S2.SS3.p1.2.m2.3.4"><gt id="S2.SS3.p1.2.m2.3.4.1.cmml" xref="S2.SS3.p1.2.m2.3.4.1"></gt><apply id="S2.SS3.p1.2.m2.3.4.2.cmml" xref="S2.SS3.p1.2.m2.3.4.2"><times id="S2.SS3.p1.2.m2.3.4.2.1.cmml" xref="S2.SS3.p1.2.m2.3.4.2.1"></times><ci id="S2.SS3.p1.2.m2.3.4.2.2.cmml" xref="S2.SS3.p1.2.m2.3.4.2.2">italic-ϕ</ci><vector id="S2.SS3.p1.2.m2.3.4.2.3.1.cmml" xref="S2.SS3.p1.2.m2.3.4.2.3.2"><ci id="S2.SS3.p1.2.m2.1.1.cmml" xref="S2.SS3.p1.2.m2.1.1">𝑖</ci><ci id="S2.SS3.p1.2.m2.2.2.cmml" xref="S2.SS3.p1.2.m2.2.2">𝑗</ci><ci id="S2.SS3.p1.2.m2.3.3.cmml" xref="S2.SS3.p1.2.m2.3.3">𝑐</ci></vector></apply><cn type="float" id="S2.SS3.p1.2.m2.3.4.3.cmml" xref="S2.SS3.p1.2.m2.3.4.3">0.5</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p1.2.m2.3c">\phi(i,j,c)&gt;0.5</annotation></semantics></math> are considered for selection.</p>
</div>
<section id="S2.SS3.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Flat NER:</h4>

<div id="S2.SS3.SSS0.Px1.p1" class="ltx_para">
<p id="S2.SS3.SSS0.Px1.p1.1" class="ltx_p">The algorithm chooses the highest-scoring non-overlapping span and continues this process until all spans are evaluated.</p>
</div>
</section>
<section id="S2.SS3.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Nested NER:</h4>

<div id="S2.SS3.SSS0.Px2.p1" class="ltx_para">
<p id="S2.SS3.SSS0.Px2.p1.1" class="ltx_p">Similar to Flat NER, but the algorithm allows selection of fully nested spans within other entities while still avoiding partial overlaps.</p>
</div>
</section>
<section id="S2.SS3.SSS0.Px3" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Algorithm Efficiency:</h4>

<div id="S2.SS3.SSS0.Px3.p1" class="ltx_para">
<p id="S2.SS3.SSS0.Px3.p1.2" class="ltx_p">The decoding is implemented using a priority queue for spans, ensuring an <math id="S2.SS3.SSS0.Px3.p1.1.m1.1" class="ltx_Math" alttext="O(n\log n)" display="inline"><semantics id="S2.SS3.SSS0.Px3.p1.1.m1.1a"><mrow id="S2.SS3.SSS0.Px3.p1.1.m1.1.1" xref="S2.SS3.SSS0.Px3.p1.1.m1.1.1.cmml"><mi id="S2.SS3.SSS0.Px3.p1.1.m1.1.1.3" xref="S2.SS3.SSS0.Px3.p1.1.m1.1.1.3.cmml">O</mi><mo lspace="0em" rspace="0em" id="S2.SS3.SSS0.Px3.p1.1.m1.1.1.2" xref="S2.SS3.SSS0.Px3.p1.1.m1.1.1.2.cmml">​</mo><mrow id="S2.SS3.SSS0.Px3.p1.1.m1.1.1.1.1" xref="S2.SS3.SSS0.Px3.p1.1.m1.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.SS3.SSS0.Px3.p1.1.m1.1.1.1.1.2" xref="S2.SS3.SSS0.Px3.p1.1.m1.1.1.1.1.1.cmml">(</mo><mrow id="S2.SS3.SSS0.Px3.p1.1.m1.1.1.1.1.1" xref="S2.SS3.SSS0.Px3.p1.1.m1.1.1.1.1.1.cmml"><mi id="S2.SS3.SSS0.Px3.p1.1.m1.1.1.1.1.1.2" xref="S2.SS3.SSS0.Px3.p1.1.m1.1.1.1.1.1.2.cmml">n</mi><mo lspace="0.167em" rspace="0em" id="S2.SS3.SSS0.Px3.p1.1.m1.1.1.1.1.1.1" xref="S2.SS3.SSS0.Px3.p1.1.m1.1.1.1.1.1.1.cmml">​</mo><mrow id="S2.SS3.SSS0.Px3.p1.1.m1.1.1.1.1.1.3" xref="S2.SS3.SSS0.Px3.p1.1.m1.1.1.1.1.1.3.cmml"><mi id="S2.SS3.SSS0.Px3.p1.1.m1.1.1.1.1.1.3.1" xref="S2.SS3.SSS0.Px3.p1.1.m1.1.1.1.1.1.3.1.cmml">log</mi><mo lspace="0.167em" id="S2.SS3.SSS0.Px3.p1.1.m1.1.1.1.1.1.3a" xref="S2.SS3.SSS0.Px3.p1.1.m1.1.1.1.1.1.3.cmml">⁡</mo><mi id="S2.SS3.SSS0.Px3.p1.1.m1.1.1.1.1.1.3.2" xref="S2.SS3.SSS0.Px3.p1.1.m1.1.1.1.1.1.3.2.cmml">n</mi></mrow></mrow><mo stretchy="false" id="S2.SS3.SSS0.Px3.p1.1.m1.1.1.1.1.3" xref="S2.SS3.SSS0.Px3.p1.1.m1.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS3.SSS0.Px3.p1.1.m1.1b"><apply id="S2.SS3.SSS0.Px3.p1.1.m1.1.1.cmml" xref="S2.SS3.SSS0.Px3.p1.1.m1.1.1"><times id="S2.SS3.SSS0.Px3.p1.1.m1.1.1.2.cmml" xref="S2.SS3.SSS0.Px3.p1.1.m1.1.1.2"></times><ci id="S2.SS3.SSS0.Px3.p1.1.m1.1.1.3.cmml" xref="S2.SS3.SSS0.Px3.p1.1.m1.1.1.3">𝑂</ci><apply id="S2.SS3.SSS0.Px3.p1.1.m1.1.1.1.1.1.cmml" xref="S2.SS3.SSS0.Px3.p1.1.m1.1.1.1.1"><times id="S2.SS3.SSS0.Px3.p1.1.m1.1.1.1.1.1.1.cmml" xref="S2.SS3.SSS0.Px3.p1.1.m1.1.1.1.1.1.1"></times><ci id="S2.SS3.SSS0.Px3.p1.1.m1.1.1.1.1.1.2.cmml" xref="S2.SS3.SSS0.Px3.p1.1.m1.1.1.1.1.1.2">𝑛</ci><apply id="S2.SS3.SSS0.Px3.p1.1.m1.1.1.1.1.1.3.cmml" xref="S2.SS3.SSS0.Px3.p1.1.m1.1.1.1.1.1.3"><log id="S2.SS3.SSS0.Px3.p1.1.m1.1.1.1.1.1.3.1.cmml" xref="S2.SS3.SSS0.Px3.p1.1.m1.1.1.1.1.1.3.1"></log><ci id="S2.SS3.SSS0.Px3.p1.1.m1.1.1.1.1.1.3.2.cmml" xref="S2.SS3.SSS0.Px3.p1.1.m1.1.1.1.1.1.3.2">𝑛</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.SSS0.Px3.p1.1.m1.1c">O(n\log n)</annotation></semantics></math> complexity, with <math id="S2.SS3.SSS0.Px3.p1.2.m2.1" class="ltx_Math" alttext="n" display="inline"><semantics id="S2.SS3.SSS0.Px3.p1.2.m2.1a"><mi id="S2.SS3.SSS0.Px3.p1.2.m2.1.1" xref="S2.SS3.SSS0.Px3.p1.2.m2.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S2.SS3.SSS0.Px3.p1.2.m2.1b"><ci id="S2.SS3.SSS0.Px3.p1.2.m2.1.1.cmml" xref="S2.SS3.SSS0.Px3.p1.2.m2.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.SSS0.Px3.p1.2.m2.1c">n</annotation></semantics></math> being the number of candidate spans.</p>
</div>
</section>
</section>
</section>
<section id="S3" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">3 </span>Experimental Setting</h2>

<section id="S3.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.1 </span>Training data</h3>

<div id="S3.SS1.p1" class="ltx_para">
<p id="S3.SS1.p1.1" class="ltx_p">Our objective is to construct a versatile NER model capable of accurately identifying a wide array of entity types across different textual domains. To achieve this, it is essential that our training dataset encompasses a diverse range of entity types. For this, we utilize the training data released by <cite class="ltx_cite ltx_citemacro_citet">Zhou et al. (<a href="#bib.bib50" title="" class="ltx_ref">2023</a>)</cite>, known as Pile-NER<span id="footnote1" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">1</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">1</sup><span class="ltx_tag ltx_tag_note">1</span>https://huggingface.co/datasets/Universal-NER/Pile-NER-type</span></span></span>. This dataset is derived from the Pile corpus <cite class="ltx_cite ltx_citemacro_citep">(Gao et al., <a href="#bib.bib12" title="" class="ltx_ref">2020</a>)</cite>, commonly used for pretraining large language models, and comprises text from diverse sources. More specifically, <cite class="ltx_cite ltx_citemacro_citet">Zhou et al. (<a href="#bib.bib50" title="" class="ltx_ref">2023</a>)</cite> sampled 50,000 texts from the Pile data and employed ChatGPT to extract their associated entity types. Notably, they did not specify the entity types to the LLMs, aiming to extract a diverse range of entity types. They used the following prompt:</p>
</div>
<figure id="S3.F3" class="ltx_figure"><img src="/html/2311.08526/assets/x4.png" id="S3.F3.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="163" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 3: </span><span id="S3.F3.2.1" class="ltx_text ltx_font_bold">Prompting ChatGPT for entity extraction</span>. This prompt was used <cite class="ltx_cite ltx_citemacro_citet">Zhou et al. (<a href="#bib.bib50" title="" class="ltx_ref">2023</a>)</cite> to construct the Pile-NER dataset.</figcaption>
</figure>
<div id="S3.SS1.p2" class="ltx_para">
<p id="S3.SS1.p2.1" class="ltx_p">Finally, after filtering bad outputs their datasets results in 44889 passages containing in total 240k entity spans and 13k distinct entity types.</p>
</div>
</section>
<section id="S3.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.2 </span>Hyperparameters </h3>

<div id="S3.SS2.p1" class="ltx_para">
<p id="S3.SS2.p1.1" class="ltx_p">Our model, GLiNER, is trained on the Pile-NER dataset, which we described in the previous section. We use the deBERTa-v3 <cite class="ltx_cite ltx_citemacro_citep">(He et al., <a href="#bib.bib13" title="" class="ltx_ref">2021</a>)</cite> as our backbone due to its proven empirical performance. All non-pretrained layers have a width dimension of 768 and a dropout rate of 0.4. Regarding the training process, we employ the AdamW optimizer <cite class="ltx_cite ltx_citemacro_citep">(Loshchilov and Hutter, <a href="#bib.bib24" title="" class="ltx_ref">2017</a>)</cite>, setting a base learning rate of 1e-5 for pretrained layers (the transformer backbone) and 5e-5 for non-pretrained layers (FFN layers, span representation). We trained our models for a maximum of 30k steps, starting with a 10% warmup phase, followed by a decay phase using a cosine scheduler.
The Pile-NER dataset natively contains only positive entities (i.e., entities that are present in the sentence), and we found it useful to include negative entity types during training. This is achieved by sampling random entities from other examples in the same batch. In addition, we follow the strategies outlined in <cite class="ltx_cite ltx_citemacro_citet">Sainz et al. (<a href="#bib.bib37" title="" class="ltx_ref">2023</a>)</cite> as a form of regularization, which includes <span id="S3.SS2.p1.1.1" class="ltx_text ltx_font_italic">shuffling entity order</span> and <span id="S3.SS2.p1.1.2" class="ltx_text ltx_font_italic">randomly dropping entities</span>. Furthermore, we limit the number of entity types to 25 per sentence during training. The larger variant of our model, GLiNER-L, takes 5 hours to train on an A100 GPU.</p>
</div>
</section>
<section id="S3.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.3 </span>Baselines</h3>

<div id="S3.SS3.p1" class="ltx_para">
<p id="S3.SS3.p1.1" class="ltx_p">In our evaluation, we compare our model, GLiNER, with several recent models designed for open-type NER. First, we examine chat models like <span id="S3.SS3.p1.1.1" class="ltx_text ltx_font_bold">ChatGPT</span> and <span id="S3.SS3.p1.1.2" class="ltx_text ltx_font_bold">Vicuna</span> <cite class="ltx_cite ltx_citemacro_citep">(Chiang et al., <a href="#bib.bib5" title="" class="ltx_ref">2023</a>)</cite>, which utilize the prompting from <cite class="ltx_cite ltx_citemacro_citet">Ye et al. (<a href="#bib.bib46" title="" class="ltx_ref">2023</a>)</cite>; we report their results reported by <cite class="ltx_cite ltx_citemacro_citet">Zhou et al. (<a href="#bib.bib50" title="" class="ltx_ref">2023</a>)</cite>. Next, we focus on three recent Large Language Models (LLMs) that have been fine-tuned for NER: <span id="S3.SS3.p1.1.3" class="ltx_text ltx_font_bold">InstructUIE</span> <cite class="ltx_cite ltx_citemacro_citep">(Wang et al., <a href="#bib.bib40" title="" class="ltx_ref">2023</a>)</cite>, based on the FlanT5 11B model and fine-tuned on various NER datasets; <span id="S3.SS3.p1.1.4" class="ltx_text ltx_font_bold">UniNER</span> <cite class="ltx_cite ltx_citemacro_citep">(Zhou et al., <a href="#bib.bib50" title="" class="ltx_ref">2023</a>)</cite>, which employs a LLaMa model fine-tuned on a dataset generated by ChatGPT; <span id="S3.SS3.p1.1.5" class="ltx_text ltx_font_bold">GoLLIE</span> <cite class="ltx_cite ltx_citemacro_citep">(Sainz et al., <a href="#bib.bib37" title="" class="ltx_ref">2023</a>)</cite>, fine-tuned to adhere to detailed annotation guidelines for enhanced performance in unseen IE tasks, utilizing CodeLLama as its base model. Finally, we include <span id="S3.SS3.p1.1.6" class="ltx_text ltx_font_bold">USM</span> <cite class="ltx_cite ltx_citemacro_citep">(Lou et al., <a href="#bib.bib25" title="" class="ltx_ref">2023</a>)</cite> in our comparison, which is similar in size to ours but features a different architecture.</p>
</div>
</section>
<section id="S3.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.4 </span>Evaluation</h3>

<section id="S3.SS4.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Datasets</h4>

<div id="S3.SS4.SSS0.Px1.p1" class="ltx_para">
<p id="S3.SS4.SSS0.Px1.p1.1" class="ltx_p">We primarily evaluate our model in a zero-shot context on common NER benchmarks, following previous works <cite class="ltx_cite ltx_citemacro_cite">Wang et al. (<a href="#bib.bib40" title="" class="ltx_ref">2023</a>); Zhou et al. (<a href="#bib.bib50" title="" class="ltx_ref">2023</a>)</cite>. The first is the <span id="S3.SS4.SSS0.Px1.p1.1.1" class="ltx_text ltx_font_italic">OOD NER Benchmark</span> (Table <a href="#S2.T1" title="Table 1 ‣ Entity Type and Span Matching ‣ 2.1 Architecture ‣ 2 Method ‣ GLiNER: Generalist Model for Named Entity Recognition using Bidirectional Transformer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>), which comprises seven diverse NER datasets from CrossNER and MIT. This benchmark is typically used for evaluating out-of-domain generalization capabilities of NER models. The second benchmark consists of <span id="S3.SS4.SSS0.Px1.p1.1.2" class="ltx_text ltx_font_italic">20 NER datasets</span> (Table <a href="#S3.T2" title="Table 2 ‣ Metric ‣ 3.4 Evaluation ‣ 3 Experimental Setting ‣ GLiNER: Generalist Model for Named Entity Recognition using Bidirectional Transformer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>) from a wide range of domains, including biomedical, news articles, and tweets. These datasets are commonly used for training supervised NER models. Additionally, we evaluate our model on multilingual NER datasets (Table <a href="#S4.T3" title="Table 3 ‣ 20 NER Benchmark ‣ 4.1 Zero-shot on English datasets ‣ 4 Results ‣ GLiNER: Generalist Model for Named Entity Recognition using Bidirectional Transformer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>) for further investigation. For this purpose, we use the recently released <span id="S3.SS4.SSS0.Px1.p1.1.3" class="ltx_text ltx_font_italic">Multiconer</span> (Multilingual Complex NER) <cite class="ltx_cite ltx_citemacro_citep">(Malmasi et al., <a href="#bib.bib27" title="" class="ltx_ref">2022</a>)</cite>, which contains data in 11 languages across various domains.</p>
</div>
</section>
<section id="S3.SS4.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Metric</h4>

<div id="S3.SS4.SSS0.Px2.p1" class="ltx_para">
<p id="S3.SS4.SSS0.Px2.p1.1" class="ltx_p">We adopt the standard NER evaluation methodology, calculating F1-score based on the exact match between predicted and actual entities.</p>
</div>
<figure id="S3.T2" class="ltx_table">
<table id="S3.T2.1" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S3.T2.1.1.1" class="ltx_tr">
<th id="S3.T2.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt"><span id="S3.T2.1.1.1.1.1" class="ltx_text ltx_font_bold">Dataset</span></th>
<th id="S3.T2.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S3.T2.1.1.1.2.1" class="ltx_text ltx_font_bold">ChatGPT</span></th>
<th id="S3.T2.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S3.T2.1.1.1.3.1" class="ltx_text ltx_font_bold">UniNER-7B</span></th>
<th id="S3.T2.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S3.T2.1.1.1.4.1" class="ltx_text ltx_font_bold">GLiNER-L</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S3.T2.1.2.1" class="ltx_tr">
<th id="S3.T2.1.2.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">ACE05</th>
<td id="S3.T2.1.2.1.2" class="ltx_td ltx_align_center ltx_border_t">26.6</td>
<td id="S3.T2.1.2.1.3" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T2.1.2.1.3.1" class="ltx_text ltx_font_bold">36.9</span></td>
<td id="S3.T2.1.2.1.4" class="ltx_td ltx_align_center ltx_border_t">27.3</td>
</tr>
<tr id="S3.T2.1.3.2" class="ltx_tr">
<th id="S3.T2.1.3.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">AnatEM</th>
<td id="S3.T2.1.3.2.2" class="ltx_td ltx_align_center">30.7</td>
<td id="S3.T2.1.3.2.3" class="ltx_td ltx_align_center">25.1</td>
<td id="S3.T2.1.3.2.4" class="ltx_td ltx_align_center"><span id="S3.T2.1.3.2.4.1" class="ltx_text ltx_font_bold">33.3</span></td>
</tr>
<tr id="S3.T2.1.4.3" class="ltx_tr">
<th id="S3.T2.1.4.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">bc2gm</th>
<td id="S3.T2.1.4.3.2" class="ltx_td ltx_align_center">40.2</td>
<td id="S3.T2.1.4.3.3" class="ltx_td ltx_align_center">46.2</td>
<td id="S3.T2.1.4.3.4" class="ltx_td ltx_align_center"><span id="S3.T2.1.4.3.4.1" class="ltx_text ltx_font_bold">47.9</span></td>
</tr>
<tr id="S3.T2.1.5.4" class="ltx_tr">
<th id="S3.T2.1.5.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">bc4chemd</th>
<td id="S3.T2.1.5.4.2" class="ltx_td ltx_align_center">35.5</td>
<td id="S3.T2.1.5.4.3" class="ltx_td ltx_align_center"><span id="S3.T2.1.5.4.3.1" class="ltx_text ltx_font_bold">47.9</span></td>
<td id="S3.T2.1.5.4.4" class="ltx_td ltx_align_center">43.1</td>
</tr>
<tr id="S3.T2.1.6.5" class="ltx_tr">
<th id="S3.T2.1.6.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">bc5cdr</th>
<td id="S3.T2.1.6.5.2" class="ltx_td ltx_align_center">52.4</td>
<td id="S3.T2.1.6.5.3" class="ltx_td ltx_align_center"><span id="S3.T2.1.6.5.3.1" class="ltx_text ltx_font_bold">68.0</span></td>
<td id="S3.T2.1.6.5.4" class="ltx_td ltx_align_center">66.4</td>
</tr>
<tr id="S3.T2.1.7.6" class="ltx_tr">
<th id="S3.T2.1.7.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Broad Tweeter</th>
<td id="S3.T2.1.7.6.2" class="ltx_td ltx_align_center">61.8</td>
<td id="S3.T2.1.7.6.3" class="ltx_td ltx_align_center"><span id="S3.T2.1.7.6.3.1" class="ltx_text ltx_font_bold">67.9</span></td>
<td id="S3.T2.1.7.6.4" class="ltx_td ltx_align_center">61.2</td>
</tr>
<tr id="S3.T2.1.8.7" class="ltx_tr">
<th id="S3.T2.1.8.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">CoNLL03</th>
<td id="S3.T2.1.8.7.2" class="ltx_td ltx_align_center">52.5</td>
<td id="S3.T2.1.8.7.3" class="ltx_td ltx_align_center"><span id="S3.T2.1.8.7.3.1" class="ltx_text ltx_font_bold">72.2</span></td>
<td id="S3.T2.1.8.7.4" class="ltx_td ltx_align_center">64.6</td>
</tr>
<tr id="S3.T2.1.9.8" class="ltx_tr">
<th id="S3.T2.1.9.8.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">FabNER</th>
<td id="S3.T2.1.9.8.2" class="ltx_td ltx_align_center">15.3</td>
<td id="S3.T2.1.9.8.3" class="ltx_td ltx_align_center"><span id="S3.T2.1.9.8.3.1" class="ltx_text ltx_font_bold">24.8</span></td>
<td id="S3.T2.1.9.8.4" class="ltx_td ltx_align_center">23.6</td>
</tr>
<tr id="S3.T2.1.10.9" class="ltx_tr">
<th id="S3.T2.1.10.9.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">FindVehicle</th>
<td id="S3.T2.1.10.9.2" class="ltx_td ltx_align_center">10.5</td>
<td id="S3.T2.1.10.9.3" class="ltx_td ltx_align_center">22.2</td>
<td id="S3.T2.1.10.9.4" class="ltx_td ltx_align_center"><span id="S3.T2.1.10.9.4.1" class="ltx_text ltx_font_bold">41.9</span></td>
</tr>
<tr id="S3.T2.1.11.10" class="ltx_tr">
<th id="S3.T2.1.11.10.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">GENIA</th>
<td id="S3.T2.1.11.10.2" class="ltx_td ltx_align_center">41.6</td>
<td id="S3.T2.1.11.10.3" class="ltx_td ltx_align_center">54.1</td>
<td id="S3.T2.1.11.10.4" class="ltx_td ltx_align_center"><span id="S3.T2.1.11.10.4.1" class="ltx_text ltx_font_bold">55.5</span></td>
</tr>
<tr id="S3.T2.1.12.11" class="ltx_tr">
<th id="S3.T2.1.12.11.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">HarveyNER</th>
<td id="S3.T2.1.12.11.2" class="ltx_td ltx_align_center">11.6</td>
<td id="S3.T2.1.12.11.3" class="ltx_td ltx_align_center">18.2</td>
<td id="S3.T2.1.12.11.4" class="ltx_td ltx_align_center"><span id="S3.T2.1.12.11.4.1" class="ltx_text ltx_font_bold">22.7</span></td>
</tr>
<tr id="S3.T2.1.13.12" class="ltx_tr">
<th id="S3.T2.1.13.12.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">MIT Movie</th>
<td id="S3.T2.1.13.12.2" class="ltx_td ltx_align_center">5.3</td>
<td id="S3.T2.1.13.12.3" class="ltx_td ltx_align_center">42.4</td>
<td id="S3.T2.1.13.12.4" class="ltx_td ltx_align_center"><span id="S3.T2.1.13.12.4.1" class="ltx_text ltx_font_bold">57.2</span></td>
</tr>
<tr id="S3.T2.1.14.13" class="ltx_tr">
<th id="S3.T2.1.14.13.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">MIT Restaurant</th>
<td id="S3.T2.1.14.13.2" class="ltx_td ltx_align_center">32.8</td>
<td id="S3.T2.1.14.13.3" class="ltx_td ltx_align_center">31.7</td>
<td id="S3.T2.1.14.13.4" class="ltx_td ltx_align_center"><span id="S3.T2.1.14.13.4.1" class="ltx_text ltx_font_bold">42.9</span></td>
</tr>
<tr id="S3.T2.1.15.14" class="ltx_tr">
<th id="S3.T2.1.15.14.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">MultiNERD</th>
<td id="S3.T2.1.15.14.2" class="ltx_td ltx_align_center">58.1</td>
<td id="S3.T2.1.15.14.3" class="ltx_td ltx_align_center">59.3</td>
<td id="S3.T2.1.15.14.4" class="ltx_td ltx_align_center"><span id="S3.T2.1.15.14.4.1" class="ltx_text ltx_font_bold">59.7</span></td>
</tr>
<tr id="S3.T2.1.16.15" class="ltx_tr">
<th id="S3.T2.1.16.15.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">ncbi</th>
<td id="S3.T2.1.16.15.2" class="ltx_td ltx_align_center">42.1</td>
<td id="S3.T2.1.16.15.3" class="ltx_td ltx_align_center">60.4</td>
<td id="S3.T2.1.16.15.4" class="ltx_td ltx_align_center"><span id="S3.T2.1.16.15.4.1" class="ltx_text ltx_font_bold">61.9</span></td>
</tr>
<tr id="S3.T2.1.17.16" class="ltx_tr">
<th id="S3.T2.1.17.16.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">OntoNotes</th>
<td id="S3.T2.1.17.16.2" class="ltx_td ltx_align_center">29.7</td>
<td id="S3.T2.1.17.16.3" class="ltx_td ltx_align_center">27.8</td>
<td id="S3.T2.1.17.16.4" class="ltx_td ltx_align_center"><span id="S3.T2.1.17.16.4.1" class="ltx_text ltx_font_bold">32.2</span></td>
</tr>
<tr id="S3.T2.1.18.17" class="ltx_tr">
<th id="S3.T2.1.18.17.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">PolyglotNER</th>
<td id="S3.T2.1.18.17.2" class="ltx_td ltx_align_center">33.6</td>
<td id="S3.T2.1.18.17.3" class="ltx_td ltx_align_center">41.8</td>
<td id="S3.T2.1.18.17.4" class="ltx_td ltx_align_center"><span id="S3.T2.1.18.17.4.1" class="ltx_text ltx_font_bold">42.9</span></td>
</tr>
<tr id="S3.T2.1.19.18" class="ltx_tr">
<th id="S3.T2.1.19.18.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">TweetNER7</th>
<td id="S3.T2.1.19.18.2" class="ltx_td ltx_align_center">40.1</td>
<td id="S3.T2.1.19.18.3" class="ltx_td ltx_align_center"><span id="S3.T2.1.19.18.3.1" class="ltx_text ltx_font_bold">42.7</span></td>
<td id="S3.T2.1.19.18.4" class="ltx_td ltx_align_center">41.4</td>
</tr>
<tr id="S3.T2.1.20.19" class="ltx_tr">
<th id="S3.T2.1.20.19.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">WikiANN</th>
<td id="S3.T2.1.20.19.2" class="ltx_td ltx_align_center">52.0</td>
<td id="S3.T2.1.20.19.3" class="ltx_td ltx_align_center">55.4</td>
<td id="S3.T2.1.20.19.4" class="ltx_td ltx_align_center"><span id="S3.T2.1.20.19.4.1" class="ltx_text ltx_font_bold">58.9</span></td>
</tr>
<tr id="S3.T2.1.21.20" class="ltx_tr">
<th id="S3.T2.1.21.20.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">WikiNeural</th>
<td id="S3.T2.1.21.20.2" class="ltx_td ltx_align_center">57.7</td>
<td id="S3.T2.1.21.20.3" class="ltx_td ltx_align_center">69.2</td>
<td id="S3.T2.1.21.20.4" class="ltx_td ltx_align_center"><span id="S3.T2.1.21.20.4.1" class="ltx_text ltx_font_bold">71.8</span></td>
</tr>
<tr id="S3.T2.1.22.21" class="ltx_tr">
<th id="S3.T2.1.22.21.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb ltx_border_t"><span id="S3.T2.1.22.21.1.1" class="ltx_text ltx_font_bold">Average</span></th>
<td id="S3.T2.1.22.21.2" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">36.5</td>
<td id="S3.T2.1.22.21.3" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">45.7</td>
<td id="S3.T2.1.22.21.4" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t"><span id="S3.T2.1.22.21.4.1" class="ltx_text ltx_font_bold">47.8</span></td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 2: </span>Z<span id="S3.T2.3.1" class="ltx_text ltx_font_bold">ero-shot performance on 20 NER datasets.</span> Results of ChatGPT and UniNER are reported from <cite class="ltx_cite ltx_citemacro_citep">(Zhou et al., <a href="#bib.bib50" title="" class="ltx_ref">2023</a>)</cite>.</figcaption>
</figure>
</section>
</section>
</section>
<section id="S4" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">4 </span>Results</h2>

<section id="S4.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.1 </span>Zero-shot on English datasets</h3>

<div id="S4.SS1.p1" class="ltx_para">
<p id="S4.SS1.p1.1" class="ltx_p">In this section, we discuss the performance of our model in a zero-shot context, i.e., by only training on the Pile-NER dataset without further fine-tuning on target datasets.</p>
</div>
<section id="S4.SS1.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">OOD NER Benchmark</h4>

<div id="S4.SS1.SSS0.Px1.p1" class="ltx_para">
<p id="S4.SS1.SSS0.Px1.p1.1" class="ltx_p">We first evaluate our model on the OOD benchmark as reported in Table <a href="#S2.T1" title="Table 1 ‣ Entity Type and Span Matching ‣ 2.1 Architecture ‣ 2 Method ‣ GLiNER: Generalist Model for Named Entity Recognition using Bidirectional Transformer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>. We compare three different sizes of our model (small, medium, and large) against the baselines. The results demonstrate our model’s impressive capability, irrespective of its size. For example, even our smallest model, with only 50M parameters, outperforms general-purpose models such as ChatGPT and Vicuna. It also shows better performance than the 11B InstructUIE, which has been instruction-tuned for the NER task. Furthermore, when compared to UniNER, which used the same training data as GLiNER, our medium-sized model (90M) achieves comparable results to UniNER-13B (55 F1 for both), despite being 140 times smaller, while our largest version outperforms it by an average of 5 points. Our most best competitor, GoLLIE, which leads among the LLMs, achieves better performance than most of our models but is still less effective than GLiNER-L. Against USM, which has a comparable number of parameters to ours, the performance is significantly lower, highlighting the superiority of our architecture.</p>
</div>
</section>
<section id="S4.SS1.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">20 NER Benchmark</h4>

<div id="S4.SS1.SSS0.Px2.p1" class="ltx_para">
<p id="S4.SS1.SSS0.Px2.p1.1" class="ltx_p">Table 2 presents a comparison of our model against ChatGPT and UniNER across 20 diverse NER datasets. First, similar to the OOD benchmark, ChatGPT significantly lags behind fine-tuned models for NER, trailing behind UniNER. Furthermore, GLiNER achieves the highest performance on 13 of these datasets, surpassing UniNER by an average of 2 points. This superior performance underscores GLiNER’s robustness and adaptability across a broad spectrum of domains. However, a notable observation is that GLiNER underperforms compared to UniNER on tweet-based NER datasets. This highlights potential areas for improvement in GLiNER’s ability to process informal, colloquial, or noisy data, typical of social media content.</p>
</div>
<figure id="S4.T3" class="ltx_table">
<table id="S4.T3.1" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="S4.T3.1.1.1" class="ltx_tr">
<th id="S4.T3.1.1.1.1" class="ltx_td ltx_th ltx_th_row ltx_border_tt"></th>
<th id="S4.T3.1.1.1.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r ltx_border_tt" rowspan="2"><span id="S4.T3.1.1.1.2.1" class="ltx_text ltx_font_bold">Language</span></th>
<th id="S4.T3.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r ltx_border_tt" rowspan="2"><span id="S4.T3.1.1.1.3.1" class="ltx_text ltx_font_bold">Sup.</span></th>
<td id="S4.T3.1.1.1.4" class="ltx_td ltx_align_center ltx_border_tt" rowspan="2"><span id="S4.T3.1.1.1.4.1" class="ltx_text ltx_font_bold">ChatGPT</span></td>
<td id="S4.T3.1.1.1.5" class="ltx_td ltx_align_center ltx_border_tt" colspan="2"><span id="S4.T3.1.1.1.5.1" class="ltx_text ltx_font_bold">GLiNER</span></td>
</tr>
<tr id="S4.T3.1.2.2" class="ltx_tr">
<th id="S4.T3.1.2.2.1" class="ltx_td ltx_th ltx_th_row"></th>
<td id="S4.T3.1.2.2.2" class="ltx_td ltx_align_center">En</td>
<td id="S4.T3.1.2.2.3" class="ltx_td ltx_align_center">Multi</td>
</tr>
<tr id="S4.T3.1.3.3" class="ltx_tr">
<th id="S4.T3.1.3.3.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t" rowspan="4"><span id="S4.T3.1.3.3.1.1" class="ltx_text">
<span id="S4.T3.1.3.3.1.1.1" class="ltx_inline-block ltx_transformed_outer" style="width:6.8pt;height:23.5pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="width:23.5pt;transform:translate(-8.32pt,-8.32pt) rotate(-90deg) ;">
<span id="S4.T3.1.3.3.1.1.1.1" class="ltx_p">Latin</span>
</span></span></span></th>
<th id="S4.T3.1.3.3.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r ltx_border_t">German</th>
<th id="S4.T3.1.3.3.3" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r ltx_border_t">64.6</th>
<td id="S4.T3.1.3.3.4" class="ltx_td ltx_align_center ltx_border_t">37.1</td>
<td id="S4.T3.1.3.3.5" class="ltx_td ltx_align_center ltx_border_t">35.6</td>
<td id="S4.T3.1.3.3.6" class="ltx_td ltx_align_center ltx_border_t"><span id="S4.T3.1.3.3.6.1" class="ltx_text ltx_font_bold">39.5</span></td>
</tr>
<tr id="S4.T3.1.4.4" class="ltx_tr">
<th id="S4.T3.1.4.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">English</th>
<th id="S4.T3.1.4.4.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">62.7</th>
<td id="S4.T3.1.4.4.3" class="ltx_td ltx_align_center">37.2</td>
<td id="S4.T3.1.4.4.4" class="ltx_td ltx_align_center"><span id="S4.T3.1.4.4.4.1" class="ltx_text ltx_font_bold">42.4</span></td>
<td id="S4.T3.1.4.4.5" class="ltx_td ltx_align_center">41.7</td>
</tr>
<tr id="S4.T3.1.5.5" class="ltx_tr">
<th id="S4.T3.1.5.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">Spanish</th>
<th id="S4.T3.1.5.5.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">58.7</th>
<td id="S4.T3.1.5.5.3" class="ltx_td ltx_align_center">34.7</td>
<td id="S4.T3.1.5.5.4" class="ltx_td ltx_align_center">38.7</td>
<td id="S4.T3.1.5.5.5" class="ltx_td ltx_align_center"><span id="S4.T3.1.5.5.5.1" class="ltx_text ltx_font_bold">42.1</span></td>
</tr>
<tr id="S4.T3.1.6.6" class="ltx_tr">
<th id="S4.T3.1.6.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">Dutch</th>
<th id="S4.T3.1.6.6.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">62.6</th>
<td id="S4.T3.1.6.6.3" class="ltx_td ltx_align_center">35.7</td>
<td id="S4.T3.1.6.6.4" class="ltx_td ltx_align_center">35.6</td>
<td id="S4.T3.1.6.6.5" class="ltx_td ltx_align_center"><span id="S4.T3.1.6.6.5.1" class="ltx_text ltx_font_bold">38.9</span></td>
</tr>
<tr id="S4.T3.1.7.7" class="ltx_tr">
<th id="S4.T3.1.7.7.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t" rowspan="7"><span id="S4.T3.1.7.7.1.1" class="ltx_text">
<span id="S4.T3.1.7.7.1.1.1" class="ltx_inline-block ltx_transformed_outer" style="width:6.8pt;height:44.9pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="width:44.9pt;transform:translate(-19.01pt,-19.01pt) rotate(-90deg) ;">
<span id="S4.T3.1.7.7.1.1.1.1" class="ltx_p">Non-Latin</span>
</span></span></span></th>
<th id="S4.T3.1.7.7.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r ltx_border_t">Bengali</th>
<th id="S4.T3.1.7.7.3" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r ltx_border_t">39.7</th>
<td id="S4.T3.1.7.7.4" class="ltx_td ltx_align_center ltx_border_t">23.3</td>
<td id="S4.T3.1.7.7.5" class="ltx_td ltx_align_center ltx_border_t">0.89</td>
<td id="S4.T3.1.7.7.6" class="ltx_td ltx_align_center ltx_border_t"><span id="S4.T3.1.7.7.6.1" class="ltx_text ltx_font_bold">25.9</span></td>
</tr>
<tr id="S4.T3.1.8.8" class="ltx_tr">
<th id="S4.T3.1.8.8.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">Persian</th>
<th id="S4.T3.1.8.8.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">52.3</th>
<td id="S4.T3.1.8.8.3" class="ltx_td ltx_align_center">25.9</td>
<td id="S4.T3.1.8.8.4" class="ltx_td ltx_align_center">14.9</td>
<td id="S4.T3.1.8.8.5" class="ltx_td ltx_align_center"><span id="S4.T3.1.8.8.5.1" class="ltx_text ltx_font_bold">30.2</span></td>
</tr>
<tr id="S4.T3.1.9.9" class="ltx_tr">
<th id="S4.T3.1.9.9.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">Hindi</th>
<th id="S4.T3.1.9.9.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">47.8</th>
<td id="S4.T3.1.9.9.3" class="ltx_td ltx_align_center">27.3</td>
<td id="S4.T3.1.9.9.4" class="ltx_td ltx_align_center">11.3</td>
<td id="S4.T3.1.9.9.5" class="ltx_td ltx_align_center"><span id="S4.T3.1.9.9.5.1" class="ltx_text ltx_font_bold">27.8</span></td>
</tr>
<tr id="S4.T3.1.10.10" class="ltx_tr">
<th id="S4.T3.1.10.10.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">Korean</th>
<th id="S4.T3.1.10.10.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">55.8</th>
<td id="S4.T3.1.10.10.3" class="ltx_td ltx_align_center"><span id="S4.T3.1.10.10.3.1" class="ltx_text ltx_font_bold">30.0</span></td>
<td id="S4.T3.1.10.10.4" class="ltx_td ltx_align_center">20.5</td>
<td id="S4.T3.1.10.10.5" class="ltx_td ltx_align_center">28.7</td>
</tr>
<tr id="S4.T3.1.11.11" class="ltx_tr">
<th id="S4.T3.1.11.11.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">Russian</th>
<th id="S4.T3.1.11.11.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">59.7</th>
<td id="S4.T3.1.11.11.3" class="ltx_td ltx_align_center">27.4</td>
<td id="S4.T3.1.11.11.4" class="ltx_td ltx_align_center">30.3</td>
<td id="S4.T3.1.11.11.5" class="ltx_td ltx_align_center"><span id="S4.T3.1.11.11.5.1" class="ltx_text ltx_font_bold">33.3</span></td>
</tr>
<tr id="S4.T3.1.12.12" class="ltx_tr">
<th id="S4.T3.1.12.12.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">Turkish</th>
<th id="S4.T3.1.12.12.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">46.8</th>
<td id="S4.T3.1.12.12.3" class="ltx_td ltx_align_center"><span id="S4.T3.1.12.12.3.1" class="ltx_text ltx_font_bold">31.9</span></td>
<td id="S4.T3.1.12.12.4" class="ltx_td ltx_align_center">22.0</td>
<td id="S4.T3.1.12.12.5" class="ltx_td ltx_align_center">30.0</td>
</tr>
<tr id="S4.T3.1.13.13" class="ltx_tr">
<th id="S4.T3.1.13.13.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">Chinese</th>
<th id="S4.T3.1.13.13.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">53.1</th>
<td id="S4.T3.1.13.13.3" class="ltx_td ltx_align_center">18.8</td>
<td id="S4.T3.1.13.13.4" class="ltx_td ltx_align_center">6.59</td>
<td id="S4.T3.1.13.13.5" class="ltx_td ltx_align_center"><span id="S4.T3.1.13.13.5.1" class="ltx_text ltx_font_bold">24.3</span></td>
</tr>
<tr id="S4.T3.1.14.14" class="ltx_tr">
<th id="S4.T3.1.14.14.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb ltx_border_r ltx_border_t" colspan="2"><span id="S4.T3.1.14.14.1.1" class="ltx_text ltx_font_bold">Average</span></th>
<th id="S4.T3.1.14.14.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_bb ltx_border_r ltx_border_t">54.9</th>
<td id="S4.T3.1.14.14.3" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">29.9</td>
<td id="S4.T3.1.14.14.4" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">23.6</td>
<td id="S4.T3.1.14.14.5" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t"><span id="S4.T3.1.14.14.5.1" class="ltx_text ltx_font_bold">32.9</span></td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 3: </span><span id="S4.T3.4.1" class="ltx_text ltx_font_bold">Zero-Shot Scores on Different Languages.</span> The baseline, <span id="S4.T3.5.2" class="ltx_text ltx_font_bold">Sup.</span>, is an XLM-R <cite class="ltx_cite ltx_citemacro_citep">(Conneau et al., <a href="#bib.bib8" title="" class="ltx_ref">2019</a>)</cite> model fine-tuned on the training set of each language separately, as reported by <cite class="ltx_cite ltx_citemacro_citet">Malmasi et al. (<a href="#bib.bib27" title="" class="ltx_ref">2022</a>)</cite>. ChatGPT evaluation is taken from <cite class="ltx_cite ltx_citemacro_citet">Lai et al. (<a href="#bib.bib16" title="" class="ltx_ref">2023</a>)</cite>. GLiNER-En employs deBERTa-v3-Large, and Multi uses mdeBERTa-v3-base.</figcaption>
</figure>
</section>
</section>
<section id="S4.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.2 </span>Zero-Shot Multilingual Evaluation</h3>

<div id="S4.SS2.p1" class="ltx_para">
<p id="S4.SS2.p1.1" class="ltx_p">In this section, we evaluate the performance of our model in a zero-shot context on unseen languages to assess its generalizability. This evaluation uses the Multiconel dataset, with results detailed in Table 3. Our model, GLiNER, is presented in two variants: <span id="S4.SS2.p1.1.1" class="ltx_text ltx_font_bold">En</span>, which employs deBERTa-v3-Large as its backbone, and <span id="S4.SS2.p1.1.2" class="ltx_text ltx_font_bold">Multi</span>, which utilizes a multilingual version of deBERTa-v3 (mdeBERTa-v3). Both versions were fine-tuned on the Pile-NER dataset. For comparative purposes, we include results from ChatGPT and a supervised baseline, the latter being fine-tuned on the training set of each dataset using separate models.</p>
</div>
<section id="S4.SS2.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Results</h4>

<div id="S4.SS2.SSS0.Px1.p1" class="ltx_para">
<p id="S4.SS2.SSS0.Px1.p1.1" class="ltx_p">As expected, the supervised baseline demonstrated superior performance, significantly outperforming the zero-shot models. Among these, GLiNER-Multi showed the most promising results, surpassing ChatGPT in most languages, which is noteworthy considering that the fine-tuning dataset, Pile-NER, consists exclusively of English examples. Interestingly, its performance on Spanish data slightly exceeded that in English. While GLiNER-En generally underperformed compared to ChatGPT on average, it achieved competitive, and at times superior, results in languages using the Latin script, such as Spanish and German. However, its performance was markedly less competitive in non-Latin languages, particularly in Bengali, where it scored only 0.89 in the F1 score.</p>
</div>
<figure id="S4.T4" class="ltx_table">
<table id="S4.T4.1" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S4.T4.1.1.1" class="ltx_tr">
<th id="S4.T4.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt ltx_border_t" rowspan="2"><span id="S4.T4.1.1.1.1.1" class="ltx_text ltx_font_bold">Dataset</span></th>
<th id="S4.T4.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt ltx_border_t"><span id="S4.T4.1.1.1.2.1" class="ltx_text ltx_font_bold">InstructUIE</span></th>
<th id="S4.T4.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt ltx_border_t"><span id="S4.T4.1.1.1.3.1" class="ltx_text ltx_font_bold">UniNER-7B</span></th>
<th id="S4.T4.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt ltx_border_t" colspan="2"><span id="S4.T4.1.1.1.4.1" class="ltx_text ltx_font_bold">GLiNER-L</span></th>
</tr>
<tr id="S4.T4.1.2.2" class="ltx_tr">
<th id="S4.T4.1.2.2.1" class="ltx_td ltx_align_center ltx_th ltx_th_column"><span id="S4.T4.1.2.2.1.1" class="ltx_text ltx_font_bold">w/o</span></th>
<th id="S4.T4.1.2.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_column"><span id="S4.T4.1.2.2.2.1" class="ltx_text ltx_font_bold">w/</span></th>
<th id="S4.T4.1.2.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_column"><span id="S4.T4.1.2.2.3.1" class="ltx_text ltx_font_bold">w/</span></th>
<th id="S4.T4.1.2.2.4" class="ltx_td ltx_align_center ltx_th ltx_th_column"><span id="S4.T4.1.2.2.4.1" class="ltx_text ltx_font_bold">w/o</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S4.T4.1.3.1" class="ltx_tr">
<th id="S4.T4.1.3.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">ACE05</th>
<td id="S4.T4.1.3.1.2" class="ltx_td ltx_align_center ltx_border_t">79.9</td>
<td id="S4.T4.1.3.1.3" class="ltx_td ltx_align_center ltx_border_t"><span id="S4.T4.1.3.1.3.1" class="ltx_text ltx_font_bold">86.7</span></td>
<td id="S4.T4.1.3.1.4" class="ltx_td ltx_align_center ltx_border_t">82.8</td>
<td id="S4.T4.1.3.1.5" class="ltx_td ltx_align_center ltx_border_t">81.3</td>
</tr>
<tr id="S4.T4.1.4.2" class="ltx_tr">
<th id="S4.T4.1.4.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">AnatEM</th>
<td id="S4.T4.1.4.2.2" class="ltx_td ltx_align_center">88.5</td>
<td id="S4.T4.1.4.2.3" class="ltx_td ltx_align_center">88.5</td>
<td id="S4.T4.1.4.2.4" class="ltx_td ltx_align_center"><span id="S4.T4.1.4.2.4.1" class="ltx_text ltx_font_bold">88.9</span></td>
<td id="S4.T4.1.4.2.5" class="ltx_td ltx_align_center">88.4</td>
</tr>
<tr id="S4.T4.1.5.3" class="ltx_tr">
<th id="S4.T4.1.5.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">bc2gm</th>
<td id="S4.T4.1.5.3.2" class="ltx_td ltx_align_center">80.7</td>
<td id="S4.T4.1.5.3.3" class="ltx_td ltx_align_center">82.4</td>
<td id="S4.T4.1.5.3.4" class="ltx_td ltx_align_center"><span id="S4.T4.1.5.3.4.1" class="ltx_text ltx_font_bold">83.7</span></td>
<td id="S4.T4.1.5.3.5" class="ltx_td ltx_align_center">82.0</td>
</tr>
<tr id="S4.T4.1.6.4" class="ltx_tr">
<th id="S4.T4.1.6.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">bc4chemd</th>
<td id="S4.T4.1.6.4.2" class="ltx_td ltx_align_center">87.6</td>
<td id="S4.T4.1.6.4.3" class="ltx_td ltx_align_center"><span id="S4.T4.1.6.4.3.1" class="ltx_text ltx_font_bold">89.2</span></td>
<td id="S4.T4.1.6.4.4" class="ltx_td ltx_align_center">87.9</td>
<td id="S4.T4.1.6.4.5" class="ltx_td ltx_align_center">86.7</td>
</tr>
<tr id="S4.T4.1.7.5" class="ltx_tr">
<th id="S4.T4.1.7.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">bc5cdr</th>
<td id="S4.T4.1.7.5.2" class="ltx_td ltx_align_center">89.0</td>
<td id="S4.T4.1.7.5.3" class="ltx_td ltx_align_center"><span id="S4.T4.1.7.5.3.1" class="ltx_text ltx_font_bold">89.3</span></td>
<td id="S4.T4.1.7.5.4" class="ltx_td ltx_align_center">88.7</td>
<td id="S4.T4.1.7.5.5" class="ltx_td ltx_align_center">88.7</td>
</tr>
<tr id="S4.T4.1.8.6" class="ltx_tr">
<th id="S4.T4.1.8.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Broad Twitter</th>
<td id="S4.T4.1.8.6.2" class="ltx_td ltx_align_center">80.3</td>
<td id="S4.T4.1.8.6.3" class="ltx_td ltx_align_center">81.2</td>
<td id="S4.T4.1.8.6.4" class="ltx_td ltx_align_center">82.5</td>
<td id="S4.T4.1.8.6.5" class="ltx_td ltx_align_center"><span id="S4.T4.1.8.6.5.1" class="ltx_text ltx_font_bold">82.7</span></td>
</tr>
<tr id="S4.T4.1.9.7" class="ltx_tr">
<th id="S4.T4.1.9.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">CoNLL03</th>
<td id="S4.T4.1.9.7.2" class="ltx_td ltx_align_center">91.5</td>
<td id="S4.T4.1.9.7.3" class="ltx_td ltx_align_center"><span id="S4.T4.1.9.7.3.1" class="ltx_text ltx_font_bold">93.3</span></td>
<td id="S4.T4.1.9.7.4" class="ltx_td ltx_align_center">92.6</td>
<td id="S4.T4.1.9.7.5" class="ltx_td ltx_align_center">92.5</td>
</tr>
<tr id="S4.T4.1.10.8" class="ltx_tr">
<th id="S4.T4.1.10.8.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">FabNER</th>
<td id="S4.T4.1.10.8.2" class="ltx_td ltx_align_center">78.4</td>
<td id="S4.T4.1.10.8.3" class="ltx_td ltx_align_center"><span id="S4.T4.1.10.8.3.1" class="ltx_text ltx_font_bold">81.9</span></td>
<td id="S4.T4.1.10.8.4" class="ltx_td ltx_align_center">77.8</td>
<td id="S4.T4.1.10.8.5" class="ltx_td ltx_align_center">74.8</td>
</tr>
<tr id="S4.T4.1.11.9" class="ltx_tr">
<th id="S4.T4.1.11.9.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">FindVehicle</th>
<td id="S4.T4.1.11.9.2" class="ltx_td ltx_align_center">87.6</td>
<td id="S4.T4.1.11.9.3" class="ltx_td ltx_align_center"><span id="S4.T4.1.11.9.3.1" class="ltx_text ltx_font_bold">98.3</span></td>
<td id="S4.T4.1.11.9.4" class="ltx_td ltx_align_center">95.7</td>
<td id="S4.T4.1.11.9.5" class="ltx_td ltx_align_center">95.2</td>
</tr>
<tr id="S4.T4.1.12.10" class="ltx_tr">
<th id="S4.T4.1.12.10.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">GENIA</th>
<td id="S4.T4.1.12.10.2" class="ltx_td ltx_align_center">75.7</td>
<td id="S4.T4.1.12.10.3" class="ltx_td ltx_align_center">77.5</td>
<td id="S4.T4.1.12.10.4" class="ltx_td ltx_align_center"><span id="S4.T4.1.12.10.4.1" class="ltx_text ltx_font_bold">78.9</span></td>
<td id="S4.T4.1.12.10.5" class="ltx_td ltx_align_center">77.4</td>
</tr>
<tr id="S4.T4.1.13.11" class="ltx_tr">
<th id="S4.T4.1.13.11.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">HarveyNER</th>
<td id="S4.T4.1.13.11.2" class="ltx_td ltx_align_center"><span id="S4.T4.1.13.11.2.1" class="ltx_text ltx_font_bold">74.7</span></td>
<td id="S4.T4.1.13.11.3" class="ltx_td ltx_align_center">74.2</td>
<td id="S4.T4.1.13.11.4" class="ltx_td ltx_align_center">68.6</td>
<td id="S4.T4.1.13.11.5" class="ltx_td ltx_align_center">67.4</td>
</tr>
<tr id="S4.T4.1.14.12" class="ltx_tr">
<th id="S4.T4.1.14.12.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">MIT Movie</th>
<td id="S4.T4.1.14.12.2" class="ltx_td ltx_align_center">89.6</td>
<td id="S4.T4.1.14.12.3" class="ltx_td ltx_align_center"><span id="S4.T4.1.14.12.3.1" class="ltx_text ltx_font_bold">90.2</span></td>
<td id="S4.T4.1.14.12.4" class="ltx_td ltx_align_center">87.9</td>
<td id="S4.T4.1.14.12.5" class="ltx_td ltx_align_center">87.5</td>
</tr>
<tr id="S4.T4.1.15.13" class="ltx_tr">
<th id="S4.T4.1.15.13.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">MIT Restaurant</th>
<td id="S4.T4.1.15.13.2" class="ltx_td ltx_align_center">82.6</td>
<td id="S4.T4.1.15.13.3" class="ltx_td ltx_align_center">82.3</td>
<td id="S4.T4.1.15.13.4" class="ltx_td ltx_align_center"><span id="S4.T4.1.15.13.4.1" class="ltx_text ltx_font_bold">83.6</span></td>
<td id="S4.T4.1.15.13.5" class="ltx_td ltx_align_center">83.3</td>
</tr>
<tr id="S4.T4.1.16.14" class="ltx_tr">
<th id="S4.T4.1.16.14.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">MultiNERD</th>
<td id="S4.T4.1.16.14.2" class="ltx_td ltx_align_center">90.3</td>
<td id="S4.T4.1.16.14.3" class="ltx_td ltx_align_center">93.7</td>
<td id="S4.T4.1.16.14.4" class="ltx_td ltx_align_center"><span id="S4.T4.1.16.14.4.1" class="ltx_text ltx_font_bold">93.8</span></td>
<td id="S4.T4.1.16.14.5" class="ltx_td ltx_align_center">93.3</td>
</tr>
<tr id="S4.T4.1.17.15" class="ltx_tr">
<th id="S4.T4.1.17.15.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">ncbi</th>
<td id="S4.T4.1.17.15.2" class="ltx_td ltx_align_center">86.2</td>
<td id="S4.T4.1.17.15.3" class="ltx_td ltx_align_center">87.0</td>
<td id="S4.T4.1.17.15.4" class="ltx_td ltx_align_center"><span id="S4.T4.1.17.15.4.1" class="ltx_text ltx_font_bold">87.8</span></td>
<td id="S4.T4.1.17.15.5" class="ltx_td ltx_align_center">87.1</td>
</tr>
<tr id="S4.T4.1.18.16" class="ltx_tr">
<th id="S4.T4.1.18.16.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">OntoNotes</th>
<td id="S4.T4.1.18.16.2" class="ltx_td ltx_align_center">88.6</td>
<td id="S4.T4.1.18.16.3" class="ltx_td ltx_align_center"><span id="S4.T4.1.18.16.3.1" class="ltx_text ltx_font_bold">89.9</span></td>
<td id="S4.T4.1.18.16.4" class="ltx_td ltx_align_center">89.0</td>
<td id="S4.T4.1.18.16.5" class="ltx_td ltx_align_center">88.1</td>
</tr>
<tr id="S4.T4.1.19.17" class="ltx_tr">
<th id="S4.T4.1.19.17.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">PolyglotNER</th>
<td id="S4.T4.1.19.17.2" class="ltx_td ltx_align_center">53.3</td>
<td id="S4.T4.1.19.17.3" class="ltx_td ltx_align_center"><span id="S4.T4.1.19.17.3.1" class="ltx_text ltx_font_bold">65.7</span></td>
<td id="S4.T4.1.19.17.4" class="ltx_td ltx_align_center">61.5</td>
<td id="S4.T4.1.19.17.5" class="ltx_td ltx_align_center">60.6</td>
</tr>
<tr id="S4.T4.1.20.18" class="ltx_tr">
<th id="S4.T4.1.20.18.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">TweetNER7</th>
<td id="S4.T4.1.20.18.2" class="ltx_td ltx_align_center"><span id="S4.T4.1.20.18.2.1" class="ltx_text ltx_font_bold">65.9</span></td>
<td id="S4.T4.1.20.18.3" class="ltx_td ltx_align_center">65.8</td>
<td id="S4.T4.1.20.18.4" class="ltx_td ltx_align_center">51.4</td>
<td id="S4.T4.1.20.18.5" class="ltx_td ltx_align_center">50.3</td>
</tr>
<tr id="S4.T4.1.21.19" class="ltx_tr">
<th id="S4.T4.1.21.19.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">WikiANN</th>
<td id="S4.T4.1.21.19.2" class="ltx_td ltx_align_center">64.5</td>
<td id="S4.T4.1.21.19.3" class="ltx_td ltx_align_center"><span id="S4.T4.1.21.19.3.1" class="ltx_text ltx_font_bold">84.9</span></td>
<td id="S4.T4.1.21.19.4" class="ltx_td ltx_align_center">83.7</td>
<td id="S4.T4.1.21.19.5" class="ltx_td ltx_align_center">82.8</td>
</tr>
<tr id="S4.T4.1.22.20" class="ltx_tr">
<th id="S4.T4.1.22.20.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">wikiNeural</th>
<td id="S4.T4.1.22.20.2" class="ltx_td ltx_align_center">88.3</td>
<td id="S4.T4.1.22.20.3" class="ltx_td ltx_align_center"><span id="S4.T4.1.22.20.3.1" class="ltx_text ltx_font_bold">93.3</span></td>
<td id="S4.T4.1.22.20.4" class="ltx_td ltx_align_center">91.3</td>
<td id="S4.T4.1.22.20.5" class="ltx_td ltx_align_center">91.4</td>
</tr>
<tr id="S4.T4.1.23.21" class="ltx_tr">
<th id="S4.T4.1.23.21.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb ltx_border_t"><span id="S4.T4.1.23.21.1.1" class="ltx_text ltx_font_bold">Average</span></th>
<td id="S4.T4.1.23.21.2" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">81.2</td>
<td id="S4.T4.1.23.21.3" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t"><span id="S4.T4.1.23.21.3.1" class="ltx_text ltx_font_bold">84.8</span></td>
<td id="S4.T4.1.23.21.4" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">82.9</td>
<td id="S4.T4.1.23.21.5" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">82.1</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 4: </span><span id="S4.T4.4.1" class="ltx_text ltx_font_bold">In-domain Supervised Finetuning.</span> All the models are fine-tuned on the mix of all training data of the benchmark. <span id="S4.T4.5.2" class="ltx_text ltx_font_bold">w/</span> indicates that the model was trained on the Pile-NER dataset before finetuning.</figcaption>
</figure>
</section>
</section>
<section id="S4.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.3 </span>In-domain Supervised tuning</h3>

<div id="S4.SS3.p1" class="ltx_para">
<p id="S4.SS3.p1.1" class="ltx_p">In our work, we also perform in-domain supervised fine-tuning (on 20 NER datasets) of our model to compare its capabilities against LLMs under this setup. Specifically, we compare our model against InstructUIE and UniNER, both of which have also been fine-tuned. The main difference is that UniNER has been pre-trained on the Pile-NER dataset before fine-tuning.</p>
</div>
<section id="S4.SS3.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Training Setup</h4>

<div id="S4.SS3.SSS0.Px1.p1" class="ltx_para">
<p id="S4.SS3.SSS0.Px1.p1.1" class="ltx_p">For the supervised setting, we primarily adhere to the same experimental setup as described in the main experiment (using deBERTa-v3 large), except for the training dataset. Regarding the training data, we follow the approach of InstructUIE: we randomly sample 10,000 data points for each dataset in the 20 NER benchmark. If a dataset does not contain 10,000 samples, we include all available data. We implement two variants of our model: the first one initializes the weights from our zero-shot model, which is a pretrained on the Pile-NER dataset. The second variant is trained without the Pile-NER dataset, same as InstructUIE.</p>
</div>
</section>
<section id="S4.SS3.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Results</h4>

<div id="S4.SS3.SSS0.Px2.p1" class="ltx_para">
<p id="S4.SS3.SSS0.Px2.p1.1" class="ltx_p">The results of our experiment are reported in Table 3. Firstly, we observe that for the in-domain fine-tuning, our GLiNER model, pretrained on Pile-NER, achieves slightly better results than the non-pretrained variant, with an average difference of 0.8. Moreover, our pretrained GLiNER model outperforms InstructUIE (with an average difference of 0.9) despite being fine-tuned on the same dataset, whereas InstructUIE is significantly larger (approximately 30 times so). This demonstrates that our proposed architecture is indeed competitive. However, our model falls behind UniNER by almost 3 points. Nevertheless, our model still manages to achieve the best score in 7 out of 20 datasets.</p>
</div>
<figure id="S4.F4" class="ltx_figure"><img src="/html/2311.08526/assets/x5.png" id="S4.F4.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="368" height="257" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 4: </span><span id="S4.F4.2.1" class="ltx_text ltx_font_bold">Zero-shot performance for different backbones.</span> It reports the avg. results on 20 NER and OOD NER datasets</figcaption>
</figure>
</section>
</section>
</section>
<section id="S5" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">5 </span>Further analysis and ablations</h2>

<div id="S5.p1" class="ltx_para">
<p id="S5.p1.1" class="ltx_p">Here we conduct different set of experiments to better investigate our model.</p>
</div>
<section id="S5.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.1 </span>Effect of Different Backbones</h3>

<div id="S5.SS1.p1" class="ltx_para">
<p id="S5.SS1.p1.1" class="ltx_p">In our work, we primarily utilize the deBERTa-v3 model as our backbone due to its strong empirical performance. However, we demonstrate here that our method is adaptable to a wide range of BiLMs.</p>
</div>
<section id="S5.SS1.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Setup</h4>

<div id="S5.SS1.SSS0.Px1.p1" class="ltx_para">
<p id="S5.SS1.SSS0.Px1.p1.1" class="ltx_p">Specifically, we investigate the performance of our model using other popular BiLMs, including BERT <cite class="ltx_cite ltx_citemacro_citep">(Devlin et al., <a href="#bib.bib10" title="" class="ltx_ref">2019</a>)</cite>, RoBERTa <cite class="ltx_cite ltx_citemacro_citep">(Liu et al., <a href="#bib.bib22" title="" class="ltx_ref">2019</a>)</cite>, AlBERT <cite class="ltx_cite ltx_citemacro_citep">(Lan et al., <a href="#bib.bib18" title="" class="ltx_ref">2019</a>)</cite>, and ELECTRA <cite class="ltx_cite ltx_citemacro_citep">(Clark et al., <a href="#bib.bib7" title="" class="ltx_ref">2020</a>)</cite>. We also conducted experiments with XLNet <cite class="ltx_cite ltx_citemacro_citep">(Yang et al., <a href="#bib.bib45" title="" class="ltx_ref">2019</a>)</cite> but did not achieve acceptable performance (achieving at most 3 F1 on the OOD benchmark) despite extensive hyperparameter tuning. For a fair comparison, we employed the base size (GLiNER-M) and tuned the learning rate for each model. We report the zero-shot results on both the OOD benchmark and the 20 NER benchmark in Figure <a href="#S4.F4" title="Figure 4 ‣ Results ‣ 4.3 In-domain Supervised tuning ‣ 4 Results ‣ GLiNER: Generalist Model for Named Entity Recognition using Bidirectional Transformer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>.</p>
</div>
</section>
<section id="S5.SS1.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Result</h4>

<div id="S5.SS1.SSS0.Px2.p1" class="ltx_para">
<p id="S5.SS1.SSS0.Px2.p1.1" class="ltx_p">The results of our experiment, as shown in the Figure <a href="#S4.F4" title="Figure 4 ‣ Results ‣ 4.3 In-domain Supervised tuning ‣ 4 Results ‣ GLiNER: Generalist Model for Named Entity Recognition using Bidirectional Transformer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>, clearly demonstrate the superiority of deBERTa-v3 over other pretrained BiLMs. It achieves the highest performance on both benchmarks by a clear margin. ELECTRA and AlBERT also show notable performance, albeit slightly lower, while BERT and RoBERTa lag behind with similar scores. However, it should be noted that all of the backbones we tested demonstrate strong performance compared to existing models. More specifically, even BERT-base, which ranks among the lower performers, achieves around 49 F1 on the OOD benchmark. This score is still 2 F1 points higher than the average for models like ChatGPT and InstructUIE.</p>
</div>
</section>
</section>
<section id="S5.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.2 </span>Effect of Pretraining on In-domain Performance</h3>

<div id="S5.SS2.p1" class="ltx_para">
<p id="S5.SS2.p1.1" class="ltx_p">In this section, we investigate the impact of pretraining on the Pile-NER dataset for supervised in-domain training on the 20 NER datasets, across various data sizes. The experiments range from 100 samples per dataset to 10,000 (full training setup). We use the same hyperparameters for all configurations. The results are reported in Figure <a href="#S5.F5" title="Figure 5 ‣ Results ‣ 5.2 Effect of Pretraining on In-domain Performance ‣ 5 Further analysis and ablations ‣ GLiNER: Generalist Model for Named Entity Recognition using Bidirectional Transformer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>.</p>
</div>
<section id="S5.SS2.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Results</h4>

<div id="S5.SS2.SSS0.Px1.p1" class="ltx_para">
<p id="S5.SS2.SSS0.Px1.p1.1" class="ltx_p">As shown in the figure, models pretrained on Pile-NER consistently outperform their counterparts that are only trained on supervised data, indicating successful positive transfer. We further observe that the gain is larger when supervised data is limited. For instance, the difference in performance is 5.6 when employing 100 samples per dataset, and the gap becomes smaller as the size of the dataset increases.</p>
</div>
<figure id="S5.F5" class="ltx_figure"><img src="/html/2311.08526/assets/x6.png" id="S5.F5.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="368" height="252" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 5: </span><span id="S5.F5.2.1" class="ltx_text ltx_font_bold">Supervised performance across different dataset sizes.</span> The evaluation is conducted on the 20 NER datasets (in table <a href="#S4.T4" title="Table 4 ‣ Results ‣ 4.2 Zero-Shot Multilingual Evaluation ‣ 4 Results ‣ GLiNER: Generalist Model for Named Entity Recognition using Bidirectional Transformer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>).</figcaption>
</figure>
<figure id="S5.T5" class="ltx_table">
<table id="S5.T5.1" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S5.T5.1.1.1" class="ltx_tr">
<th id="S5.T5.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt"><span id="S5.T5.1.1.1.1.1" class="ltx_text ltx_font_bold">Negative Samples</span></th>
<th id="S5.T5.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S5.T5.1.1.1.2.1" class="ltx_text ltx_font_bold">Prec</span></th>
<th id="S5.T5.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S5.T5.1.1.1.3.1" class="ltx_text ltx_font_bold">Rec</span></th>
<th id="S5.T5.1.1.1.4" class="ltx_td ltx_nopad_r ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S5.T5.1.1.1.4.1" class="ltx_text ltx_font_bold">F1</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S5.T5.1.2.1" class="ltx_tr">
<th id="S5.T5.1.2.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">0%</th>
<td id="S5.T5.1.2.1.2" class="ltx_td ltx_align_center ltx_border_t"><span id="S5.T5.1.2.1.2.1" class="ltx_text" style="color:#FF0000;">49.3</span></td>
<td id="S5.T5.1.2.1.3" class="ltx_td ltx_align_center ltx_border_t">58.1</td>
<td id="S5.T5.1.2.1.4" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t">53.3</td>
</tr>
<tr id="S5.T5.1.3.2" class="ltx_tr">
<th id="S5.T5.1.3.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">50%</th>
<td id="S5.T5.1.3.2.2" class="ltx_td ltx_align_center"><span id="S5.T5.1.3.2.2.1" class="ltx_text ltx_font_bold">62.3</span></td>
<td id="S5.T5.1.3.2.3" class="ltx_td ltx_align_center"><span id="S5.T5.1.3.2.3.1" class="ltx_text ltx_font_bold">59.7</span></td>
<td id="S5.T5.1.3.2.4" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S5.T5.1.3.2.4.1" class="ltx_text ltx_font_bold">60.9</span></td>
</tr>
<tr id="S5.T5.1.4.3" class="ltx_tr">
<th id="S5.T5.1.4.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">75%</th>
<td id="S5.T5.1.4.3.2" class="ltx_td ltx_align_center ltx_border_bb">61.1</td>
<td id="S5.T5.1.4.3.3" class="ltx_td ltx_align_center ltx_border_bb"><span id="S5.T5.1.4.3.3.1" class="ltx_text" style="color:#FF0000;">56.5</span></td>
<td id="S5.T5.1.4.3.4" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_bb">58.6</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 5: </span><span id="S5.T5.3.1" class="ltx_text ltx_font_bold">Effect of negative entity types sampling.</span></figcaption>
</figure>
</section>
</section>
<section id="S5.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.3 </span>Ablations</h3>

<section id="S5.SS3.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Negative Entity Sampling</h4>

<div id="S5.SS3.SSS0.Px1.p1" class="ltx_para">
<p id="S5.SS3.SSS0.Px1.p1.1" class="ltx_p">The original Pile-NER dataset, curated by <cite class="ltx_cite ltx_citemacro_citet">Zhou et al. (<a href="#bib.bib50" title="" class="ltx_ref">2023</a>)</cite>, features passages with positive entity instances, i.e., entities that are directly present in the text. To better align training with real-world scenarios, where some entity types might be absent, we implemented negative entity sampling as mentioned in Section <a href="#S3.SS2" title="3.2 Hyperparameters ‣ 3 Experimental Setting ‣ GLiNER: Generalist Model for Named Entity Recognition using Bidirectional Transformer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.2</span></a>. In this study, we evaluate different sampling ratios: 0% (only positive entities), 50%, and 75% negative entities.
Table <a href="#S5.T5" title="Table 5 ‣ Results ‣ 5.2 Effect of Pretraining on In-domain Performance ‣ 5 Further analysis and ablations ‣ GLiNER: Generalist Model for Named Entity Recognition using Bidirectional Transformer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a> shows that training with only positive entities results in lower precision but higher recall, indicating that the model often makes false positive errors. Conversely, using 75% negative entities increases precision but decreases recall, as the abundance of negatives makes the model more cautious, leading to missed correct entities. A 50% negative entity ratio proves to be the most effective, providing a balanced approach.</p>
</div>
</section>
<section id="S5.SS3.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Entity type dropping</h4>

<div id="S5.SS3.SSS0.Px2.p1" class="ltx_para">
<p id="S5.SS3.SSS0.Px2.p1.1" class="ltx_p">In the experiment, we employed a strategy of randomly varying the number of entity prompts during training. This approach aimed to expose the model to different quantities of entity types in each training instance, thereby increasing its adaptability to handle scenarios with varying numbers of entities. The usage of this technique results in an average improvement of over 1.4 points in out-of-domain evaluation, as shown in the Figure 5.</p>
</div>
</section>
</section>
</section>
<section id="S6" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">6 </span>Related Works</h2>

<section id="S6.SS0.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Named Entity Recognition</h4>

<div id="S6.SS0.SSS0.Px1.p1" class="ltx_para">
<p id="S6.SS0.SSS0.Px1.p1.1" class="ltx_p">NER is a well-established task in the field of NLP, with numerous applications. Initially, NER models relied on rule-based system <cite class="ltx_cite ltx_citemacro_citep">(Weischedel et al., <a href="#bib.bib43" title="" class="ltx_ref">1996</a>)</cite> that were built using handcrafted algorithms and gazetteers <cite class="ltx_cite ltx_citemacro_citep">(Mikheev et al., <a href="#bib.bib28" title="" class="ltx_ref">1999</a>; Nadeau et al., <a href="#bib.bib31" title="" class="ltx_ref">2006</a>; Zamin and Oxley, <a href="#bib.bib47" title="" class="ltx_ref">2011</a>)</cite>. However, these models had limitations in terms of scalability and adaptability to new domains or languages. To overcome these issues, machine learning approaches have been proposed <cite class="ltx_cite ltx_citemacro_citep">(Lafferty et al., <a href="#bib.bib15" title="" class="ltx_ref">2001</a>)</cite>. In the early stages, NER tasks were designed as sequence labeling <cite class="ltx_cite ltx_citemacro_citep">(Huang et al., <a href="#bib.bib14" title="" class="ltx_ref">2015</a>; Lample et al., <a href="#bib.bib17" title="" class="ltx_ref">2016</a>; Akbik et al., <a href="#bib.bib2" title="" class="ltx_ref">2018</a>)</cite> where the objective was to predict tagged sequences (e.g., BILOU tags <cite class="ltx_cite ltx_citemacro_citep">(Ratinov and Roth, <a href="#bib.bib35" title="" class="ltx_ref">2009</a>)</cite>). Since then, several paradigm shifts have occurred: span-based approaches treating NER as span classification <cite class="ltx_cite ltx_citemacro_citep">(Sarawagi and Cohen, <a href="#bib.bib38" title="" class="ltx_ref">2004</a>; Fu et al., <a href="#bib.bib11" title="" class="ltx_ref">2021</a>; Li et al., <a href="#bib.bib21" title="" class="ltx_ref">2021</a>; Zaratiana et al., <a href="#bib.bib48" title="" class="ltx_ref">2023</a>)</cite>; NER being treated as a question answering problem <cite class="ltx_cite ltx_citemacro_citep">(Li et al., <a href="#bib.bib20" title="" class="ltx_ref">2019</a>)</cite>; and even as a generation task <cite class="ltx_cite ltx_citemacro_citep">(Yan et al., <a href="#bib.bib44" title="" class="ltx_ref">2021</a>)</cite>.</p>
</div>
</section>
<section id="S6.SS0.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Zero-shot learning for NER</h4>

<div id="S6.SS0.SSS0.Px2.p1" class="ltx_para">
<p id="S6.SS0.SSS0.Px2.p1.1" class="ltx_p">The advent of large-scale autoregressive models has recently transformed many paradigms in NLP through natural language prompting <cite class="ltx_cite ltx_citemacro_citep">(Min et al., <a href="#bib.bib29" title="" class="ltx_ref">2022</a>; Wei et al., <a href="#bib.bib42" title="" class="ltx_ref">2022</a>; Qin et al., <a href="#bib.bib33" title="" class="ltx_ref">2023</a>)</cite>. This is also the case for NER <cite class="ltx_cite ltx_citemacro_citep">(Li et al., <a href="#bib.bib19" title="" class="ltx_ref">2022</a>; Ashok and Lipton, <a href="#bib.bib3" title="" class="ltx_ref">2023</a>; Agrawal et al., <a href="#bib.bib1" title="" class="ltx_ref">2022</a>)</cite>. Others have fine-tuned these models for tasks to better align their capabilities with the requirements of entity recognition <cite class="ltx_cite ltx_citemacro_citep">(Cui et al., <a href="#bib.bib9" title="" class="ltx_ref">2021</a>; Zhou et al., <a href="#bib.bib50" title="" class="ltx_ref">2023</a>)</cite> or information extraction in general <cite class="ltx_cite ltx_citemacro_citep">(Lou et al., <a href="#bib.bib25" title="" class="ltx_ref">2023</a>; Wang et al., <a href="#bib.bib40" title="" class="ltx_ref">2023</a>; Sainz et al., <a href="#bib.bib37" title="" class="ltx_ref">2023</a>; Lu et al., <a href="#bib.bib26" title="" class="ltx_ref">2022</a>)</cite>. This is done through-instruction tuning <cite class="ltx_cite ltx_citemacro_citep">(Mishra et al., <a href="#bib.bib30" title="" class="ltx_ref">2021</a>; Wang et al., <a href="#bib.bib41" title="" class="ltx_ref">2022</a>; Longpre et al., <a href="#bib.bib23" title="" class="ltx_ref">2023</a>)</cite>.</p>
</div>
<figure id="S6.F6" class="ltx_figure"><img src="/html/2311.08526/assets/x7.png" id="S6.F6.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="170" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 6: </span><span id="S6.F6.2.1" class="ltx_text ltx_font_bold">Randomly dropping entity types.</span> We report the results with and without negative entity sampling.</figcaption>
</figure>
</section>
</section>
<section id="S7" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">7 </span>Conclusion</h2>

<div id="S7.p1" class="ltx_para">
<p id="S7.p1.1" class="ltx_p">In this paper, we introduced GLiNER, a new method for identifying various types of entities in text using bidirectional language models. Our model not only outperforms state-of-the-art Large Language Models like ChatGPT in zero-shot scenarios but also offers a more resource-efficient alternative, crucial for environments with limited computing power. GLiNER is versatile, performing well in multiple languages, including those it wasn’t trained on. In future work, we aim to further improve GLiNER’s design for enhanced performance and to better adapt it for low-resource languages.</p>
</div>
</section>
<section id="bib" class="ltx_bibliography">
<h2 class="ltx_title ltx_title_bibliography">References</h2>

<ul class="ltx_biblist">
<li id="bib.bib1" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Agrawal et al. (2022)</span>
<span class="ltx_bibblock">
Monica Agrawal, Stefan Hegselmann, Hunter Lang, Yoon Kim, and David A. Sontag. 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://api.semanticscholar.org/CorpusID:249062918" title="" class="ltx_ref ltx_href">Large language models are few-shot clinical information extractors</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib1.1.1" class="ltx_emph ltx_font_italic">Conference on Empirical Methods in Natural Language Processing</em>.

</span>
</li>
<li id="bib.bib2" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Akbik et al. (2018)</span>
<span class="ltx_bibblock">
Alan Akbik, Duncan Blythe, and Roland Vollgraf. 2018.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://aclanthology.org/C18-1139" title="" class="ltx_ref ltx_href">Contextual string embeddings for sequence labeling</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib2.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 27th International Conference on Computational Linguistics</em>, pages 1638–1649, Santa Fe, New Mexico, USA. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib3" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ashok and Lipton (2023)</span>
<span class="ltx_bibblock">
Dhananjay Ashok and Zachary Chase Lipton. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://api.semanticscholar.org/CorpusID:258887456" title="" class="ltx_ref ltx_href">Promptner: Prompting for named entity recognition</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib3.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/2305.15444.

</span>
</li>
<li id="bib.bib4" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Brown et al. (2020)</span>
<span class="ltx_bibblock">
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://api.semanticscholar.org/CorpusID:218971783" title="" class="ltx_ref ltx_href">Language models are few-shot learners</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib4.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/2005.14165.

</span>
</li>
<li id="bib.bib5" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chiang et al. (2023)</span>
<span class="ltx_bibblock">
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://lmsys.org/blog/2023-03-30-vicuna/" title="" class="ltx_ref ltx_href">Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality</a>.

</span>
</li>
<li id="bib.bib6" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chung et al. (2022)</span>
<span class="ltx_bibblock">
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="http://arxiv.org/abs/2210.11416" title="" class="ltx_ref ltx_href">Scaling instruction-finetuned language models</a>.

</span>
</li>
<li id="bib.bib7" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Clark et al. (2020)</span>
<span class="ltx_bibblock">
Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020.

</span>
<span class="ltx_bibblock"><a target="_blank" href="http://arxiv.org/abs/2003.10555" title="" class="ltx_ref ltx_href">Electra: Pre-training text encoders as discriminators rather than generators</a>.

</span>
</li>
<li id="bib.bib8" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Conneau et al. (2019)</span>
<span class="ltx_bibblock">
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://api.semanticscholar.org/CorpusID:207880568" title="" class="ltx_ref ltx_href">Unsupervised cross-lingual representation learning at scale</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib8.1.1" class="ltx_emph ltx_font_italic">Annual Meeting of the Association for Computational Linguistics</em>.

</span>
</li>
<li id="bib.bib9" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Cui et al. (2021)</span>
<span class="ltx_bibblock">
Leyang Cui, Yu Wu, Jian Liu, Sen Yang, and Yue Zhang. 2021.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://api.semanticscholar.org/CorpusID:235313658" title="" class="ltx_ref ltx_href">Template-based named entity recognition using bart</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib9.1.1" class="ltx_emph ltx_font_italic">Findings</em>.

</span>
</li>
<li id="bib.bib10" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Devlin et al. (2019)</span>
<span class="ltx_bibblock">
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://api.semanticscholar.org/CorpusID:52967399" title="" class="ltx_ref ltx_href">Bert: Pre-training of deep bidirectional transformers for language understanding</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib10.1.1" class="ltx_emph ltx_font_italic">North American Chapter of the Association for Computational Linguistics</em>.

</span>
</li>
<li id="bib.bib11" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Fu et al. (2021)</span>
<span class="ltx_bibblock">
Jinlan Fu, Xuanjing Huang, and Pengfei Liu. 2021.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/2021.acl-long.558" title="" class="ltx_ref ltx_href">SpanNER: Named entity re-/recognition as span prediction</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib11.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)</em>, pages 7183–7195, Online. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib12" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gao et al. (2020)</span>
<span class="ltx_bibblock">
Leo Gao, Stella Rose Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://api.semanticscholar.org/CorpusID:230435736" title="" class="ltx_ref ltx_href">The pile: An 800gb dataset of diverse text for language modeling</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib12.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/2101.00027.

</span>
</li>
<li id="bib.bib13" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">He et al. (2021)</span>
<span class="ltx_bibblock">
Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://api.semanticscholar.org/CorpusID:244346093" title="" class="ltx_ref ltx_href">Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib13.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/2111.09543.

</span>
</li>
<li id="bib.bib14" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Huang et al. (2015)</span>
<span class="ltx_bibblock">
Zhiheng Huang, Wei Xu, and Kai Yu. 2015.

</span>
<span class="ltx_bibblock"><a target="_blank" href="http://arxiv.org/abs/1508.01991" title="" class="ltx_ref ltx_href">Bidirectional lstm-crf models for sequence tagging</a>.

</span>
</li>
<li id="bib.bib15" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lafferty et al. (2001)</span>
<span class="ltx_bibblock">
John D. Lafferty, Andrew McCallum, and Fernando Pereira. 2001.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://api.semanticscholar.org/CorpusID:219683473" title="" class="ltx_ref ltx_href">Conditional random fields: Probabilistic models for segmenting and labeling sequence data</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib15.1.1" class="ltx_emph ltx_font_italic">International Conference on Machine Learning</em>.

</span>
</li>
<li id="bib.bib16" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lai et al. (2023)</span>
<span class="ltx_bibblock">
Viet Dac Lai, Nghia Trung Ngo, Amir Pouran Ben Veyseh, Hieu Man, Franck Dernoncourt, Trung Bui, and Thien Huu Nguyen. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://api.semanticscholar.org/CorpusID:258079179" title="" class="ltx_ref ltx_href">Chatgpt beyond english: Towards a comprehensive evaluation of large language models in multilingual learning</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib16.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/2304.05613.

</span>
</li>
<li id="bib.bib17" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lample et al. (2016)</span>
<span class="ltx_bibblock">
Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016.

</span>
<span class="ltx_bibblock">Neural architectures for named entity recognition.

</span>
<span class="ltx_bibblock">In <em id="bib.bib17.1.1" class="ltx_emph ltx_font_italic">North American Chapter of the Association for Computational Linguistics</em>.

</span>
</li>
<li id="bib.bib18" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lan et al. (2019)</span>
<span class="ltx_bibblock">
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://api.semanticscholar.org/CorpusID:202888986" title="" class="ltx_ref ltx_href">Albert: A lite bert for self-supervised learning of language representations</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib18.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/1909.11942.

</span>
</li>
<li id="bib.bib19" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Li et al. (2022)</span>
<span class="ltx_bibblock">
Dongfang Li, Baotian Hu, and Qingcai Chen. 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://api.semanticscholar.org/CorpusID:252819341" title="" class="ltx_ref ltx_href">Prompt-based text entailment for low-resource named entity recognition</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib19.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/2211.03039.

</span>
</li>
<li id="bib.bib20" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Li et al. (2019)</span>
<span class="ltx_bibblock">
Xiaoya Li, Jingrong Feng, Yuxian Meng, Qinghong Han, Fei Wu, and Jiwei Li. 2019.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://api.semanticscholar.org/CorpusID:204902024" title="" class="ltx_ref ltx_href">A unified mrc framework for named entity recognition</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib20.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/1910.11476.

</span>
</li>
<li id="bib.bib21" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Li et al. (2021)</span>
<span class="ltx_bibblock">
Yangming Li, lemao liu, and Shuming Shi. 2021.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://openreview.net/forum?id=5jRVa89sZk" title="" class="ltx_ref ltx_href">Empirical analysis of unlabeled entity problem in named entity recognition</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib21.1.1" class="ltx_emph ltx_font_italic">International Conference on Learning Representations</em>.

</span>
</li>
<li id="bib.bib22" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Liu et al. (2019)</span>
<span class="ltx_bibblock">
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://api.semanticscholar.org/CorpusID:198953378" title="" class="ltx_ref ltx_href">Roberta: A robustly optimized bert pretraining approach</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib22.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/1907.11692.

</span>
</li>
<li id="bib.bib23" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Longpre et al. (2023)</span>
<span class="ltx_bibblock">
S. Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V. Le, Barret Zoph, Jason Wei, and Adam Roberts. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://api.semanticscholar.org/CorpusID:256415991" title="" class="ltx_ref ltx_href">The flan collection: Designing data and methods for effective instruction tuning</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib23.1.1" class="ltx_emph ltx_font_italic">International Conference on Machine Learning</em>.

</span>
</li>
<li id="bib.bib24" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Loshchilov and Hutter (2017)</span>
<span class="ltx_bibblock">
Ilya Loshchilov and Frank Hutter. 2017.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://api.semanticscholar.org/CorpusID:53592270" title="" class="ltx_ref ltx_href">Decoupled weight decay regularization</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib24.1.1" class="ltx_emph ltx_font_italic">International Conference on Learning Representations</em>.

</span>
</li>
<li id="bib.bib25" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lou et al. (2023)</span>
<span class="ltx_bibblock">
Jie Lou, Yaojie Lu, Dai Dai, Wei Jia, Hongyu Lin, Xianpei Han, Le Sun, and Hua Wu. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://api.semanticscholar.org/CorpusID:255546103" title="" class="ltx_ref ltx_href">Universal information extraction as unified semantic matching</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib25.1.1" class="ltx_emph ltx_font_italic">AAAI Conference on Artificial Intelligence</em>.

</span>
</li>
<li id="bib.bib26" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lu et al. (2022)</span>
<span class="ltx_bibblock">
Yaojie Lu, Qing Liu, Dai Dai, Xinyan Xiao, Hongyu Lin, Xianpei Han, Le Sun, and Hua Wu. 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://api.semanticscholar.org/CorpusID:247619149" title="" class="ltx_ref ltx_href">Unified structure generation for universal information extraction</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib26.1.1" class="ltx_emph ltx_font_italic">Annual Meeting of the Association for Computational Linguistics</em>.

</span>
</li>
<li id="bib.bib27" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Malmasi et al. (2022)</span>
<span class="ltx_bibblock">
Shervin Malmasi, Anjie Fang, Besnik Fetahu, Sudipta Kar, and Oleg Rokhlenko. 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://api.semanticscholar.org/CorpusID:251953674" title="" class="ltx_ref ltx_href">Multiconer: A large-scale multilingual dataset for complex named entity recognition</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib27.1.1" class="ltx_emph ltx_font_italic">International Conference on Computational Linguistics</em>.

</span>
</li>
<li id="bib.bib28" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Mikheev et al. (1999)</span>
<span class="ltx_bibblock">
Andrei Mikheev, Marc Moens, and Claire Grover. 1999.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://api.semanticscholar.org/CorpusID:7332330" title="" class="ltx_ref ltx_href">Named entity recognition without gazetteers</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib28.1.1" class="ltx_emph ltx_font_italic">Conference of the European Chapter of the Association for Computational Linguistics</em>.

</span>
</li>
<li id="bib.bib29" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Min et al. (2022)</span>
<span class="ltx_bibblock">
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://api.semanticscholar.org/CorpusID:247155069" title="" class="ltx_ref ltx_href">Rethinking the role of demonstrations: What makes in-context learning work?</a>

</span>
<span class="ltx_bibblock"><em id="bib.bib29.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/2202.12837.

</span>
</li>
<li id="bib.bib30" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Mishra et al. (2021)</span>
<span class="ltx_bibblock">
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2021.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://api.semanticscholar.org/CorpusID:237421373" title="" class="ltx_ref ltx_href">Cross-task generalization via natural language crowdsourcing instructions</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib30.1.1" class="ltx_emph ltx_font_italic">Annual Meeting of the Association for Computational Linguistics</em>.

</span>
</li>
<li id="bib.bib31" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Nadeau et al. (2006)</span>
<span class="ltx_bibblock">
David Nadeau, Peter D. Turney, and Stan Matwin. 2006.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://api.semanticscholar.org/CorpusID:17071982" title="" class="ltx_ref ltx_href">Unsupervised named-entity recognition: Generating gazetteers and resolving ambiguity</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib31.1.1" class="ltx_emph ltx_font_italic">Canadian Conference on AI</em>.

</span>
</li>
<li id="bib.bib32" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">OpenAI (2023)</span>
<span class="ltx_bibblock">
OpenAI. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="http://arxiv.org/abs/2303.08774" title="" class="ltx_ref ltx_href">Gpt-4 technical report</a>.

</span>
</li>
<li id="bib.bib33" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Qin et al. (2023)</span>
<span class="ltx_bibblock">
Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://api.semanticscholar.org/CorpusID:256827430" title="" class="ltx_ref ltx_href">Is chatgpt a general-purpose natural language processing task solver?</a>

</span>
<span class="ltx_bibblock"><em id="bib.bib33.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/2302.06476.

</span>
</li>
<li id="bib.bib34" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Raffel et al. (2019)</span>
<span class="ltx_bibblock">
Colin Raffel, Noam M. Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://api.semanticscholar.org/CorpusID:204838007" title="" class="ltx_ref ltx_href">Exploring the limits of transfer learning with a unified text-to-text transformer</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib34.1.1" class="ltx_emph ltx_font_italic">J. Mach. Learn. Res.</em>, 21:140:1–140:67.

</span>
</li>
<li id="bib.bib35" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ratinov and Roth (2009)</span>
<span class="ltx_bibblock">
Lev-Arie Ratinov and Dan Roth. 2009.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://api.semanticscholar.org/CorpusID:1859014" title="" class="ltx_ref ltx_href">Design challenges and misconceptions in named entity recognition</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib35.1.1" class="ltx_emph ltx_font_italic">Conference on Computational Natural Language Learning</em>.

</span>
</li>
<li id="bib.bib36" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Rozière et al. (2023)</span>
<span class="ltx_bibblock">
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, I. Evtimov, Joanna Bitton, Manish P Bhatt, Cristian Cantón Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre D’efossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://api.semanticscholar.org/CorpusID:261100919" title="" class="ltx_ref ltx_href">Code llama: Open foundation models for code</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib36.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/2308.12950.

</span>
</li>
<li id="bib.bib37" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Sainz et al. (2023)</span>
<span class="ltx_bibblock">
Oscar Sainz, Iker García-Ferrero, Rodrigo Agerri, Oier Lopez de Lacalle, German Rigau, and Eneko Agirre. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://api.semanticscholar.org/CorpusID:263671572" title="" class="ltx_ref ltx_href">Gollie: Annotation guidelines improve zero-shot information-extraction</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib37.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/2310.03668.

</span>
</li>
<li id="bib.bib38" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Sarawagi and Cohen (2004)</span>
<span class="ltx_bibblock">
Sunita Sarawagi and William W. Cohen. 2004.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://api.semanticscholar.org/CorpusID:14036493" title="" class="ltx_ref ltx_href">Semi-markov conditional random fields for information extraction</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib38.1.1" class="ltx_emph ltx_font_italic">Neural Information Processing Systems</em>.

</span>
</li>
<li id="bib.bib39" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Touvron et al. (2023)</span>
<span class="ltx_bibblock">
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://api.semanticscholar.org/CorpusID:257219404" title="" class="ltx_ref ltx_href">Llama: Open and efficient foundation language models</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib39.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/2302.13971.

</span>
</li>
<li id="bib.bib40" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2023)</span>
<span class="ltx_bibblock">
Xiao Wang, Wei Zhou, Can Zu, Han Xia, Tianze Chen, Yuan Zhang, Rui Zheng, Junjie Ye, Qi Zhang, Tao Gui, Jihua Kang, J. Yang, Siyuan Li, and Chunsai Du. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://api.semanticscholar.org/CorpusID:258179792" title="" class="ltx_ref ltx_href">Instructuie: Multi-task instruction tuning for unified information extraction</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib40.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/2304.08085.

</span>
</li>
<li id="bib.bib41" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2022)</span>
<span class="ltx_bibblock">
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Gary Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Maitreya Patel, Kuntal Kumar Pal, M. Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Shailaja Keyur Sampat, Savan Doshi, Siddharth Deepak Mishra, Sujan Reddy, Sumanta Patro, Tanay Dixit, Xudong Shen, Chitta Baral, Yejin Choi, Noah A. Smith, Hannaneh Hajishirzi, and Daniel Khashabi. 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://api.semanticscholar.org/CorpusID:253098274" title="" class="ltx_ref ltx_href">Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib41.1.1" class="ltx_emph ltx_font_italic">Conference on Empirical Methods in Natural Language Processing</em>.

</span>
</li>
<li id="bib.bib42" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wei et al. (2022)</span>
<span class="ltx_bibblock">
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Huai hsin Chi, F. Xia, Quoc Le, and Denny Zhou. 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://api.semanticscholar.org/CorpusID:246411621" title="" class="ltx_ref ltx_href">Chain of thought prompting elicits reasoning in large language models</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib42.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/2201.11903.

</span>
</li>
<li id="bib.bib43" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Weischedel et al. (1996)</span>
<span class="ltx_bibblock">
Ralph Weischedel, Sean Boisen, Daniel Bikel, Robert Bobrow, Michael Crystal, William Ferguson, Allan Wechsler, and The PLUM Research Group. 1996.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.3115/1119018.1119050" title="" class="ltx_ref ltx_href">Progress in information extraction</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib43.1.1" class="ltx_emph ltx_font_italic">TIPSTER TEXT PROGRAM PHASE II: Proceedings of a Workshop held at Vienna, Virginia, May 6-8, 1996</em>, pages 127–138, Vienna, Virginia, USA. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib44" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yan et al. (2021)</span>
<span class="ltx_bibblock">
Hang Yan, Tao Gui, Junqi Dai, Qipeng Guo, Zheng Zhang, and Xipeng Qiu. 2021.

</span>
<span class="ltx_bibblock"><a target="_blank" href="http://arxiv.org/abs/2106.01223" title="" class="ltx_ref ltx_href">A unified generative framework for various ner subtasks</a>.

</span>
</li>
<li id="bib.bib45" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yang et al. (2019)</span>
<span class="ltx_bibblock">
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://api.semanticscholar.org/CorpusID:195069387" title="" class="ltx_ref ltx_href">Xlnet: Generalized autoregressive pretraining for language understanding</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib45.1.1" class="ltx_emph ltx_font_italic">Neural Information Processing Systems</em>.

</span>
</li>
<li id="bib.bib46" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ye et al. (2023)</span>
<span class="ltx_bibblock">
Junjie Ye, Xuanting Chen, Nuo Xu, Can Zu, Zekai Shao, Shichun Liu, Yuhan Cui, Zeyang Zhou, Chao Gong, Yang Shen, Jie Zhou, Siming Chen, Tao Gui, Qi Zhang, and Xuanjing Huang. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://api.semanticscholar.org/CorpusID:257632113" title="" class="ltx_ref ltx_href">A comprehensive capability analysis of gpt-3 and gpt-3.5 series models</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib46.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/2303.10420.

</span>
</li>
<li id="bib.bib47" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zamin and Oxley (2011)</span>
<span class="ltx_bibblock">
Norshuhani Zamin and Alan Oxley. 2011.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://api.semanticscholar.org/CorpusID:27060266" title="" class="ltx_ref ltx_href">Building a corpus-derived gazetteer for named entity recognition</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib47.1.1" class="ltx_emph ltx_font_italic">International Conference on Software Engineering and Computer Systems</em>.

</span>
</li>
<li id="bib.bib48" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zaratiana et al. (2023)</span>
<span class="ltx_bibblock">
Urchade Zaratiana, Nadi Tomeh, Niama Elkhbir, Pierre Holat, and Thierry Charnois. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://openreview.net/forum?id=vNrmEgfGIg3" title="" class="ltx_ref ltx_href">Filtered semi-markov CRF</a>.

</span>
</li>
<li id="bib.bib49" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zaratiana et al. (2022)</span>
<span class="ltx_bibblock">
Urchade Zaratiana, Nadi Tomeh, Pierre Holat, and Thierry Charnois. 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/2022.umios-1.1" title="" class="ltx_ref ltx_href">Named entity recognition as structured span prediction</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib49.1.1" class="ltx_emph ltx_font_italic">Proceedings of the Workshop on Unimodal and Multimodal Induction of Linguistic Structures (UM-IoS)</em>, pages 1–10, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib50" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhou et al. (2023)</span>
<span class="ltx_bibblock">
Wenxuan Zhou, Sheng Zhang, Yu Gu, Muhao Chen, and Hoifung Poon. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="http://arxiv.org/abs/2308.03279" title="" class="ltx_ref ltx_href">Universalner: Targeted distillation from large language models for open named entity recognition</a>.

</span>
</li>
</ul>
</section>
</article>
</div>
<div class="ar5iv-footer"><a href="/html/2311.08525" class="ar5iv-nav-button ar5iv-nav-button-prev">◄</a>
    <a class="ar5iv-home-button" href="/"><img height="40" alt="ar5iv homepage" src="/assets/ar5iv.png"></a>
    <a href="/feeling_lucky" class="ar5iv-text-button">Feeling<br>lucky?</a>
    <a href="/log/2311.08526" class="ar5iv-text-button ar5iv-severity-warning">Conversion<br>report</a>
    <a class="ar5iv-text-button" target="_blank" href="https://github.com/dginev/ar5iv/issues/new?template=improve-article--arxiv-id-.md&title=Improve+article+2311.08526">Report<br>an issue</a>
    <a href="https://arxiv.org/abs/2311.08526" class="ar5iv-text-button arxiv-ui-theme">View&nbsp;original<br>on&nbsp;arXiv</a><a href="/html/2311.08527" class="ar5iv-nav-button ar5iv-nav-button-next">►</a>
</div><footer class="ltx_page_footer">
<a class="ar5iv-toggle-color-scheme" href="javascript:toggleColorScheme()" title="Toggle ar5iv color scheme"><span class="color-scheme-icon"></span></a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/license" target="_blank">Copyright</a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/policies/privacy_policy" target="_blank">Privacy Policy</a>

<div class="ltx_page_logo">Generated  on Tue Feb 27 19:11:26 2024 by <a target="_blank" href="http://dlmf.nist.gov/LaTeXML/" class="ltx_LaTeXML_logo"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg==" alt="Mascot Sammy"></a>
</div></footer>
</div>

    <script>
      var canMathML = typeof(MathMLElement) == "function";
      if (!canMathML) {
        var body = document.querySelector("body");
        body.firstElementChild.setAttribute('style', 'opacity: 0;');
        var loading = document.createElement("div");
        loading.setAttribute("id", "mathjax-loading-spinner");
        var message = document.createElement("div");
        message.setAttribute("id", "mathjax-loading-message");
        message.innerText = "Typesetting Equations...";
        body.prepend(loading);
        body.prepend(message);

        var el = document.createElement("script");
        el.src = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js";
        document.querySelector("head").appendChild(el);

        window.MathJax = {
          startup: {
            pageReady: () => {
              return MathJax.startup.defaultPageReady().then(() => {
                body.removeChild(loading);
                body.removeChild(message);
                body.firstElementChild.removeAttribute('style');
              }); } } };
      }
    </script>
    <script>
    // Auxiliary function, building the preview feature when
    // an inline citation is clicked
    function clicked_cite(e) {
      e.preventDefault();
      let cite = this.closest('.ltx_cite');
      let next = cite.nextSibling;
      if (next && next.nodeType == Node.ELEMENT_NODE && next.getAttribute('class') == "ar5iv-bibitem-preview") {
        next.remove();
        return; }
      // Before adding a preview modal,
      // cleanup older previews, in case they're still open
      document.querySelectorAll('span.ar5iv-bibitem-preview').forEach(function(node) {
        node.remove();
      })

      // Create the preview
      preview = document.createElement('span');
      preview.setAttribute('class','ar5iv-bibitem-preview');
      let target = document.getElementById(this.getAttribute('href').slice(1));
      target.childNodes.forEach(function (child) {
        preview.append(child.cloneNode(true));
      });
      let close_x = document.createElement('button');
      close_x.setAttribute("aria-label","Close modal for bibliography item preview");
      close_x.textContent = "×";
      close_x.setAttribute('class', 'ar5iv-button-close-preview');
      close_x.setAttribute('onclick','this.parentNode.remove()');
      preview.append(close_x);
      preview.querySelectorAll('.ltx_tag_bibitem').forEach(function(node) {
        node.remove();
      });
      cite.parentNode.insertBefore(preview, cite.nextSibling);
      return;
    }
    // Global Document initialization:
    // - assign the preview feature to all inline citation links
    document.querySelectorAll(".ltx_cite .ltx_ref").forEach(function (link) {
      link.addEventListener("click", clicked_cite);
    });
    </script>
    </body>
</html>

2311.08526
Named Entity Recognition (NER)  is essential in various Natural Language Processing (NLP)  applications. Traditional NER models are effective but limited to a set of predefined entity types. In contrast, Large Language Models (LLMs)  can extract arbitrary entities through natural language instructions, offering greater flexibility. However, their size and cost, particularly for those accessed via APIs like ChatGPT, make them impractical in resource-limited scenarios. In this paper, we introduce a compact NER model trained to identify any type of entity. Leveraging a bidirectional transformer encoder, our model, GLiNER, facilitates parallel entity extraction, an advantage over the slow sequential token generation of LLMs. Through comprehensive testing, GLiNER demonstrate strong performance, outperforming both ChatGPT and fine-tuned LLMs in zero-shot evaluations on various NER benchmarks.
abstract
GLiNER: Generalist Model for Named Entity Recognition using Bidirectional Transformer
Abstract
2311.08526
1IntroductionNamed Entity Recognition plays a crucial role in various real-world applications, such as constructing knowledge graphs. Traditional NER models are limited to a predefined set of entity types. Expanding the number of entity types can be beneficial for many applications but may involve labeling additional datasets. The emergence of Large Language Models, like GPT-3, has introduced a new era for open-type NER by enabling the identification of any types of entity types only by natural language instruction. This shift signifies a significant departure from the inflexibility observed in traditional models. However, powerful LLMs typically consist of billions of parameters and thus require substantial computing resources. Although it is possible to access some LLMs via APIs, using them at scale can incur high costs.Figure 1:BiLM for Open NER.Previous models, such as UniNER(Fig. a) , approach the task of open type NER by prompting LLMs with natural language instructions (using a multi-turn dialogue style) . Our proposed GLiNER utilizes small bidirectional LMs and treats the NER task as matching entity types with textual spans in a latent space.Figure 2:Model architecture. GLiNER employs a BiLM and takes as input entity type prompts and a sentence/text. Each entity is separated by a learned token[ENT]. The BiLM outputs representations for each token. Entity embeddings are passed into a FeedForward Network, while input word representations are passed into a span representation layer to compute embeddings for each span. Finally, we compute a matching score between entity representations and span representations (using dot product and sigmoid activation) . For instance, in the figure, the span representation of (0, 1) , corresponding to "Alain Farley," has a high matching score with the entity embeddings of "Person".Recently, researchers have explored the fine-tuning of open source language models such as LLaMafor named entity recognition tasks., for example, introduced InstructUIE, a fine-tuned FlanT5-11Bmodel on existing information extraction datasets, achieving excellent performance in zero-shot settings. Additionally, GoLLIEwas introduced as an extension of InstructUIE work by by finetuning a CodeLLaMausing detailed annotation guidelines, resulting in significant performance improvements. Another recent proposal by, called UniversalNER, involves the fine-tuning of LLMs using diverse data from various domains annotated with ChatGPT instead of relying on standard NER datasets. Their approach enables the replication and even surpassing of the original capability of ChatGPT when evaluated in zero-shot on several NER datasets. While these works have achieved remarkable results, they present certain limitations we seek to address: They use autoregressive language models, which can be slow due to token-by-token generation; Moreover, they employ large models with several billion parameters, limiting their deployment in compute-limited scenarios. Furthurmore, as NER is treated as a text generation problem, the generation of entities is done in several decoding steps, and there is no way to perform the prediction of multiple entity types in parallel.In our work, we propose a model that addresses the above-mentioned problems. Instead of relying on large autoregressive models, we utilize smaller-scale Bidirectional Language Models (BiLM) , such as BERTor deBERTa. The core concept of our model involves treating the task of Open NER as matching entity type embeddings to textual span representations in latent space, rather than as a generation task. This approach naturally solves the scalability issues of autoregressive models and allows for bidirectional context processing, which enables richer representations. When trained on the dataset released by, which comprises texts from numerous domains and thousands of entity types, our model demonstrates impressive zero-shot performance. More specifilcally, it outperforms both ChatGPT and fine-tuned LLMs on zero-shot NER datasets (Table1) . Our model’s robustness is further evident in its ability to handle languages not even encountered during training. Specifically, our model surpasses ChatGPT in 8 of 10 languages (Table3)  that were not included in its training data.
introduction
GLiNER: Generalist Model for Named Entity Recognition using Bidirectional Transformer
1Introduction
2311.08526
2MethodThis section presents our model, GLiNER, which is train to extract any types of entity using a Bidirectional Language Models. Our model has three main components: i)  a pre-trained textual encoder (a BiLM such as BERT) , ii)  a span representation module which computes span embeddings from token embeddings, iii)  an entity representation module which computes entity embeddings that the model seeks to extract. The goal is to have entity and span embeddings in the same latent space to assess their compatibility (degree of matching) . The overall architecture of our model is depicted in Figure2.2.1ArchitectureInput formatThe input to our model comprises a unified sequence combining entity types (expressed in natural language)  and the input text from which entities are to be extracted. The input format is as follows:[ENT]token represents a special token placed before each entity type and the[SEP]token functions as a delimiter, separating the sequence of entity types from the input text. They are initialized randomly at the start of training.Token representationThe token encoder processes the unified input to compute interactions between all tokens (both entity types and input text) , producing contextualized representations. Letrepresent the encoder’s output for each entity type, corresponding to all the[ENT]token representations. Similarly,denotes the representation of each word in the input text. For words tokenized into multiple subwords, we use the representation of the first subword, which is a standard choice in the NER literature.Entity and Span RepresentationIn our model, we aim to encode entity types and span embeddings into a unified latent space. The entity representation is computed by refining the initial representationusing a two-layer feedforward network, resulting in. The representation of a span starting at positionand ending at positionin the input text,, is computed as:(1) Here,FFNdenotes a two-layer feedforward network, andrepresents the concatenation operation. In practice, The computation of all span representations can be easily parallelized. Moreover, we set an upper bound to the length (K=12)  of the span in order to keep linear complexity, without harming recall.Entity Type and Span MatchingTo evaluate whether a spancorresponds to entity type, we calculate the following matching score:(2) In this equation,denotes a sigmoid activation function. As we train with binary cross-entropy loss (see next sec.2.2) ,can be interpreted as the probability of the spanbeing of type.ModelParamsMovieRestaurantAILiteratureMusicPoliticsScienceAverageVicuna-7B7B6.05.312.816.117.020.513.013.0Vicuna-13B13B0.90.422.722.726.627.022.017.5USM0.3B37.717.728.256.044.936.144.037.8ChatGPT–5.332.852.439.866.668.567.047.5InstructUIE11B63.021.049.047.253.248.149.247.2UniNER-7B7B42.431.753.659.367.060.961.153.7UniNER-13B13B48.736.254.260.964.561.463.555.6GoLLIE7B63.043.459.162.767.857.255.558.0GLiNER-S50M46.933.350.760.060.961.555.652.7GLiNER-M90M42.937.351.859.769.468.658.155.4GLiNER-L0.3B57.242.957.264.469.672.662.660.9Table 1:Zero-Shot Scores on Out-of-Domain NER Benchmark.We report the performance of GLiNER with various DeBERTa-v3model sizes. Results for Vicuna, ChatGPT, and UniNER are from; USM and InstructUIE are from; and GoLLIE is from.2.2TrainingDuring training, our objective is to optimize model parameters to enhance the matching score for correct span-type pairs (positive pairs)  and reduce it for incorrect pairs (negative pairs) . A spanpaired with an entity typeforms a positive pair ()  if the span is labeled with typein the training data. Otherwise, it is a negative pair () . The training loss for an individual example, comprising spansand entity types, is defined as:(3) The variablerepresents a pair of span/entity type andis an indicator function, which returns 1 when the specified condition is true and 0 otherwise. This loss function corresponds to binary cross-entropy.2.3Decoding algorithmIn the decoding phase, we employ a greedy span sectionthat selects entity spans based on matching scores, to ensure task/dataset specific constraints. This strategy is applied independently to each sentence. Only, spanswith matching scoresare considered for selection.Flat NER:The algorithm chooses the highest-scoring non-overlapping span and continues this process until all spans are evaluated.Nested NER:Similar to Flat NER, but the algorithm allows selection of fully nested spans within other entities while still avoiding partial overlaps.Algorithm Efficiency:The decoding is implemented using a priority queue for spans, ensuring ancomplexity, withbeing the number of candidate spans.
body
GLiNER: Generalist Model for Named Entity Recognition using Bidirectional Transformer
2Method
2311.08526
3Experimental Setting3.1Training dataOur objective is to construct a versatile NER model capable of accurately identifying a wide array of entity types across different textual domains. To achieve this, it is essential that our training dataset encompasses a diverse range of entity types. For this, we utilize the training data released by, known as Pile-NER111https://huggingface.co/datasets/Universal-NER/Pile-NER-type. This dataset is derived from the Pile corpus, commonly used for pretraining large language models, and comprises text from diverse sources. More specifically,sampled 50,000 texts from the Pile data and employed ChatGPT to extract their associated entity types. Notably, they did not specify the entity types to the LLMs, aiming to extract a diverse range of entity types. They used the following prompt:Figure 3:Prompting ChatGPT for entity extraction. This prompt was usedto construct the Pile-NER dataset.Finally, after filtering bad outputs their datasets results in 44889 passages containing in total 240k entity spans and 13k distinct entity types.3.2HyperparametersOur model, GLiNER, is trained on the Pile-NER dataset, which we described in the previous section. We use the deBERTa-v3as our backbone due to its proven empirical performance. All non-pretrained layers have a width dimension of 768 and a dropout rate of 0.4. Regarding the training process, we employ the AdamW optimizer, setting a base learning rate of 1e-5 for pretrained layers (the transformer backbone)  and 5e-5 for non-pretrained layers (FFN layers, span representation) . We trained our models for a maximum of 30k steps, starting with a 10% warmup phase, followed by a decay phase using a cosine scheduler.
The Pile-NER dataset natively contains only positive entities (i.e., entities that are present in the sentence) , and we found it useful to include negative entity types during training. This is achieved by sampling random entities from other examples in the same batch. In addition, we follow the strategies outlined inas a form of regularization, which includesshuffling entity orderandrandomly dropping entities. Furthermore, we limit the number of entity types to 25 per sentence during training. The larger variant of our model, GLiNER-L, takes 5 hours to train on an A100 GPU.3.3BaselinesIn our evaluation, we compare our model, GLiNER, with several recent models designed for open-type NER. First, we examine chat models likeChatGPTandVicuna, which utilize the prompting from; we report their results reported by. Next, we focus on three recent Large Language Models (LLMs)  that have been fine-tuned for NER:InstructUIE, based on the FlanT5 11B model and fine-tuned on various NER datasets;UniNER, which employs a LLaMa model fine-tuned on a dataset generated by ChatGPT;GoLLIE, fine-tuned to adhere to detailed annotation guidelines for enhanced performance in unseen IE tasks, utilizing CodeLLama as its base model. Finally, we includeUSMin our comparison, which is similar in size to ours but features a different architecture.3.4EvaluationDatasetsWe primarily evaluate our model in a zero-shot context on common NER benchmarks, following previous works. The first is theOOD NER Benchmark(Table1) , which comprises seven diverse NER datasets from CrossNER and MIT. This benchmark is typically used for evaluating out-of-domain generalization capabilities of NER models. The second benchmark consists of20 NER datasets(Table2)  from a wide range of domains, including biomedical, news articles, and tweets. These datasets are commonly used for training supervised NER models. Additionally, we evaluate our model on multilingual NER datasets (Table3)  for further investigation. For this purpose, we use the recently releasedMulticoner(Multilingual Complex NER) , which contains data in 11 languages across various domains.MetricWe adopt the standard NER evaluation methodology, calculating F1-score based on the exact match between predicted and actual entities.DatasetChatGPTUniNER-7BGLiNER-LACE0526.636.927.3AnatEM30.725.133.3bc2gm40.246.247.9bc4chemd35.547.943.1bc5cdr52.468.066.4Broad Tweeter61.867.961.2CoNLL0352.572.264.6FabNER15.324.823.6FindVehicle10.522.241.9GENIA41.654.155.5HarveyNER11.618.222.7MIT Movie5.342.457.2MIT Restaurant32.831.742.9MultiNERD58.159.359.7ncbi42.160.461.9OntoNotes29.727.832.2PolyglotNER33.641.842.9TweetNER740.142.741.4WikiANN52.055.458.9WikiNeural57.769.271.8Average36.545.747.8Table 2:Zero-shot performance on 20 NER datasets.Results of ChatGPT and UniNER are reported from.
body
GLiNER: Generalist Model for Named Entity Recognition using Bidirectional Transformer
3Experimental Setting
2311.08526
4Results4.1Zero-shot on English datasetsIn this section, we discuss the performance of our model in a zero-shot context, i.e., by only training on the Pile-NER dataset without further fine-tuning on target datasets.OOD NER BenchmarkWe first evaluate our model on the OOD benchmark as reported in Table1. We compare three different sizes of our model (small, medium, and large)  against the baselines. The results demonstrate our model’s impressive capability, irrespective of its size. For example, even our smallest model, with only 50M parameters, outperforms general-purpose models such as ChatGPT and Vicuna. It also shows better performance than the 11B InstructUIE, which has been instruction-tuned for the NER task. Furthermore, when compared to UniNER, which used the same training data as GLiNER, our medium-sized model (90M)  achieves comparable results to UniNER-13B (55 F1 for both) , despite being 140 times smaller, while our largest version outperforms it by an average of 5 points. Our most best competitor, GoLLIE, which leads among the LLMs, achieves better performance than most of our models but is still less effective than GLiNER-L. Against USM, which has a comparable number of parameters to ours, the performance is significantly lower, highlighting the superiority of our architecture.20 NER BenchmarkTable 2 presents a comparison of our model against ChatGPT and UniNER across 20 diverse NER datasets. First, similar to the OOD benchmark, ChatGPT significantly lags behind fine-tuned models for NER, trailing behind UniNER. Furthermore, GLiNER achieves the highest performance on 13 of these datasets, surpassing UniNER by an average of 2 points. This superior performance underscores GLiNER’s robustness and adaptability across a broad spectrum of domains. However, a notable observation is that GLiNER underperforms compared to UniNER on tweet-based NER datasets. This highlights potential areas for improvement in GLiNER’s ability to process informal, colloquial, or noisy data, typical of social media content.LanguageSup.ChatGPTGLiNEREnMultiLatinGerman64.637.135.639.5English62.737.242.441.7Spanish58.734.738.742.1Dutch62.635.735.638.9Non-LatinBengali39.723.30.8925.9Persian52.325.914.930.2Hindi47.827.311.327.8Korean55.830.020.528.7Russian59.727.430.333.3Turkish46.831.922.030.0Chinese53.118.86.5924.3Average54.929.923.632.9Table 3:Zero-Shot Scores on Different Languages.The baseline,Sup., is an XLM-Rmodel fine-tuned on the training set of each language separately, as reported by. ChatGPT evaluation is taken from. GLiNER-En employs deBERTa-v3-Large, and Multi uses mdeBERTa-v3-base.4.2Zero-Shot Multilingual EvaluationIn this section, we evaluate the performance of our model in a zero-shot context on unseen languages to assess its generalizability. This evaluation uses the Multiconel dataset, with results detailed in Table 3. Our model, GLiNER, is presented in two variants:En, which employs deBERTa-v3-Large as its backbone, andMulti, which utilizes a multilingual version of deBERTa-v3 (mdeBERTa-v3) . Both versions were fine-tuned on the Pile-NER dataset. For comparative purposes, we include results from ChatGPT and a supervised baseline, the latter being fine-tuned on the training set of each dataset using separate models.ResultsAs expected, the supervised baseline demonstrated superior performance, significantly outperforming the zero-shot models. Among these, GLiNER-Multi showed the most promising results, surpassing ChatGPT in most languages, which is noteworthy considering that the fine-tuning dataset, Pile-NER, consists exclusively of English examples. Interestingly, its performance on Spanish data slightly exceeded that in English. While GLiNER-En generally underperformed compared to ChatGPT on average, it achieved competitive, and at times superior, results in languages using the Latin script, such as Spanish and German. However, its performance was markedly less competitive in non-Latin languages, particularly in Bengali, where it scored only 0.89 in the F1 score.DatasetInstructUIEUniNER-7BGLiNER-Lw/ow/w/w/oACE0579.986.782.881.3AnatEM88.588.588.988.4bc2gm80.782.483.782.0bc4chemd87.689.287.986.7bc5cdr89.089.388.788.7Broad Twitter80.381.282.582.7CoNLL0391.593.392.692.5FabNER78.481.977.874.8FindVehicle87.698.395.795.2GENIA75.777.578.977.4HarveyNER74.774.268.667.4MIT Movie89.690.287.987.5MIT Restaurant82.682.383.683.3MultiNERD90.393.793.893.3ncbi86.287.087.887.1OntoNotes88.689.989.088.1PolyglotNER53.365.761.560.6TweetNER765.965.851.450.3WikiANN64.584.983.782.8wikiNeural88.393.391.391.4Average81.284.882.982.1Table 4:In-domain Supervised Finetuning.All the models are fine-tuned on the mix of all training data of the benchmark.w/indicates that the model was trained on the Pile-NER dataset before finetuning.4.3In-domain Supervised tuningIn our work, we also perform in-domain supervised fine-tuning (on 20 NER datasets)  of our model to compare its capabilities against LLMs under this setup. Specifically, we compare our model against InstructUIE and UniNER, both of which have also been fine-tuned. The main difference is that UniNER has been pre-trained on the Pile-NER dataset before fine-tuning.Training SetupFor the supervised setting, we primarily adhere to the same experimental setup as described in the main experiment (using deBERTa-v3 large) , except for the training dataset. Regarding the training data, we follow the approach of InstructUIE: we randomly sample 10,000 data points for each dataset in the 20 NER benchmark. If a dataset does not contain 10,000 samples, we include all available data. We implement two variants of our model: the first one initializes the weights from our zero-shot model, which is a pretrained on the Pile-NER dataset. The second variant is trained without the Pile-NER dataset, same as InstructUIE.ResultsThe results of our experiment are reported in Table 3. Firstly, we observe that for the in-domain fine-tuning, our GLiNER model, pretrained on Pile-NER, achieves slightly better results than the non-pretrained variant, with an average difference of 0.8. Moreover, our pretrained GLiNER model outperforms InstructUIE (with an average difference of 0.9)  despite being fine-tuned on the same dataset, whereas InstructUIE is significantly larger (approximately 30 times so) . This demonstrates that our proposed architecture is indeed competitive. However, our model falls behind UniNER by almost 3 points. Nevertheless, our model still manages to achieve the best score in 7 out of 20 datasets.Figure 4:Zero-shot performance for different backbones.It reports the avg. results on 20 NER and OOD NER datasets
body
GLiNER: Generalist Model for Named Entity Recognition using Bidirectional Transformer
4Results
2311.08526
5Further analysis and ablationsHere we conduct different set of experiments to better investigate our model.5.1Effect of Different BackbonesIn our work, we primarily utilize the deBERTa-v3 model as our backbone due to its strong empirical performance. However, we demonstrate here that our method is adaptable to a wide range of BiLMs.SetupSpecifically, we investigate the performance of our model using other popular BiLMs, including BERT, RoBERTa, AlBERT, and ELECTRA. We also conducted experiments with XLNetbut did not achieve acceptable performance (achieving at most 3 F1 on the OOD benchmark)  despite extensive hyperparameter tuning. For a fair comparison, we employed the base size (GLiNER-M)  and tuned the learning rate for each model. We report the zero-shot results on both the OOD benchmark and the 20 NER benchmark in Figure4.ResultThe results of our experiment, as shown in the Figure4, clearly demonstrate the superiority of deBERTa-v3 over other pretrained BiLMs. It achieves the highest performance on both benchmarks by a clear margin. ELECTRA and AlBERT also show notable performance, albeit slightly lower, while BERT and RoBERTa lag behind with similar scores. However, it should be noted that all of the backbones we tested demonstrate strong performance compared to existing models. More specifically, even BERT-base, which ranks among the lower performers, achieves around 49 F1 on the OOD benchmark. This score is still 2 F1 points higher than the average for models like ChatGPT and InstructUIE.5.2Effect of Pretraining on In-domain PerformanceIn this section, we investigate the impact of pretraining on the Pile-NER dataset for supervised in-domain training on the 20 NER datasets, across various data sizes. The experiments range from 100 samples per dataset to 10,000 (full training setup) . We use the same hyperparameters for all configurations. The results are reported in Figure5.ResultsAs shown in the figure, models pretrained on Pile-NER consistently outperform their counterparts that are only trained on supervised data, indicating successful positive transfer. We further observe that the gain is larger when supervised data is limited. For instance, the difference in performance is 5.6 when employing 100 samples per dataset, and the gap becomes smaller as the size of the dataset increases.Figure 5:Supervised performance across different dataset sizes.The evaluation is conducted on the 20 NER datasets (in table4) .Negative SamplesPrecRecF10%49.358.153.350%62.359.760.975%61.156.558.6Table 5:Effect of negative entity types sampling.5.3AblationsNegative Entity SamplingThe original Pile-NER dataset, curated by, features passages with positive entity instances, i.e., entities that are directly present in the text. To better align training with real-world scenarios, where some entity types might be absent, we implemented negative entity sampling as mentioned in Section3.2. In this study, we evaluate different sampling ratios: 0% (only positive entities) , 50%, and 75% negative entities.
Table5shows that training with only positive entities results in lower precision but higher recall, indicating that the model often makes false positive errors. Conversely, using 75% negative entities increases precision but decreases recall, as the abundance of negatives makes the model more cautious, leading to missed correct entities. A 50% negative entity ratio proves to be the most effective, providing a balanced approach.Entity type droppingIn the experiment, we employed a strategy of randomly varying the number of entity prompts during training. This approach aimed to expose the model to different quantities of entity types in each training instance, thereby increasing its adaptability to handle scenarios with varying numbers of entities. The usage of this technique results in an average improvement of over 1.4 points in out-of-domain evaluation, as shown in the Figure 5.
body
GLiNER: Generalist Model for Named Entity Recognition using Bidirectional Transformer
5Further analysis and ablations
2311.08526
6Related WorksNamed Entity RecognitionNER is a well-established task in the field of NLP, with numerous applications. Initially, NER models relied on rule-based systemthat were built using handcrafted algorithms and gazetteers. However, these models had limitations in terms of scalability and adaptability to new domains or languages. To overcome these issues, machine learning approaches have been proposed. In the early stages, NER tasks were designed as sequence labelingwhere the objective was to predict tagged sequences (e.g., BILOU tags) . Since then, several paradigm shifts have occurred: span-based approaches treating NER as span classification; NER being treated as a question answering problem; and even as a generation task.Zero-shot learning for NERThe advent of large-scale autoregressive models has recently transformed many paradigms in NLP through natural language prompting. This is also the case for NER. Others have fine-tuned these models for tasks to better align their capabilities with the requirements of entity recognitionor information extraction in general. This is done through-instruction tuning.Figure 6:Randomly dropping entity types.We report the results with and without negative entity sampling.
body
GLiNER: Generalist Model for Named Entity Recognition using Bidirectional Transformer
6Related Works
2311.08526
7ConclusionIn this paper, we introduced GLiNER, a new method for identifying various types of entities in text using bidirectional language models. Our model not only outperforms state-of-the-art Large Language Models like ChatGPT in zero-shot scenarios but also offers a more resource-efficient alternative, crucial for environments with limited computing power. GLiNER is versatile, performing well in multiple languages, including those it wasn’t trained on. In future work, we aim to further improve GLiNER’s design for enhanced performance and to better adapt it for low-resource languages.
conclusion
GLiNER: Generalist Model for Named Entity Recognition using Bidirectional Transformer
7Conclusion