<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">

<head>  <title>[2408.07852] Training Language Models on the Knowledge Graph: Insights on Hallucinations and Their Detectability</title>
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <link rel="apple-touch-icon" sizes="180x180" href="/static/browse/0.3.4/images/icons/apple-touch-icon.png">
  <link rel="icon" type="image/png" sizes="32x32" href="/static/browse/0.3.4/images/icons/favicon-32x32.png">
  <link rel="icon" type="image/png" sizes="16x16" href="/static/browse/0.3.4/images/icons/favicon-16x16.png">
  <link rel="manifest" href="/static/browse/0.3.4/images/icons/site.webmanifest">
  <link rel="mask-icon" href="/static/browse/0.3.4/images/icons/safari-pinned-tab.svg" color="#5bbad5">
  <meta name="msapplication-TileColor" content="#da532c">
  <meta name="theme-color" content="#ffffff">
  <link rel="stylesheet" type="text/css" media="screen" href="/static/browse/0.3.4/css/arXiv.css?v=20240822" />
  <link rel="stylesheet" type="text/css" media="print" href="/static/browse/0.3.4/css/arXiv-print.css?v=20200611" />
  <link rel="stylesheet" type="text/css" media="screen" href="/static/browse/0.3.4/css/browse_search.css" />
  <script language="javascript" src="/static/browse/0.3.4/js/accordion.js" /></script>
  
    <link rel="stylesheet" type="text/css" media="screen" href="/static/browse/0.3.4/css/slider.css?v=1.14" />
    <script src="//code.jquery.com/jquery-latest.min.js" type="text/javascript"></script>
    <script type="text/javascript" src="/static/browse/0.3.4/js/donate.js?v=1.11"></script><link rel="canonical" href="/abs/2408.07852"/><meta property="og:type" content="website" />
<meta property="og:site_name" content="arXiv.org" />
<meta property="og:title" content="Training Language Models on the Knowledge Graph: Insights on Hallucinations and Their Detectability" />
<meta property="og:url" content="https://arxiv.org/abs/2408.07852v1" />
<meta property="og:image" content="/static/browse/0.3.4/images/arxiv-logo-fb.png" />
<meta property="og:image:secure_url" content="/static/browse/0.3.4/images/arxiv-logo-fb.png" />
<meta property="og:image:width" content="1200" />
<meta property="og:image:height" content="700" />
<meta property="og:image:alt" content="arXiv logo"/>
<meta property="og:description" content="While many capabilities of language models (LMs) improve with increased training budget, the influence of scale on hallucinations is not yet fully understood. Hallucinations come in many forms, and there is no universally accepted definition. We thus focus on studying only those hallucinations where a correct answer appears verbatim in the training set. To fully control the training data content, we construct a knowledge graph (KG)-based dataset, and use it to train a set of increasingly large LMs. We find that for a fixed dataset, larger and longer-trained LMs hallucinate less. However, hallucinating on $\leq5$% of the training data requires an order of magnitude larger model, and thus an order of magnitude more compute, than Hoffmann et al. (2022) reported was optimal. Given this costliness, we study how hallucination detectors depend on scale. While we see detector size improves performance on fixed LM&#39;s outputs, we find an inverse relationship between the scale of the LM and the detectability of its hallucinations."/>
<meta name="twitter:site" content="@arxiv"/>
<meta name="twitter:card" content="summary"/>
<meta name="twitter:title" content="Training Language Models on the Knowledge Graph: Insights on..."/>
<meta name="twitter:description" content="While many capabilities of language models (LMs) improve with increased training budget, the influence of scale on hallucinations is not yet fully understood. Hallucinations come in many forms,..."/>
<meta name="twitter:image" content="https://static.arxiv.org/icons/twitter/arxiv-logo-twitter-square.png"/>
<meta name="twitter:image:alt" content="arXiv logo"/>
  <link rel="stylesheet" media="screen" type="text/css" href="/static/browse/0.3.4/css/tooltip.css"/><link rel="stylesheet" media="screen" type="text/css" href="https://static.arxiv.org/js/bibex-dev/bibex.css?20200709"/>  <script src="/static/browse/0.3.4/js/mathjaxToggle.min.js" type="text/javascript"></script>  <script src="//code.jquery.com/jquery-latest.min.js" type="text/javascript"></script>
  <script src="//cdn.jsdelivr.net/npm/js-cookie@2/src/js.cookie.min.js" type="text/javascript"></script>
  <script src="//cdn.jsdelivr.net/npm/dompurify@2.3.5/dist/purify.min.js"></script>
  <script src="/static/browse/0.3.4/js/toggle-labs.js?20230622" type="text/javascript"></script>
  <script src="/static/browse/0.3.4/js/cite.js" type="text/javascript"></script><meta name="citation_title" content="Training Language Models on the Knowledge Graph: Insights on Hallucinations and Their Detectability" /><meta name="citation_author" content="Hron, Jiri" /><meta name="citation_author" content="Culp, Laura" /><meta name="citation_author" content="Elsayed, Gamaleldin" /><meta name="citation_author" content="Liu, Rosanne" /><meta name="citation_author" content="Adlam, Ben" /><meta name="citation_author" content="Bileschi, Maxwell" /><meta name="citation_author" content="Bohnet, Bernd" /><meta name="citation_author" content="Co-Reyes, JD" /><meta name="citation_author" content="Fiedel, Noah" /><meta name="citation_author" content="Freeman, C. Daniel" /><meta name="citation_author" content="Gur, Izzeddin" /><meta name="citation_author" content="Kenealy, Kathleen" /><meta name="citation_author" content="Lee, Jaehoon" /><meta name="citation_author" content="Liu, Peter J." /><meta name="citation_author" content="Mishra, Gaurav" /><meta name="citation_author" content="Mordatch, Igor" /><meta name="citation_author" content="Nova, Azade" /><meta name="citation_author" content="Novak, Roman" /><meta name="citation_author" content="Parisi, Aaron" /><meta name="citation_author" content="Pennington, Jeffrey" /><meta name="citation_author" content="Rizkowsky, Alex" /><meta name="citation_author" content="Simpson, Isabelle" /><meta name="citation_author" content="Sedghi, Hanie" /><meta name="citation_author" content="Sohl-dickstein, Jascha" /><meta name="citation_author" content="Swersky, Kevin" /><meta name="citation_author" content="Vikram, Sharad" /><meta name="citation_author" content="Warkentin, Tris" /><meta name="citation_author" content="Xiao, Lechao" /><meta name="citation_author" content="Xu, Kelvin" /><meta name="citation_author" content="Snoek, Jasper" /><meta name="citation_author" content="Kornblith, Simon" /><meta name="citation_date" content="2024/08/14" /><meta name="citation_online_date" content="2024/08/14" /><meta name="citation_pdf_url" content="http://arxiv.org/pdf/2408.07852" /><meta name="citation_arxiv_id" content="2408.07852" /><meta name="citation_abstract" content="While many capabilities of language models (LMs) improve with increased training budget, the influence of scale on hallucinations is not yet fully understood. Hallucinations come in many forms, and there is no universally accepted definition. We thus focus on studying only those hallucinations where a correct answer appears verbatim in the training set. To fully control the training data content, we construct a knowledge graph (KG)-based dataset, and use it to train a set of increasingly large LMs. We find that for a fixed dataset, larger and longer-trained LMs hallucinate less. However, hallucinating on $\leq5$% of the training data requires an order of magnitude larger model, and thus an order of magnitude more compute, than Hoffmann et al. (2022) reported was optimal. Given this costliness, we study how hallucination detectors depend on scale. While we see detector size improves performance on fixed LM&#39;s outputs, we find an inverse relationship between the scale of the LM and the detectability of its hallucinations." />
</head>

<body  class="with-cu-identity">
  
  <aside class="slider-wrapper bps-banner forum grey">
    <a class="close-slider bps-banner" href="#"><img src="/static/browse/0.3.4/images/icons/close-slider.png" alt="close this message"></a>
  	<div class="columns">
      <img role="presentation" class="bps-banner-image" src="/static/browse/0.3.4/images/events/arxiv-forum-logotype-title.png" alt="arXiv Accessibility Forum 2024">
      <div class="copy-donation bps-banner">
        <h2 id="forum-session-title"></h2>
        <p id="forum-session-link-small"></p>
      </div>
      <div class="amount-donation bps-banner">
          <div id="forum-session-link-box" class="donate-cta"></div>
      </div>
    </div>
  </aside>
  <script src="/static/browse/0.3.4/js/rotate_forum_info.js" type="text/javascript"></script>
  <div class="flex-wrap-footer">
    <header>
      <a href="#content" class="is-sr-only">Skip to main content</a>
      <!-- start desktop header -->
      <div class="columns is-vcentered is-hidden-mobile" id="cu-identity">
        <div class="column" id="cu-logo">
          <a href="https://www.cornell.edu/"><img src="/static/browse/0.3.4/images/icons/cu/cornell-reduced-white-SMALL.svg" alt="Cornell University" /></a>
        </div><div class="column banner-minimal forum">
            <!-- /from July 7 at 1:00 AM to Sept 2 at 1:00 AM --><p>This week: the arXiv Accessibility Forum</p>
            <a href="https://accessibility2024.arxiv.org/schedule" target="_blank">Forum Schedule</a></div><div class="column" id="support-ack">
          <span id="support-ack-url">We gratefully acknowledge support from the Simons Foundation, <a href="https://info.arxiv.org/about/ourmembers.html">member institutions</a>, and all contributors.</span>
          <a href="https://info.arxiv.org/about/donate.html" class="btn-header-donate">Donate</a>
        </div>
      </div>

      <div id="header" class="is-hidden-mobile">
<a aria-hidden="true" href="{url_path('ignore_me')}"></a>
  <div class="header-breadcrumbs is-hidden-mobile">
    <a href="/"><img src="/static/browse/0.3.4/images/arxiv-logo-one-color-white.svg" alt="arxiv logo" style="height:40px;"/></a> <span>&gt;</span> <a href="/list/cs/recent">cs</a> <span>&gt;</span> arXiv:2408.07852
  </div>
          <div class="search-block level-right">
    <form class="level-item mini-search" method="GET" action="https://arxiv.org/search">
      <div class="field has-addons">
        <div class="control">
          <input class="input is-small" type="text" name="query" placeholder="Search..." aria-label="Search term or terms" />
          <p class="help"><a href="https://info.arxiv.org/help">Help</a> | <a href="https://arxiv.org/search/advanced">Advanced Search</a></p>
        </div>
        <div class="control">
          <div class="select is-small">
            <select name="searchtype" aria-label="Field to search">
              <option value="all" selected="selected">All fields</option>
              <option value="title">Title</option>
              <option value="author">Author</option>
              <option value="abstract">Abstract</option>
              <option value="comments">Comments</option>
              <option value="journal_ref">Journal reference</option>
              <option value="acm_class">ACM classification</option>
              <option value="msc_class">MSC classification</option>
              <option value="report_num">Report number</option>
              <option value="paper_id">arXiv identifier</option>
              <option value="doi">DOI</option>
              <option value="orcid">ORCID</option>
              <option value="author_id">arXiv author ID</option>
              <option value="help">Help pages</option>
              <option value="full_text">Full text</option>
            </select>
          </div>
        </div>
        <input type="hidden" name="source" value="header">
        <button class="button is-small is-cul-darker">Search</button>
      </div>
    </form>
  </div>
     </div><!-- /end desktop header -->

      <div class="mobile-header">
        <div class="columns is-mobile">
          <div class="column logo-arxiv"><a href="https://arxiv.org/"><img src="/static/browse/0.3.4/images/arxiv-logomark-small-white.svg" alt="arXiv logo" style="height:60px;" /></a></div>
          <div class="column logo-cornell"><a href="https://www.cornell.edu/">
            <picture>
              <source media="(min-width: 501px)"
                srcset="/static/browse/0.3.4/images/icons/cu/cornell-reduced-white-SMALL.svg  400w"
                sizes="400w" />
              <source srcset="/static/browse/0.3.4/images/icons/cu/cornell_seal_simple_black.svg 2x" />
              <img src="/static/browse/0.3.4/images/icons/cu/cornell-reduced-white-SMALL.svg" alt="Cornell University Logo" />
            </picture>
          </a></div>
          <div class="column nav" id="toggle-container" role="menubar">
            <button class="toggle-control"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512" class="icon filter-white"><title>open search</title><path d="M505 442.7L405.3 343c-4.5-4.5-10.6-7-17-7H372c27.6-35.3 44-79.7 44-128C416 93.1 322.9 0 208 0S0 93.1 0 208s93.1 208 208 208c48.3 0 92.7-16.4 128-44v16.3c0 6.4 2.5 12.5 7 17l99.7 99.7c9.4 9.4 24.6 9.4 33.9 0l28.3-28.3c9.4-9.4 9.4-24.6.1-34zM208 336c-70.7 0-128-57.2-128-128 0-70.7 57.2-128 128-128 70.7 0 128 57.2 128 128 0 70.7-57.2 128-128 128z"/></svg></button>
            <div class="mobile-toggle-block toggle-target">
              <form class="mobile-search-form" method="GET" action="https://arxiv.org/search">
                <div class="field has-addons">
                  <input class="input" type="text" name="query" placeholder="Search..." aria-label="Search term or terms" />
                  <input type="hidden" name="source" value="header">
                  <input type="hidden" name="searchtype" value="all">
                  <button class="button">GO</button>
                </div>
              </form>
            </div>

            <button class="toggle-control"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512" class="icon filter-white" role="menu"><title>open navigation menu</title><path d="M16 132h416c8.837 0 16-7.163 16-16V76c0-8.837-7.163-16-16-16H16C7.163 60 0 67.163 0 76v40c0 8.837 7.163 16 16 16zm0 160h416c8.837 0 16-7.163 16-16v-40c0-8.837-7.163-16-16-16H16c-8.837 0-16 7.163-16 16v40c0 8.837 7.163 16 16 16zm0 160h416c8.837 0 16-7.163 16-16v-40c0-8.837-7.163-16-16-16H16c-8.837 0-16 7.163-16 16v40c0 8.837 7.163 16 16 16z"/ ></svg></button>
            <div class="mobile-toggle-block toggle-target">
              <nav class="mobile-menu" aria-labelledby="mobilemenulabel">
                <h2 id="mobilemenulabel">quick links</h2>
                <ul>
                    <li><a href="https://arxiv.org/login">Login</a></li>
                    <li><a href="https://info.arxiv.org/help">Help Pages</a></li>
                    <li><a href="https://info.arxiv.org/about">About</a></li>
                </ul>
              </nav>
            </div>
          </div>
        </div>
      </div><!-- /end mobile-header -->
    </header>

    <main>
      <div id="content">
<!--
rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
         xmlns:dc="http://purl.org/dc/elements/1.1/"
         xmlns:trackback="http://madskills.com/public/xml/rss/module/trackback/">
    <rdf:Description
        rdf:about="/abs/2408.07852"
        dc:identifier="/abs/2408.07852"
        dc:title="Training Language Models on the Knowledge Graph: Insights on Hallucinations and Their Detectability"
        trackback:ping="/trackback/2408.07852" />
    </rdf:RDF>
--><div id="abs-outer">

  <div class="leftcolumn">
    <div class="subheader">
      <h1>Computer Science > Computation and Language</h1>
    </div>

    <div class="header-breadcrumbs-mobile">
      <strong>arXiv:2408.07852</strong> (cs)
    </div>
<link rel="stylesheet" type="text/css" href="/static/base/1.0.1/css/abs.css">
<div id="content-inner">
  <div id="abs">
    <div class="dateline">
  [Submitted on 14 Aug 2024]</div>
    <h1 class="title mathjax"><span class="descriptor">Title:</span>Training Language Models on the Knowledge Graph: Insights on Hallucinations and Their Detectability</h1>
    <div class="authors"><span class="descriptor">Authors:</span><a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Hron,+J">Jiri Hron</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Culp,+L">Laura Culp</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Elsayed,+G">Gamaleldin Elsayed</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Liu,+R">Rosanne Liu</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Adlam,+B">Ben Adlam</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Bileschi,+M">Maxwell Bileschi</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Bohnet,+B">Bernd Bohnet</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Co-Reyes,+J">JD Co-Reyes</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Fiedel,+N">Noah Fiedel</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Freeman,+C+D">C. Daniel Freeman</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Gur,+I">Izzeddin Gur</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Kenealy,+K">Kathleen Kenealy</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Lee,+J">Jaehoon Lee</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Liu,+P+J">Peter J. Liu</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Mishra,+G">Gaurav Mishra</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Mordatch,+I">Igor Mordatch</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Nova,+A">Azade Nova</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Novak,+R">Roman Novak</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Parisi,+A">Aaron Parisi</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Pennington,+J">Jeffrey Pennington</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Rizkowsky,+A">Alex Rizkowsky</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Simpson,+I">Isabelle Simpson</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Sedghi,+H">Hanie Sedghi</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Sohl-dickstein,+J">Jascha Sohl-dickstein</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Swersky,+K">Kevin Swersky</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Vikram,+S">Sharad Vikram</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Warkentin,+T">Tris Warkentin</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Xiao,+L">Lechao Xiao</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Xu,+K">Kelvin Xu</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Snoek,+J">Jasper Snoek</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Kornblith,+S">Simon Kornblith</a></div>            <div id="download-button-info" hidden>View a PDF of the paper titled Training Language Models on the Knowledge Graph: Insights on Hallucinations and Their Detectability, by Jiri Hron and 30 other authors</div>
    <a class="mobile-submission-download" href="/pdf/2408.07852">View PDF</a>
    <a class="mobile-submission-download" href="https://arxiv.org/html/2408.07852v1">HTML (experimental)</a>



    <blockquote class="abstract mathjax">
            <span class="descriptor">Abstract:</span>While many capabilities of language models (LMs) improve with increased training budget, the influence of scale on hallucinations is not yet fully understood. Hallucinations come in many forms, and there is no universally accepted definition. We thus focus on studying only those hallucinations where a correct answer appears verbatim in the training set. To fully control the training data content, we construct a knowledge graph (KG)-based dataset, and use it to train a set of increasingly large LMs. We find that for a fixed dataset, larger and longer-trained LMs hallucinate less. However, hallucinating on $\leq5$% of the training data requires an order of magnitude larger model, and thus an order of magnitude more compute, than Hoffmann et al. (2022) reported was optimal. Given this costliness, we study how hallucination detectors depend on scale. While we see detector size improves performance on fixed LM&#39;s outputs, we find an inverse relationship between the scale of the LM and the detectability of its hallucinations.
    </blockquote>

    <!--CONTEXT-->
    <div class="metatable">
      <table summary="Additional metadata">        <tr>
          <td class="tablecell label">Comments:</td>
          <td class="tablecell comments mathjax">Published at COLM 2024. 16 pages, 11 figures</td>
        </tr>
<tr>
          <td class="tablecell label">Subjects:</td>
          <td class="tablecell subjects">
            <span class="primary-subject">Computation and Language (cs.CL)</span>; Artificial Intelligence (cs.AI); Machine Learning (cs.LG)</td>
        </tr><tr>
          <td class="tablecell label">Cite as:</td>
          <td class="tablecell arxivid"><span class="arxivid"><a href="https://arxiv.org/abs/2408.07852">arXiv:2408.07852</a> [cs.CL]</span></td>
        </tr>
        <tr>
          <td class="tablecell label">&nbsp;</td>
          <td class="tablecell arxividv">(or <span class="arxivid">
              <a href="https://arxiv.org/abs/2408.07852v1">arXiv:2408.07852v1</a> [cs.CL]</span> for this version)
          </td>
        </tr>
        <tr>
          <td class="tablecell label">&nbsp;</td>
          <td class="tablecell arxivdoi">              <a href="https://doi.org/10.48550/arXiv.2408.07852"  id="arxiv-doi-link">https://doi.org/10.48550/arXiv.2408.07852</a><div class="button-and-tooltip">
              <button class="more-info" aria-describedby="more-info-desc-1">
                <svg height="15" role="presentation" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512"><path fill="currentColor" d="M256 8C119.043 8 8 119.083 8 256c0 136.997 111.043 248 248 248s248-111.003 248-248C504 119.083 392.957 8 256 8zm0 110c23.196 0 42 18.804 42 42s-18.804 42-42 42-42-18.804-42-42 18.804-42 42-42zm56 254c0 6.627-5.373 12-12 12h-88c-6.627 0-12-5.373-12-12v-24c0-6.627 5.373-12 12-12h12v-64h-12c-6.627 0-12-5.373-12-12v-24c0-6.627 5.373-12 12-12h64c6.627 0 12 5.373 12 12v100h12c6.627 0 12 5.373 12 12v24z" class=""></path></svg>
                <span class="visually-hidden">Focus to learn more</span>
              </button>
              <!-- tooltip description -->
              <div role="tooltip" id="more-info-desc-1">
                <span class="left-corner"></span>                  arXiv-issued DOI via DataCite</div>
            </div>
          </td>
        </tr></table>
    </div>
  </div>
</div>
    <div class="submission-history">
      <h2>Submission history</h2> From: Rosanne Liu [<a href="/show-email/ae0f604b/2408.07852">view email</a>]      <br/>    <strong>[v1]</strong>
        Wed, 14 Aug 2024 23:34:28 UTC (1,374 KB)<br/>
</div>
  </div>
  <!--end leftcolumn-->
<div class="extra-services">    <div class="full-text">
      <a name="other"></a>
      <span class="descriptor">Full-text links:</span>
      <h2>Access Paper:</h2>
      <ul>
  <div id="download-button-info" hidden>
View a PDF of the paper titled Training Language Models on the Knowledge Graph: Insights on Hallucinations and Their Detectability, by Jiri Hron and 30 other authors</div><li><a href="/pdf/2408.07852" aria-describedby="download-button-info" accesskey="f" class="abs-button download-pdf">View PDF</a></li><li><a href="https://arxiv.org/html/2408.07852v1" class="abs-button" id="latexml-download-link">HTML (experimental)</a></li><li><a href="/src/2408.07852" class="abs-button download-eprint">TeX Source</a></li><li><a href="/format/2408.07852" class="abs-button download-format">Other Formats</a></li></ul>
      <div class="abs-license"><a href="http://creativecommons.org/licenses/by/4.0/" title="Rights to this article" class="has_license">
          <img alt="license icon" role="presentation" src="https://arxiv.org/icons/licenses/by-4.0.png"/>
          <span>view license</span>
        </a></div>
    </div>
    <!--end full-text-->    <div class="browse">
    Current browse context: <div class="current">cs.CL</div>

  <div class="prevnext">
<span class="arrow">
      <a class="abs-button prev-url" href="/prevnext?id=2408.07852&amp;function=prev&amp;context=cs.CL"
         accesskey="p" title="previous in cs.CL (accesskey p)">&lt;&nbsp;prev</a>
    </span>
    <span class="is-hidden-mobile">&nbsp; | &nbsp;</span>    <span class="arrow">
      <a class="abs-button next-url" href="/prevnext?id=2408.07852&amp;function=next&amp;context=cs.CL" accesskey="n"
         title="next in cs.CL (accesskey n)">next&nbsp;&gt;</a>
    </span><br/>
  </div><div class="list">
    <a class="abs-button abs-button-grey abs-button-small context-new" href="/list/cs.CL/new">new</a>
    <span class="is-hidden-mobile"> | </span>
    <a class="abs-button abs-button-grey abs-button-small context-recent" href="/list/cs.CL/recent">recent</a>
    <span class="is-hidden-mobile"> | </span><a class="abs-button abs-button-grey abs-button-small context-id" href="/list/cs.CL/2024-08">2024-08</a>
  </div><div class="abs-switch-cat">
    Change to browse by:
    <div class="switch context-change">
        <a href="/abs/2408.07852?context=cs">cs</a><br class="is-hidden-mobile">
        <a class="subclass" href="/abs/2408.07852?context=cs.AI">cs.AI</a><br class="is-hidden-mobile">
        <a class="subclass" href="/abs/2408.07852?context=cs.LG">cs.LG</a><br class="is-hidden-mobile">
    </div>
  </div>

    </div>

    <div class="extra-ref-cite">
      <h3>References &amp; Citations</h3>
      <ul>
        <li><a  class="abs-button abs-button-small cite-ads" href="https://ui.adsabs.harvard.edu/abs/arXiv:2408.07852">NASA ADS</a></li><li><a  class="abs-button abs-button-small cite-google-scholar" href="https://scholar.google.com/scholar_lookup?arxiv_id=2408.07852" target="_blank" rel="noopener">Google Scholar</a></li>
        <li><a  class="abs-button abs-button-small cite-semantic-scholar" href="https://api.semanticscholar.org/arXiv:2408.07852" target="_blank" rel="noopener">Semantic Scholar</a></li>
      </ul>
      <div style="clear:both;"></div>
    </div>

<div class='extra-ref-cite'>
    <a id='bib-cite-css' hidden='true' href='/static/browse/0.3.4/css/cite.css'>a</a>

    <span id='bib-cite-trigger' class="bib-cite-button abs-button">export BibTeX citation</span>
    <span id='bib-cite-loading' hidden='true'>Loading...</span>
</div>

<div id='bib-cite-modal' class='bib-modal' hidden='true'>
    <div class='bib-modal-content'>
        <div class='bib-modal-title'>
            <h2>BibTeX formatted citation</h2>
            <span class='bib-modal-close' >&times;</span>
        </div>
        <div>
            <textarea id='bib-cite-target' class="bib-citation-content" aria-label="loading the citation">loading...</textarea>
        </div>
        <div>
            <span>Data provided by: </span>
            <a id='bib-cite-source-api'></a>
        </div>
    </div>
</div><div class="bookmarks">
  <div><h3>Bookmark</h3></div><a class="abs-button abs-button-grey abs-button-small" href="http://www.bibsonomy.org/BibtexHandler?requTask=upload&amp;url=https://arxiv.org/abs/2408.07852&amp;description=Training Language Models on the Knowledge Graph: Insights on Hallucinations and Their Detectability"
     title="Bookmark on BibSonomy">
    <img src="/static/browse/0.3.4/images/icons/social/bibsonomy.png"
         alt="BibSonomy logo"/>
  </a>
  <a class="abs-button abs-button-grey abs-button-small" href="https://reddit.com/submit?url=https://arxiv.org/abs/2408.07852&amp;title=Training Language Models on the Knowledge Graph: Insights on Hallucinations and Their Detectability"
     title="Bookmark on Reddit">
    <img src="/static/browse/0.3.4/images/icons/social/reddit.png"
         alt="Reddit logo"/>
  </a>
</div>  </div>
  <!--end extra-services-->
<!-- LABS AREA -->
<div id="labstabs">
  <div class="labstabs"><input type="radio" name="tabs" id="tabone"checked="checked">
    <label for="tabone">Bibliographic Tools</label>
    <div class="tab labs-display-bib">
      <h1>Bibliographic and Citation Tools</h1>
      <div class="toggle">
        <div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input id="bibex-toggle" type="checkbox" class="lab-toggle">
              <span class="slider"></span>
              <span class="is-sr-only">Bibliographic Explorer Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-bibex">Bibliographic Explorer</span> <em>(<a href="https://info.arxiv.org/labs/showcase.html#arxiv-bibliographic-explorer">What is the Explorer?</a>)</em>
          </div>
        </div>
<div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input
                id="litmaps-toggle"
                type="checkbox"
                class="lab-toggle"
                data-script-url="/static/browse/0.3.4/js/litmaps.js?20210617"
                aria-labelledby="label-for-litmaps">
              <span class="slider"></span>
              <span class="is-sr-only">Litmaps Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-litmaps">Litmaps</span> <em>(<a href="https://www.litmaps.co/" target="_blank">What is Litmaps?</a>)</em>
          </div>
        </div>
        <div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input
                id="scite-toggle"
                type="checkbox"
                class="lab-toggle"
                data-script-url="/static/browse/0.3.4/js/scite.js?20210617"
                aria-labelledby="label-for-scite">
              <span class="slider"></span>
              <span class="is-sr-only">scite.ai Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-scite">scite Smart Citations</span> <em>(<a href="https://www.scite.ai/" target="_blank">What are Smart Citations?</a>)</em>
          </div>
        </div>
      </div>
        <div class="labs-content-placeholder labs-display" style="display: none;"></div>
        <div style="min-height: 15px" id="litmaps-open-in"></div>
        <div style="min-height: 15px" id="scite-open-in"></div>
    </div>


    <input type="radio" name="tabs" id="tabtwo">
    <label for="tabtwo">Code, Data, Media</label>
    <div class="tab">
      <h1>Code, Data and Media Associated with this Article</h1>
      <div class="toggle">
        <div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input        
                id="catalyzex-toggle"
                data-script-url="/static/browse/0.3.4/js/catalyzex.js"
                type="checkbox" class="lab-toggle" aria-labelledby="label-for-cx">
              <span class="slider"></span>
              <span class="is-sr-only">Links to Code Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-cx">CatalyzeX Code Finder for Papers</span> <em>(<a href="https://www.catalyzex.com" target="_blank">What is CatalyzeX?</a>)</em>
          </div>
        </div>

        <div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input
                id="dagshub-toggle"
                data-script-url="/static/browse/0.3.4/js/dagshub.js"
                type="checkbox" class="lab-toggle" aria-labelledby="label-for-dagshub">
              <span class="slider"></span>
              <span class="is-sr-only">DagsHub Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-dagshub">DagsHub</span> <em>(<a href="https://dagshub.com/" target="_blank">What is DagsHub?</a>)</em>
          </div>
        </div>
  
        <div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input
                id="gotitpub-toggle"
                data-script-url="/static/browse/0.3.4/js/gotitpub.js"
                type="checkbox" class="lab-toggle" aria-labelledby="label-for-gotitpub">
              <span class="slider"></span>
              <span class="is-sr-only">GotitPub Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-gotitpub">Gotit.pub</span> <em>(<a href="http://gotit.pub/faq" target="_blank">What is GotitPub?</a>)</em>
          </div>
        </div>

        <div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input
                id="paperwithcode-toggle"
                data-script-url="/static/browse/0.3.4/js/paperswithcode.js"
                type="checkbox" class="lab-toggle" aria-labelledby="label-for-pwc">
              <span class="slider"></span>
              <span class="is-sr-only">Links to Code Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-pwc">Papers with Code</span> <em>(<a href="https://paperswithcode.com/" target="_blank">What is Papers with Code?</a>)</em>
          </div>
        </div>


        <div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input
                id="sciencecast-toggle"
                data-script-url="/static/browse/0.3.4/js/sciencecast.js"
                type="checkbox" class="lab-toggle" aria-labelledby="label-for-sciencecast">
              <span class="slider"></span>
              <span class="is-sr-only">ScienceCast Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-sciencecast">ScienceCast</span> <em>(<a href="https://sciencecast.org/welcome" target="_blank">What is ScienceCast?</a>)</em>
          </div>
        </div>
      </div>

      <div id="catalyzex-output" style="display:none"></div>
      <div id="dagshub-output" style="display:none"></div>
      <div id="gotitpub-output" style="display:none"></div>
      <div id="pwc-output" style="display:none"></div>
      <div id="pwc-data-output" style="display:none"></div>
      <div id="sciencecast-output" style="display:none"></div>
    </div>


      <input type="radio" name="tabs" id="labstabs-demos-input">
      <label for="labstabs-demos-input" id="labstabs-demos-label">Demos</label>
      <div class="tab">
        <h1>Demos</h1>
        <div class="toggle">
          <div class="columns is-mobile lab-row">
            <div class="column lab-switch">
              <label class="switch">
                <input
                  id="replicate-toggle"
                  data-script-url="/static/browse/0.3.4/js/replicate.js"
                  type="checkbox" class="lab-toggle" aria-labelledby="label-for-replicate">
                <span class="slider"></span>
                <span class="is-sr-only">Replicate Toggle</span>
              </label>
            </div>
            <div class="column lab-name">
              <span id="label-for-replicate">Replicate</span> <em>(<a href="https://replicate.com/docs/arxiv/about" target="_blank">What is Replicate?</a>)</em>
            </div>
          </div>
          <div class="columns is-mobile lab-row">
            <div class="column lab-switch">
              <label class="switch">
                <input
                  id="spaces-toggle"
                  data-script-url="/static/browse/0.3.4/js/spaces.js"
                  type="checkbox" class="lab-toggle" aria-labelledby="label-for-spaces">
                <span class="slider"></span>
                <span class="is-sr-only">Spaces Toggle</span>
              </label>
            </div>
            <div class="column lab-name">
              <span id="label-for-spaces">Hugging Face Spaces</span> <em>(<a href="https://huggingface.co/docs/hub/spaces" target="_blank">What is Spaces?</a>)</em>
            </div>
          </div>
          <div class="columns is-mobile lab-row">
            <div class="column lab-switch">
              <label class="switch">
                <input
                  id="txyz-toggle"
                  data-script-url="/static/browse/0.3.4/js/txyz.js"
                  type="checkbox" class="lab-toggle" aria-labelledby="label-for-txyz">
                <span class="slider"></span>
                <span class="is-sr-only">Spaces Toggle</span>
              </label>
            </div>
            <div class="column lab-name">
              <span id="label-for-txyz">TXYZ.AI</span> <em>(<a href="https://txyz.ai" target="_blank">What is TXYZ.AI?</a>)</em>
            </div>
          </div>
        </div>
        <div id="replicate-output"></div>
        <div id="spaces-output"></div>
        <div id="txyz-output"></div>
      </div>
      <input type="radio" name="tabs" id="tabfour">
      <label for="tabfour">Related Papers</label>
      <div class="tab">
        <h1>Recommenders and Search Tools</h1>
        <div class="toggle">
          <div class="columns is-mobile lab-row">
            <div class="column lab-switch">
              <label class="switch">
                <input id="influenceflower-toggle"
                data-script-url="/static/browse/0.3.4/js/influenceflower.js"
                type="checkbox" class="lab-toggle" aria-labelledby="label-for-influenceflower">
                <span class="slider"></span>
                <span class="is-sr-only">Link to Influence Flower</span>
              </label>
            </div>
            <div class="column lab-name">
              <span id="label-for-influenceflower">Influence Flower</span> <em>(<a href="https://influencemap.cmlab.dev/" target="_blank">What are Influence Flowers?</a>)</em>
            </div>
          </div>
          <div class="columns is-mobile lab-row">
            <div class="column lab-switch">
              <label class="switch">
                <input id="connectedpapers-toggle" type="checkbox" class="lab-toggle"
                       data-script-url="/static/browse/0.3.4/js/connectedpapers.js"
                       aria-labelledby="label-for-connected-papers" >
                <span class="slider"></span>
                <span class="is-sr-only">Connected Papers Toggle</span>
              </label>
            </div>
            <div class="column lab-name">
              <span id="label-for-connected-papers">Connected Papers</span> <em>(<a href="https://www.connectedpapers.com/about" target="_blank">What is Connected Papers?</a>)</em>
            </div>
          </div>
          <div class="columns is-mobile lab-row">
            <div class="column lab-switch">
              <label class="switch">
                <input id="core-recommender-toggle" type="checkbox" class="lab-toggle" aria-labelledby="label-for-core">
                <span class="slider"></span>
                <span class="is-sr-only">Core recommender toggle</span>
              </label>
            </div>
            <div class="column lab-name">
              <span id="label-for-core">CORE Recommender</span> <em>(<a href="https://core.ac.uk/services/recommender">What is CORE?</a>)</em>
            </div>
          </div></div>
        <div id="influenceflower-output"></div>
        <div id="influenceflower-output-graph" style="display:none">
          <ul class="flower-tabs">
            <li class="active"><a class="btn tab-btn" onclick="openTab(event, 'tab-author')">Author</a></li>
            <li><a class="btn tab-btn" onclick="openTab(event, 'tab-venue')">Venue</a></li>
            <li><a class="btn tab-btn" onclick="openTab(event, 'tab-inst')">Institution</a></li>
            <li><a class="btn tab-btn" onclick="openTab(event, 'tab-topic')">Topic</a></li>
          </ul>
          <div class="flower-tab-content">
            <div class="tab-flower active" id="tab-author"><svg id="flower-graph-author"></svg></div>
            <div class="tab-flower" id="tab-venue"><svg id="flower-graph-venue"></svg></div>
            <div class="tab-flower" id="tab-inst"><svg id="flower-graph-inst"></svg></div>
            <div class="tab-flower" id="tab-topic"><svg id="flower-graph-topic"></svg></div>
          </div>
        </div>
        <div style="min-height: 15px" id="connectedpapers-output"></div>
        <div id="coreRecommenderOutput"></div>
        <div id="iarxivOutput"></div>
      </div>

      <input type="radio" name="tabs" id="tabfive">
      <label for="tabfive">
        About arXivLabs
      </label>
      <div class="tab">
        <div class="columns">
          <div class="column">
            <h1>arXivLabs: experimental projects with community collaborators</h1>
            <p>arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.</p>
            <p>Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.</p>
            <p>Have an idea for a project that will add value for arXiv's community? <a href="https://info.arxiv.org/labs/index.html"><strong>Learn more about arXivLabs</strong></a>.</p>
          </div>
          <div class="column is-narrow is-full-mobile">
            <p class="icon-labs"><svg xmlns="http://www.w3.org/2000/svg" role="presentation" viewBox="0 0 635.572 811"><path d="M175.6 676v27h-27v-27zm-54 27v27h27v-27zm-27 27v27h27v-27zm396-54v27h-27v-27zm0 27v27h27v-27zm27 27v27h27v-27zm-27-414h27v27h-27zm27 0h27v-27h-27zm27-27h27v-27h-27zm-396 45h-27v-27h27zm-27-54h-27v27h27zm-27-27h-27v27h27z"/><path d="M94.6 730v27h-27v-27zm477 0v27h-27v-27zm-27-495h27v27h-27zm-450 18h-27v-27h27zm477 9h27v27h-27zm-54 495h27v27h-27zm-423 0h27v27h-27zm-54-504h27v27h-27z" fill="#666"/><path d="M67.6 730v27h-27v-27zm54 54v27h-27v-27zm0-108v27h27v-27zm-27 27v27h27v-27zm-81 0v27h27v-27zm585 27v27h-27v-27zm-108-54v27h27v-27zm27 27v27h27v-27zm81 0v27h27v-27zm-54-495h27v27h-27zm-54 108h27v-27h-27zm27-27h27v-27h-27zm0-81h27v-27h-27zm-423 18h-27v-27h27zm54 54h-27v27h27zm-27-27h-27v27h27zm0-81h-27v27h27zm423 612v27h-27v-27zm81-522v27h-27v-27zm-585-9v27h-27v-27z" fill="#999"/><path d="M94.6 784v27h-27v-27zm-27-27v27h27v-27zm-27-54v27h27v-27zm27 0v27h27v-27zm0-27v27h27v-27zm27 0v27h27v-27zm0-27v27h27v-27zm27 0v27h27v-27zm-108 81v27h27v-27zm558 54v27h-27v-27zm-27-27v27h27v-27zm27-54v27h27v-27zm-27 0v27h27v-27zm0-27v27h27v-27zm-27 0v27h27v-27zm0-27v27h27v-27zm-27 0v27h27v-27zm108 81v27h27v-27zm0-495h27v27h-27zm-27 27h27v-27h-27zm-54-27h27v-27h-27zm0 27h27v-27h-27zm-27 0h27v-27h-27zm0 27h27v-27h-27zm-27 0h27v-27h-27zm0 27h27v-27h-27zm81-108h27v-27h-27zm-504 45h-27v-27h27zm27-27h-27v27h27zm54-27h-27v27h27zm0 27h-27v27h27zm27 0h-27v27h27zm0 27h-27v27h27zm27 0h-27v27h27zm0 27h-27v27h27zm-81-108h-27v27h27z" fill="#ccc"/><path d="M598.6 665.1H41.5C-76.5 667 176 280.2 176 280.2h53a46.5 46.5 0 0162.8-56.3 29.2 29.2 0 1128.5 35.9h-1a46.5 46.5 0 01-1.5 20.3l142.5-.1s255.3 387 138.3 385.1zM291 181a29.3 29.3 0 10-29.2-29.3A29.3 29.3 0 00291 181zm65.4-66.8a22.4 22.4 0 10-22.5-22.4 22.4 22.4 0 0022.5 22.4z" fill="#fc0"/><path d="M245.5 172V10h153v162s324 495 198 495h-558c-126 0 207-495 207-495zm126 54h56m-13 72h56m-9 72h56m-20 72h56m-22 72h56m-29 72h56m-457-45c20.8 41.7 87.3 81 160.7 81 72.1 0 142.1-38.2 163.4-81" fill="none" stroke="#000" stroke-miterlimit="10" stroke-width="20"/><path d="M273.3 421.7c0 31-9.8 56.3-21.9 56.3s-21.8-25.2-21.8-56.3 9.8-56.3 21.8-56.3 21.9 25.2 21.9 56.3zm114.4-56.3c-12 0-21.8 25.2-21.8 56.3s9.7 56.3 21.8 56.3 21.9-25.2 21.9-56.3-9.8-56.3-21.9-56.3zM150.1 526.6c-18.2 6.7-27.5 22.9-23.2 30.2s14.8-5.5 33-12.2 37.4-4.9 33-12.2-24.5-12.6-42.8-5.8zm296 5.8c-4.2 7.3 14.9 5.5 33.1 12.2s28.7 19.5 33 12.2-5-23.5-23.2-30.2-38.5-1.5-42.8 5.8z"/></svg></p>
          </div>
        </div>
      </div>

    </div>
</div>
<!-- END LABS AREA -->
  <div class="endorsers">
    <a href="/auth/show-endorsers/2408.07852" class="endorser-who">Which authors of this paper are endorsers?</a> |
    <a id="mathjax_toggle" href="javascript:setMathjaxCookie()">Disable MathJax</a> (<a href="https://info.arxiv.org/help/mathjax.html">What is MathJax?</a>)
    <span class="help" style="font-style: normal; float: right; margin-top: 0; margin-right: 1em;"></span>
  </div>
  <script type="text/javascript" language="javascript">mathjaxToggle();</script>
</div>
      </div>
    </main>

    <footer style="clear: both;">
      <div class="columns is-desktop" role="navigation" aria-label="Secondary" style="margin: -0.75em -0.75em 0.75em -0.75em">
        <!-- Macro-Column 1 -->
        <div class="column" style="padding: 0;">
          <div class="columns">
            <div class="column">
              <ul style="list-style: none; line-height: 2;">
                <li><a href="https://info.arxiv.org/about">About</a></li>
                <li><a href="https://info.arxiv.org/help">Help</a></li>
              </ul>
            </div>
            <div class="column">
              <ul style="list-style: none; line-height: 2;">
                <li>
                  <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512" class="icon filter-black" role="presentation"><title>contact arXiv</title><desc>Click here to contact arXiv</desc><path d="M502.3 190.8c3.9-3.1 9.7-.2 9.7 4.7V400c0 26.5-21.5 48-48 48H48c-26.5 0-48-21.5-48-48V195.6c0-5 5.7-7.8 9.7-4.7 22.4 17.4 52.1 39.5 154.1 113.6 21.1 15.4 56.7 47.8 92.2 47.6 35.7.3 72-32.8 92.3-47.6 102-74.1 131.6-96.3 154-113.7zM256 320c23.2.4 56.6-29.2 73.4-41.4 132.7-96.3 142.8-104.7 173.4-128.7 5.8-4.5 9.2-11.5 9.2-18.9v-19c0-26.5-21.5-48-48-48H48C21.5 64 0 85.5 0 112v19c0 7.4 3.4 14.3 9.2 18.9 30.6 23.9 40.7 32.4 173.4 128.7 16.8 12.2 50.2 41.8 73.4 41.4z"/></svg>
                  <a href="https://info.arxiv.org/help/contact.html"> Contact</a>
                </li>
                <li>
                  <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512" class="icon filter-black" role="presentation"><title>subscribe to arXiv mailings</title><desc>Click here to subscribe</desc><path d="M476 3.2L12.5 270.6c-18.1 10.4-15.8 35.6 2.2 43.2L121 358.4l287.3-253.2c5.5-4.9 13.3 2.6 8.6 8.3L176 407v80.5c0 23.6 28.5 32.9 42.5 15.8L282 426l124.6 52.2c14.2 6 30.4-2.9 33-18.2l72-432C515 7.8 493.3-6.8 476 3.2z"/></svg>
                  <a href="https://info.arxiv.org/help/subscribe"> Subscribe</a>
                </li>
              </ul>
            </div>
          </div>
        </div>
        <!-- End Macro-Column 1 -->
        <!-- Macro-Column 2 -->
        <div class="column" style="padding: 0;">
          <div class="columns">
            <div class="column">
              <ul style="list-style: none; line-height: 2;">
                <li><a href="https://info.arxiv.org/help/license/index.html">Copyright</a></li>
                <li><a href="https://info.arxiv.org/help/policies/privacy_policy.html">Privacy Policy</a></li>
              </ul>
            </div>
            <div class="column sorry-app-links">
              <ul style="list-style: none; line-height: 2;">
                <li><a href="https://info.arxiv.org/help/web_accessibility.html">Web Accessibility Assistance</a></li>
                <li>
                  <p class="help">
                    <a class="a11y-main-link" href="https://status.arxiv.org" target="_blank">arXiv Operational Status <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 256 512" class="icon filter-dark_grey" role="presentation"><path d="M224.3 273l-136 136c-9.4 9.4-24.6 9.4-33.9 0l-22.6-22.6c-9.4-9.4-9.4-24.6 0-33.9l96.4-96.4-96.4-96.4c-9.4-9.4-9.4-24.6 0-33.9L54.3 103c9.4-9.4 24.6-9.4 33.9 0l136 136c9.5 9.4 9.5 24.6.1 34z"/></svg></a><br>
                    Get status notifications via
                    <a class="is-link" href="https://subscribe.sorryapp.com/24846f03/email/new" target="_blank"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512" class="icon filter-black" role="presentation"><path d="M502.3 190.8c3.9-3.1 9.7-.2 9.7 4.7V400c0 26.5-21.5 48-48 48H48c-26.5 0-48-21.5-48-48V195.6c0-5 5.7-7.8 9.7-4.7 22.4 17.4 52.1 39.5 154.1 113.6 21.1 15.4 56.7 47.8 92.2 47.6 35.7.3 72-32.8 92.3-47.6 102-74.1 131.6-96.3 154-113.7zM256 320c23.2.4 56.6-29.2 73.4-41.4 132.7-96.3 142.8-104.7 173.4-128.7 5.8-4.5 9.2-11.5 9.2-18.9v-19c0-26.5-21.5-48-48-48H48C21.5 64 0 85.5 0 112v19c0 7.4 3.4 14.3 9.2 18.9 30.6 23.9 40.7 32.4 173.4 128.7 16.8 12.2 50.2 41.8 73.4 41.4z"/></svg>email</a>
                    or <a class="is-link" href="https://subscribe.sorryapp.com/24846f03/slack/new" target="_blank"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512" class="icon filter-black" role="presentation"><path d="M94.12 315.1c0 25.9-21.16 47.06-47.06 47.06S0 341 0 315.1c0-25.9 21.16-47.06 47.06-47.06h47.06v47.06zm23.72 0c0-25.9 21.16-47.06 47.06-47.06s47.06 21.16 47.06 47.06v117.84c0 25.9-21.16 47.06-47.06 47.06s-47.06-21.16-47.06-47.06V315.1zm47.06-188.98c-25.9 0-47.06-21.16-47.06-47.06S139 32 164.9 32s47.06 21.16 47.06 47.06v47.06H164.9zm0 23.72c25.9 0 47.06 21.16 47.06 47.06s-21.16 47.06-47.06 47.06H47.06C21.16 243.96 0 222.8 0 196.9s21.16-47.06 47.06-47.06H164.9zm188.98 47.06c0-25.9 21.16-47.06 47.06-47.06 25.9 0 47.06 21.16 47.06 47.06s-21.16 47.06-47.06 47.06h-47.06V196.9zm-23.72 0c0 25.9-21.16 47.06-47.06 47.06-25.9 0-47.06-21.16-47.06-47.06V79.06c0-25.9 21.16-47.06 47.06-47.06 25.9 0 47.06 21.16 47.06 47.06V196.9zM283.1 385.88c25.9 0 47.06 21.16 47.06 47.06 0 25.9-21.16 47.06-47.06 47.06-25.9 0-47.06-21.16-47.06-47.06v-47.06h47.06zm0-23.72c-25.9 0-47.06-21.16-47.06-47.06 0-25.9 21.16-47.06 47.06-47.06h117.84c25.9 0 47.06 21.16 47.06 47.06 0 25.9-21.16 47.06-47.06 47.06H283.1z"/></svg>slack</a>
                  </p>
                </li>
              </ul>
            </div>
          </div>
        </div> <!-- end MetaColumn 2 -->
        <!-- End Macro-Column 2 -->
      </div>
    </footer>
  </div>

  <script src="/static/base/1.0.1/js/member_acknowledgement.js"></script>

</body>

</html>
2408.07852
 While many capabilities of *language models* (LMs) improve with increased training budget, the influence of scale on hallucinations is not yet fully understood. Hallucinations come in many forms, and there is no universally accepted definition. We thus focus on studying only those hallucinations where a correct answer appears verbatim in the training set. To fully control the training data content, we construct a *knowledge graph* (KG)-based dataset, and use it to train a set of increasingly large LMs. We find that for a fixed dataset, larger and longer-trained LMs hallucinate less. However, hallucinating on ≤ 5% of the training data requires an order of magnitude larger model, and thus an order of magnitude more compute, than Hoffmann et al. (2022) reported was optimal. Given this costliness, we study how hallucination detectors depend on scale. While we see detector size improves performance on fixed LM's outputs, we find an inverse relationship between the scale of the LM and the detectability of its hallucinations. 
abstract
Training Language Models On The Knowledge Graph: Insights On Hallucinations And Their Detectability
Abstract
2408.07852
 Despite rapid progress in generative and predictive capabilities, hallucinations remain a significant challenge for large language models (Gemini, 2023; OpenAI, 2023). Although researchers have carefully studied "scaling laws" (Kaplan et al., 2020; Hoffmann et al., 2022)—an empirical phenomenon where LM performance improves as dataset and model size increase—little is known about how hallucinations depend on scale. To fill this gap, the first issue at hand is to precisely define and quantify hallucinations. However, in natural language setting this is very hard as language expressions can be ambitious, and the exact knowledge content in training data is notoriously unclear. Knowledge graph (KG), on the other hand, offers full controllablility of its factual content: it is straightforward to query whether a generated fact from an LM indeed exists in the dataset or not, hence offering quantifiable measure of hallucination. Training LMs on KG allows us to study the extent to which LMs misrepresent their training data, and how this phenomenon depends on scale.  ![1_image_0.png](1_image_0.png)  Figure 1: **Data and the training pipeline.** <S TKN>, <P TKN> and <O TKN> are special tokens indicating subject, predicate, and object, respectively. (a) The original data exist in the form of a Knowledge Graph (KG), where nodes representing subjects and objects are connected by predicates (arrows). (b) The KG is then formatted into triplets: subject, predicate, object, and further prefixed with special tokens indicating their identity. Such formatted data are used to pretrain autoregressive LMs with the common next-token-prediction loss. (c) Pretrained LMs are evaluated by prefixing with subject and predicate alongside special tokens to predict objects. (d) On top of pretrained LMs, detectors are trained to detect the presence of hallucinations during generation.  We therefore train a number of increasingly large LMs from scratch on data extracted from a KG. A KG contains knowledge about the world in the form of [subject, predicate, object] triplets (Figure 1(a)). We concatenate the triplets into strings, and use these to train the LMs by optimising the auto-regressive log-likelihood (Figure 1(b)). When evaluating, we prompt the model with a subject and predicate, and let it complete the object (Figure 1(c)).  A generation is considered a hallucination if the prediction does not match any object that appears with given subject-predicate pair in the training set.  The speciality to this setup is that each unique piece of information is only seen once per epoch: each triplet contains a unique piece of information that is distinctive from any other triplet. This is unlike any traditional LMs trained over natural language, where the same information content may appear multiple times even in deduplicated datasets, due to the flexibility of natural language. This difference engenders three major observations that deviate from our current understanding of LM (pre)training on text.  First, multi-epoch training is necessary, as one might expect in this setting, for the training loss to converge and hallucination rate to go down to an acceptable level. Secondly, the scaling between the model size and the number of training tokens is different from what is currently considered optimal (Hoffmann et al., 2022); LMs several times larger are needed to eliminate training set hallucinations. What's more, counter to standard scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022), the non-duplication of triplets also means that increasing the training set size (adding more pieces of information) leads to worse performance for any fixed LM size and training length. To decrease the training set hallucination rate, one can train for longer and set the generation temperature to zero, but this often comes at the cost of worse generalisation to data not seen during training. Thus, there is a trade-off between training set hallucination rate and other LM capabilities.  Equipped with a range of LMs trained from KG with various hallucination rates, we then investigate methods that prevent hallucinations by detecting—and optionally also correcting—them post-hoc, as well as the scaling behaviour of such methods (Figure 1(d)).  While we do find that larger detectors perform better, and that detection accuracy improves with the LM size and length of pretraining, there is a caveat: the improvement in accuracy with LM scale is confounded by the already lower hallucination rate of the more powerful LMs. Examining the detectors' precision-recall (PR) curves, we find an *inverse* relationship between the LM size and the detector's area under the PR curve (AUC-PR). In other words,  | Dataset     | Language Model   | Detector    | Size (total triplets)   |    |             | |-------------|------------------|-------------|-------------------------|----|-------------| | Trained on? | Eval-ed on?      | Trained on? | Eval-ed on?             |    |             | | FVS, 1%     | ✓                | ✓           | ✓                       | ✗  | 6,505,590   | | FVS, 10%    | ✓                | ✓           | ✓                       | ✗  | 65,150,414  | | FVS, 100%   | ✓                | ✓           | ✗                       | ✗  | 555,969,021 | | PVS, 10%    | ✓                | ✓           | ✗                       | ✓  | 159,502     | | PVS, 100%   | ✓                | ✓           | ✗                       | ✓  | 1,606,058   | | IVS         | ✗                | ✓           | ✗                       | ✓  | 19,311      |  Table 1: **Datasets created with different levels of visibility and scale.** The *fully visible set* (FVS) is used for training of both the LMs and detectors (except for the 100% version). The partially visible set (PVS) is only used for the training of LMs, not the detectors. The invisible set (IVS) is a fully held-out test used only for some of the evaluations. For each visibility level, different dataset sizes are created by sampling 1% or 10% of the respective full set.  The total number of triplets contained in dataset version is shown. We always evaluate LMs on both training and test sets, while only evaluate detectors on held-out test sets.  Dataset Training epochs  1 2 10 20 100 200  FVS + PVS, 1% ✓ ✓ ✓ ✓ ✓ ✓  FVS + PVS, 10% ✓ ✓ ✓ ✓ ✗ ✗ FVS + PVS, 100% ✓ ✓ ✗ ✗ ✗ ✗  Table 2: **Length of training** for different dataset size. We train a maximum of 200 epochs, on the smallest subset, and as the training set grows, we reduce the training length.  the more powerful the LM, the harder it is to detect its hallucinations. This is despite the fact that we have control over what the model is exposed to, which has been thought to be key to making hallucination detectors work well (Schulman, 2023).  The rest of the paper is structured as follows. We describe the dataset and training of LMs in Section 2, investigate the interplay between hallucinations and LM scale in Section 3, and study the relation between LM scale and hallucination detectability in Section 4. The remaining sections are devoted to the discussion of limitations and takeaways. 
introduction
Training Language Models On The Knowledge Graph: Insights On Hallucinations And Their Detectability
Introduction
2408.07852
 A core challenge in studying LM hallucinations is that we typically do not know what information the model was exposed to during training. Without this knowledge, we cannot determine whether the model output is wrong because (a) it has never been trained on a given fact, (b) it was trained on it but did not memorize it, or (c) the model memorized factually incorrect information that did appear in the training set. To avoid these issues, which are further confounded by various pretraining and finetuning strategies used in state-of-the-art LMs, we train LMs from scratch (Section 2.2) on data specifically constructed to give us perfect control over the information a model sees (Section 2.1). This will later enable us to investigate how model and dataset scale affects hallucinations (Section 3), and their detectability (Section 4). 
body
Training Language Models On The Knowledge Graph: Insights On Hallucinations And Their Detectability
Controlling What An Lm Knows
2408.07852
 We propose using a *Knowledge Graph* (KG) as a way of controlling the information a model sees. KGs are structured, factual data, that are often used within organisations to feed knowledge into various applications. We use KG as it provides a repository of information which is self-consistent, and mirrors the structure of information in the real-world; the hope is that this mirrored structure will ensure that the character of any hallucinations we see is somewhat similar to hallucinations we would see from models trained on data more typical for LM training. The main benefit to using a KG, however, is that we know exactly what a model has seen, and since it is structured data, we can easily query the data to see if its predictions are correct.  The KG we use (Google, 2012) contains semantic triples: a Subject, Predicate, and Object.  We further insert special tokens before each of the Subject, Predicate, and Object, as indicated in Figure 1, and use the concatenated strings to train our LMs. The processing removes the ambiguity of natural language, which makes the task both easier and harder for an LM. The task is easier because the samples are now *structured*: LMs no longer need to pick up the intricacies of grammar and distinguish different phrasings, and can instead just focus on learning facts. It is, however, also harder, because there is very little correlation between data samples, unless they share items, and thus very little positive transfer between learning one fact to the other.  In later sections (Sections 3 and 4), we will be training and evaluating LMs and hallucination detectors. We therefore need to carefully design data splits to fully understand the impact of data. We design datasets that reflect three levels of visibility: 1) a *fully visible set* (FVS) that both the LMs and detectors are trained on, 2) a *partially visible set* (PVS) that only the LMs are trained on, and finally 3) an *invisible set* (IVS) that neither the LM or the detector have seen. We then vary the sizes of FVS and PVS to study the effect of scale.  To construct these three sets of triplets, we perform an i.i.d. split at the subject level.  Some subject-predicate pairs are associated with multiple objects (e.g., names of tracks on an album). In these cases, we need to ensure that all objects associated with a given subject-predicate belong to the same set, as otherwise we might label correctly deduced object predictions as hallucinations. A similar issue can exist at the subject level, e.g., age can be deduced from date of birth. Several subject-predicate pairs are associated with hundreds of objects. To simplify evaluations, we remove all subject-predicates linked with more than 20 objects. This eliminates extreme long-tails that would be hard to meaningfully analyse. Table 1 reports the resulting dataset size. 
body
Training Language Models On The Knowledge Graph: Insights On Hallucinations And Their Detectability
The Knowledge Graph Dataset
2408.07852
 We trained decoder-only transformer LMs (Vaswani et al., 2017) with varying number of non-embedding parameters (3.15M–1.61B), on various sizes of the KG dataset (1%–100%).  The parameters are optimized with respect to the autoregressive cross-entropy loss over formatted strings created from triplets with special tokens (Section 2.1).  Where a single triplet does not fill the context window (256 tokens), we used packing (on average, ∼20 triplets fit into the context window). For optimization, we used Adam (Kingma & Ba, 2014) with linear learning rate warmup from 0 to our base learning rate (4K steps), followed by cosine decay down to 5% of the base rate. We varied the total number of steps to study the effect of multi-epoch training (see Tables 2 and 4 for details). The base learning rate is set to a constant divided by the square root of the number of LM's non-embedding parameters. The constant was determined by a hyperparameter search over 2.5, 5, and 10 (due to compute limitations this exploration was not done for all models).  The exact learning rates we used can be found in Table 3 in Appendix. 
body
Training Language Models On The Knowledge Graph: Insights On Hallucinations And Their Detectability
Training Lms On The Knowledge Graph
2408.07852
 Scaling laws are an empirical phenomenon where the cross-entropy loss of LMs decay as a power law of the model and training set size (Kaplan et al., 2020; Hoffmann et al., 2022).  Since cross-entropy is related to the accuracy of model predictions, one can wonder whether hallucination rates follow a similar trend. Figure 2 shows this is not the case. While for a fixed dataset size, larger, longer-trained models tend to hallucinate less, increasing the dataset size yields a higher, rather than a lower, hallucination rate (top left vs. top right); a similar trend is observed for the cross-entropy loss—evaluated on full triplets, not just the object—in Figure 3. This is because of two factors: (1) many triplets in the KG require  ![4_image_0.png](4_image_0.png)  ![4_image_1.png](4_image_1.png)  ![5_image_0.png](5_image_0.png)  ![5_image_1.png](5_image_1.png)  memorization (i.e., correct answer cannot be inferred from other data points); (2) each triplet appears in the training set only once. The bottom two plots in Figure 2 demonstrate the necessity of memorization; no model attains less than 50% hallucination rate on data not seen at training time. Examples of facts requiring memorization are names of tracks on an album, or dates of birth.  Another consequence of the necessity to memorize and of the lack of repetition is that 20+ epochs are needed to achieve close to minimal hallucination rate for a given LM size. This stands in sharp contrast to the current practice of training LMs for only one or a few epochs (Chowdhery et al., 2023; Hoffmann et al., 2022; Touvron et al., 2023); we show for models trained for fewer (1 or 2) epochs in the beginning of each line in Figure 2.  An unfortunate side-effect of training for 20+ epochs seems to be a decreasing ability to generalise to unseen data (note the eventually upwards slope in the bottom plots). This trend is even more pronounced at temp = 0.0 (Figure 8), presenting a trade-off between hallucination rates and other model performance measures. While even the most capable modern LMs hallucinate (Gemini, 2023; OpenAI, 2023), repetition of facts in their training data likely alleviates the issue, as also suggested in Kandpal et al. (2023). Excessive repetition might however be harmful (Hernandez et al., 2022; Xue et al., 2023; Muennighoff et al., 2023), presenting a challenge for curation of training data.  It is striking that, in our case, even when training on 1% of the full KG, the hallucination rate on *data seen during training* remains ∼5% for the largest and longest-trained LM. This hallucination rate can be pushed down to ∼1.5% by lowering the temperature from 1.0 to 0.0. However, the KG contains a large number of subject-predicate pairs with multiple associated objects (e.g., names of all authors of a particular paper), and reducing sampling temperature leads the model to generate fewer of these objects. In Figure 4, we show how varying temperature affects both *precision*, defined as 1-hallucination rate, and *recall*, defined as the average proportion of objects in the original training data that the model generates at least once when sampling 16 completions for each subject-predicate pair. This highlights an issue with focusing on hallucination rate only: an easy way to hallucinate less is to make fewer claims. A related issue is that finetuning LMs to refuse to answer when uncertain may potentially lead to overly conservative behaviour.  Our best-performing LMs are an order of magnitude larger than the predicted Chinchillaoptimal size for the number of tokens on which we train (Hoffmann et al., 2022), and therefore even larger than the even smaller models geared towards efficient inference (Touvron et al., 2023). This suggests that even if we just wanted LMs to rarely hallucinate on the data that they have seen during pretraining, we would need a computational budget several times higher than what is currently considered optimal. A more efficient alternative may be retrieval-augmentation (Lewis et al., 2020; Borgeaud et al., 2022), or methods for expressing uncertainty or self-correcting (e.g., Dhuliawala et al., 2023; OpenAI, 2023; Li et al., 2023). In Section 4, we study influence of scale on methods that belong to the latter category. 
body
Training Language Models On The Knowledge Graph: Insights On Hallucinations And Their Detectability
Hallucination Rate And How It Scales
2408.07852
 In Section 3, we have seen hallucination rates typically decay with LM size and training length. However, even LMs much larger than currently considered optimal—for given training set size—continue to hallucinate ∼5% of the time on data seen and ∼50% on data unseen during training (Figure 2). Our experiments also exhibit a trade-off between indistribution and out-of-distribution hallucination rates (Figures 2 and 8). We therefore need to understand whether it is possible to further reduce hallucination rates by other means.  There are many types of alternative interventions, ranging from retrieval to model self-correction (see Appendix A). One promising direction is *hallucination detectors* which try to identify hallucinations either from the LM output itself, or from the LM's internal representations (e.g., Kadavath et al., 2022; Dhuliawala et al., 2023; OpenAI, 2023; Li et al., 2023). Our aim is to understand (i) how the effectiveness of hallucination detectors depends on the scale and training length of the LM they are judging, (ii) what types of detectors perform better, and (iii) if there is evidence other interventions beyond detectors are needed.  We explore two types of detection tasks: - sentence: The detector ingests both *original* subject-predicate and the *predicted* object tokens, and judges if the object is hallucinated.1 - token: The detector takes embedding of a token from a given layer of the trained LM, and is asked to say if it is hallucinated. For a given subject-predicate input, a predicted object token is labelled as hallucinated if it and the tokens preceding it do not match the token prefix of any entry in the KG. The goal is to predict the first hallucinated token; any tokens that come after are discarded from the detector's training and evaluation data.  For each of the tasks, we experiment with two types of hallucination detectors: - head: Adds a new readout head on top of the pretrained LM that produced the output (remaining LM weights are frozen). For the sentence task, this is equivalent to adding two new tokens to the dictionary (hallucination & non-hallucination), and finetuning the LM readout layer to predict whether the preceding triplet is hallucinated. The same holds for the token task, but the point of prediction changes; we also experiment with training detectors based on token representations from after each transformer block (Figure 10).  - full: Takes the pretrained LM as a starting point, and finetunes all its weights. For this setup, we only consider the case where the readout head is attached to the top layer embeddings in the token task.  7 The combination of token task and head detector applied to the top layer embeddings is similar to the approach taken in Kadavath et al. (2022). For token and full, the results should provide an upper bound on the performance achievable by finetuning the whole LM to say 'I don't know' (IDK), as we do not force the detector to preserve any of the other LM capabilities.2 For the sentence task, the setup is related to techniques based on post-hoc verification, including self-critique (e.g., OpenAI, 2023). The combination with full detectors in particular should provide an upper bound on performance achievable by self-critique, as we force the preserve any of the other LM capabilities.3 We trained a distinct detector (of each type) for every combination of a pretrained LM and detection task. Training and evaluation data are obtained by generating 5 object predictions for every subject-predicate in the LM training set. This data is split into a detector training (90%), validation (5%), and test sets (5%). The validation set is used for hyperparameter tuning and early stopping, the test set for measuring performance.  For the token experiments, we use the Adam optimizer (Kingma & Ba, 2014), with 1K warm-up steps and cosine decay (5K steps for head, 20K steps for full). Peak learning rates 1e-4 was used for head, and 5e-5 for full detectors. Training length and learning rates were determined using the validation set, optimizing for high AUC-PR. We did not finetune every possible combination of LM, detector task, and algorithm due to the large size of the Cartesian product (over 300). Visually, the above choices were optimal (or close) for most setups, but may not be optimal.  For the sentence experiments, the Adam optimizer alone underperformed for the full detectors, so we switched to *linear probing then full finetuning* (LPFT; Kumar et al., 2022).  LPFT works in two stages: (i) only the readout layer is optimized, with all other weights frozen; (ii) both readout and other layer weights are optimized. We used the Adafactor optimizer (Shazeer & Stern, 2018) for both stages. In the first, we used a cosine decay schedule with peak learning rate 1e-2 for 10K steps. In the second, we used linear warmup (10K steps) combined with cosine decay (250K steps) with peak learning rate 1e-3. For head detectors, only the first stage was used. The number of training steps and learning rates were determined using the validation set, aiming for high AUC-PR. Unlike in the token setup, we found it harder to find a common set of hyperparameters that would work for all detectors. 
body
Training Language Models On The Knowledge Graph: Insights On Hallucinations And Their Detectability
Hallucination Detectability And How It Scales 4.1 Setup
2408.07852
 Figure 5 shows how the pretrained LM size affects overall accuracy across tasks and approaches (Section 4.1). As expected, the full detectors outperform head, as they tend to be more flexible. Since this is also the case for other metrics, we focus on full detectors in the rest of this section. The token task formulation generally yielded better detector accuracy than the sentence task. Detector accuracy also tends to grow with the size of the underlying LM. However, these results are confounded by sensitivity of the accuracy metric to the underlying LM hallucination rate (see Figure 2). For example, if the rate is only 5%, even a naive detector which catches all hallucinations attains 95% accuracy.  We use AUC-PR to assess how well our detectors identify hallucinations. In Figure 6, we observe that: (a) sentence task formulation is superior in terms of AUC-PR, and (b) the lower the LM's hallucination rate, the worse the detector's AUC-PR. Per Figure 2, the lowest hallucination rates are achieved by the largest longest trained models. Thus, for a fixed training set size, there is an *inverse relationship* between the FLOPs spend on the LM pretraining and the detectability of its hallucinations.  Figure 7 emphasizes the inverse relationship between LM scale and hallucination detectability, showing the PR curves corresponding to the sentence task AUC-PR values in Figure 6. Note the general ordering of the curves, with those corresponding to detectors for the largest LMs being the lowest, and the ones for the smallest LMs being the highest.  ![8_image_0.png](8_image_0.png)  ![8_image_1.png](8_image_1.png) 
body
Training Language Models On The Knowledge Graph: Insights On Hallucinations And Their Detectability
Results
2408.07852
 Several factors may limit correspondence of our results to behaviour of state-of-the-art (SOTA) LMs. Firstly, the KG differs from the data normally used for LM training (e.g., no repetition of facts in the corpora, no semantic ambiguity, simple sentence structure).  Secondly, the LMs we train are significantly smaller than SOTA LMs, for which we adjust by using a proportionally smaller dataset. While we believe the qualitative interpretation of our results would generalize, we cannot rule out appearance of emergent capabilities  ![9_image_0.png](9_image_0.png)  (Wei et al., 2022) which may, e.g., significantly reduce the hallucination rate on unseen data. Thirdly, we only study hallucinations that result in not remembering something that appeared verbatim in the training data. While this has enabled most of our analysis, some of our conclusions may not translate to other types of hallucinations.  For hallucination detectors, we focused on general trends rather than exhaustive coverage of existing methods, and it is possible that alternative methods could have yielded better results. However, the qualitative consistency of the results relating to dependence of detector performance on LM scale suggests these are general properties of hallucination detectors.  Finally, dataset imbalance driven by varying hallucination rates between our LMs had a strong effect on performance of the hallucination detectors. We did not experiment with methods to adjust for such imbalance, beyond implicitly measuring performance at different score cut-offs via AUC-PR and the PR curves. 
body
Training Language Models On The Knowledge Graph: Insights On Hallucinations And Their Detectability
Limitations
2408.07852
 We explored the relationship between LM hallucinations and their scale. In particular, our paper sheds light on what it would take to eliminate factual hallucinations in the case where the correct answers—and no wrong answers—appear verbatim in the training set. Our study can also be seen as studying how much computation is needed for an LM to memorize all the knowledge content of a dataset. For a fixed dataset size, larger longer trained LMs hallucinate less. However, we found increasing dataset size also increased hallucination rate, given a fixed number of training epochs and LM size. This is because a larger dataset means more facts to learn. Similar to Kandpal et al. (2023), we hypothesise that standard LM training data contains repeated facts—and other repetitive text structures that do not require hard memorization—which makes the effect of dataset size on loss positive.  We found achieving a ≤5% hallucination rate on training data requires LMs much larger than currently considered optimal (Hoffmann et al., 2022), trained for much longer (20+ epochs). Moreover, the longer training can hurt LM generalisation, presenting a trade-off between hallucination rate and other performance metrics. Given the difficulty of eliminating hallucinations outright, we explored how detectability of LM hallucinations depends on scale. We found an inverse relationship between LM scale and the detectability of its hallucinations. That is, while larger, longer-trained LMs hallucinate less, detecting their remaining hallucinations becomes increasingly hard. 
conclusion
Training Language Models On The Knowledge Graph: Insights On Hallucinations And Their Detectability
Conclusion
2408.07852
 Ayush Agrawal, Lester Mackey, and Adam Kalai. Do language models know when they're hallucinating references? *arXiv*, abs/2306.03341, 2023.  Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. Improving language models by retrieving from trillions of tokens. In *ICML*, 2022.  Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. In *ICLR*, 2023.  Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. *JMLR*, 2023.  Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason Weston. Chain-of-verification reduces hallucination in large language models, 2023. Nouha Dziri, Andrea Madotto, Osmar Za¨ıane, and Avishek Joey Bose. Neural path hunter: Reducing hallucination in dialogue systems via path grounding. In *EMNLP*, 2021.  Nouha Dziri, Sivan Milton, Mo Yu, Osmar Zaiane, and Siva Reddy. On the origin of hallucinations in conversational models: Is it the datasets or the models? In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2022. Team Gemini. Gemini: a family of highly capable multimodal models. *arXiv*, 2023.  Google. Introducing the knowledge graph: things, not strings, 2012. URL https://blog.  google/products/search/introducing-knowledge-graph-things-not/. Accessed: 202401-15.  Roger Baker Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, Evan Hubinger, Kamil.e Lukovsiut.e, Karina Nguyen, Nicholas Joseph, Sam McCandlish, Jared Kaplan, and Sam Bowman. Studying large language model generalization with influence functions. *arXiv*, 2023.  Kelvin Guu, Albert Webson, Ellie Pavlick, Lucas Dixon, Ian Tenney, and Tolga Bolukbasi.  Simfluence: Modeling the influence of individual training examples by simulating training runs. *arXiv*, 2023.  Danny Hernandez, Tom Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer ElShowk, Nelson Elhage, Zac Hatfield-Dodds, Tom Henighan, Tristan Hume, et al. Scaling laws and interpretability of learning from repeated data. *arXiv*, 2022.  Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. *arXiv*, 2022.  Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. *ACM Comput. Surv.*, 2023.  Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Kaplan. Language models (mostly) know what they know, 2022. N Kandpal, H Deng, A Roberts, E Wallace, and C Raffel. Large language models struggle to learn long-tail knowledge. In *ICML*, 2023.  Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. *arXiv*, 2020. Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. *arXiv*, 2014.  Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform out-of-distribution. *arXiv*, 2022.  Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen-tau Yih, Tim Rockt ¨ aschel, et al. Retrieval- ¨ augmented generation for knowledge-intensive NLP tasks. *NeurIPS*, 2020.  Kenneth Li, Oam Patel, Fernanda Vi'egas, Hans-Rudiger Pfister, and Martin Wattenberg. ¨ Inference-time intervention: Eliciting truthful answers from a language model. *arXiv*, abs/2306.03341, 2023.  Niklas Muennighoff, Alexander M Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Nouamane Tazi, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. Scaling data-constrained language models. *arXiv*, 2023.  OpenAI. Gpt-4 technical report. *ArXiv*, abs/2303.08774, 2023.  Hannah Rashkin, Vitaly Nikolaev, Matthew Lamm, Michael Collins, Dipanjan Das, Slav Petrov, Gaurav Singh Tomar, Iulia Turc, and D. Reitter. Measuring attribution in natural language generation models. *arXiv*, abs/2112.12870, 2021. URL https://api.  semanticscholar.org/CorpusID:245502761. John Schulman. Reinforcement learning from human feedback: Progress and challenges, 2023. URL https://www.youtube.com/live/hhiLw5Q UFg?si=xH2Oq4C5QbpfV0SL&t=767.  Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In *ICML*, 2018.  Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Rozi ´ ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. ` LLaMA: Open and efficient foundation language models. *arXiv*, 2023.  Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. *NeurIPS*, 2017.  Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. *arXiv*, 2022. Fuzhao Xue, Yao Fu, Wangchunshu Zhou, Zangwei Zheng, and Yang You. To repeat or not to repeat: Insights from scaling LLM under token-crisis. *arXiv*, 2023. 
references
Training Language Models On The Knowledge Graph: Insights On Hallucinations And Their Detectability
References
2408.07852
 Hallucination in the context of natural language processing (NLP) is usually defined as the phenomenon of model-generated text that is seemingly fluent and coherent, but is actually nonsensical or unfaithful to the provided source content (Ji et al., 2023). Although the study of hallucinations in LMs is nascent, there is significant interest due to the challenges and risks they pose. Here we discuss a variety of relevant work, but direct the reader to (Ji et al., 2023) for a more comprehensive survey.  Kandpal et al. (2023) found that LMs struggle to learn long-tail knowledge that is not seen by the model multiple times during training. This is consistent with our finding that multiple passes over triplets are needed for the model to memorize facts. Kadavath et al. (2022) performed a large scale study to determine if models can give an indication of whether they will give a correct response to a multiple choice question. They found that for the majority of questions, the model was well calibrated in predicting correctness, with the exception of long-tail or out-of-distribution data. Carlini et al. (2023) study memorization in LMs and, similarly to this work, find that models memorize more with respect to scale and the number of times a concept is seen during training. We find that even in larger models, the hallucination rate remains above 5% on the training set even after 100 passes over the data. Dziri et al. (2022) found that the phenomenon of hallucinations is compounded by datasets and benchmarks which are themselves rife with hallucinated data. Agrawal et al. (2023) study hallucinations of references and find that inconsistency of multiple sampled references is a strong indicator of hallucinations.  Some recent work (Guu et al., 2023; Grosse et al., 2023) has developed methods to attribute predictions to individual training examples using influence functions. However, computing influence functions exhaustively over the training data remains prohibitively expensive. Therefore, we study hallucinations in the setting where we can know and control relationships between facts contained in the data that is exposed to the model.  Mitigation techniques range from automated model rewrites (Dhuliawala et al., 2023; OpenAI, 2023), attribution and grounding (Rashkin et al., 2021), to classification (probing) based on model internals. Dziri et al. (2021) use a critic and retrieval augmented dialogue model to retrieve facts from a KG. Li et al. (2023) use linear probes to detect internal model attention heads that are more likely to indicate truthfulness, and use this to push the model more towards truthful responses. In this work, we try to assess, in a best case scenario, how well either works to mitigate hallucinations. 
body
Training Language Models On The Knowledge Graph: Insights On Hallucinations And Their Detectability
Appendix A Related Work
2408.07852
 As mentioned in Section 2.2, the base learning rate for each model size is set to be a constant (a hyperparameter searched over 2.5, 5, and 10) divided by the square root of the number of LM's non-embedding parameters. The resulting exact learning rates are listed in Table 3.  Model size Learning rate  3.15M 0.002885  25.2M 0.001  113M 0.00047  403M 0.00025  1.61B 0.00011  Table 3: Base learning rates used for various model sizes.  13  ![13_image_0.png](13_image_0.png)  Table 4: Number of training steps for different data sizes. 
body
Training Language Models On The Knowledge Graph: Insights On Hallucinations And Their Detectability
B Learning Rates, Number Of Training Steps
2408.07852
 ![13_image_1.png](13_image_1.png)    ![14_image_0.png](14_image_0.png)   ![15_image_0.png](15_image_0.png)   ![15_image_1.png](15_image_1.png)  
body
Training Language Models On The Knowledge Graph: Insights On Hallucinations And Their Detectability
C Additional Plots