<!DOCTYPE html><html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>[2407.09276] H2O-Danube3 Technical Report</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="H2O-Danube3 Technical Report">
<meta name="twitter:image:src" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta name="twitter:image:alt" content="ar5iv logo">
<meta property="og:title" content="H2O-Danube3 Technical Report">
<meta property="og:site_name" content="ar5iv">
<meta property="og:image" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta property="og:type" content="article">
<meta property="og:url" content="https://ar5iv.labs.arxiv.org/html/2407.09276">

<!--Generated on Mon Aug  5 18:02:16 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.-->
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

<script>
  function detectColorScheme(){
    var theme="light";
    var current_theme = localStorage.getItem("ar5iv_theme");
    if(current_theme){
      if(current_theme == "dark"){
        theme = "dark";
      } }
    else if(!window.matchMedia) { return false; }
    else if(window.matchMedia("(prefers-color-scheme: dark)").matches) {
      theme = "dark"; }
    if (theme=="dark") {
      document.documentElement.setAttribute("data-theme", "dark");
    } else {
      document.documentElement.setAttribute("data-theme", "light"); } }

  detectColorScheme();

  function toggleColorScheme(){
    var current_theme = localStorage.getItem("ar5iv_theme");
    if (current_theme) {
      if (current_theme == "light") {
        localStorage.setItem("ar5iv_theme", "dark"); }
      else {
        localStorage.setItem("ar5iv_theme", "light"); } }
    else {
        localStorage.setItem("ar5iv_theme", "dark"); }
    detectColorScheme(); }
</script>
<link media="all" rel="stylesheet" href="/assets/ar5iv-fonts.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv-site.0.2.2.css">
</head>
<body>
<div class="ltx_page_main">
<div class="ltx_page_content">
<article class="ltx_document ltx_authors_1line">
<h1 class="ltx_title ltx_title_document">H2O-Danube3 Technical Report</h1>
<div class="ltx_authors">
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">
Pascal Pfeiffer
   Philipp Singer
   Yauhen Babakhin 
<br class="ltx_break">   <span id="id6.6.id1" class="ltx_text ltx_font_bold">Gabor Fodor</span>
   <span id="id7.7.id2" class="ltx_text ltx_font_bold">Nischay Dhankhar</span>
   <span id="id8.8.id3" class="ltx_text ltx_font_bold">Sri Satish Ambati
<br class="ltx_break"></span>H2O.ai 
<br class="ltx_break"><span id="id9.9.id4" class="ltx_text ltx_font_typewriter">{firstname.lastname, sri}@h2o.ai</span>
</span></span>
</div>

<section id="S1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">1 </span>Abstract</h2>

<div id="S1.p1" class="ltx_para">
<p id="S1.p1.2" class="ltx_p">We present <em id="S1.p1.2.1" class="ltx_emph ltx_font_italic">H2O-Danube3</em>, a series of small language models consisting of <span id="S1.p1.2.2" class="ltx_text ltx_font_italic">H2O-Danube3-4B</span>, trained on <math id="S1.p1.1.m1.1" class="ltx_Math" alttext="6T" display="inline"><semantics id="S1.p1.1.m1.1a"><mrow id="S1.p1.1.m1.1.1" xref="S1.p1.1.m1.1.1.cmml"><mn id="S1.p1.1.m1.1.1.2" xref="S1.p1.1.m1.1.1.2.cmml">6</mn><mo lspace="0em" rspace="0em" id="S1.p1.1.m1.1.1.1" xref="S1.p1.1.m1.1.1.1.cmml">​</mo><mi id="S1.p1.1.m1.1.1.3" xref="S1.p1.1.m1.1.1.3.cmml">T</mi></mrow><annotation-xml encoding="MathML-Content" id="S1.p1.1.m1.1b"><apply id="S1.p1.1.m1.1.1.cmml" xref="S1.p1.1.m1.1.1"><times id="S1.p1.1.m1.1.1.1.cmml" xref="S1.p1.1.m1.1.1.1"></times><cn type="integer" id="S1.p1.1.m1.1.1.2.cmml" xref="S1.p1.1.m1.1.1.2">6</cn><ci id="S1.p1.1.m1.1.1.3.cmml" xref="S1.p1.1.m1.1.1.3">𝑇</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.p1.1.m1.1c">6T</annotation></semantics></math> tokens and <span id="S1.p1.2.3" class="ltx_text ltx_font_italic">H2O-Danube3-500M</span>, trained on <math id="S1.p1.2.m2.1" class="ltx_Math" alttext="4T" display="inline"><semantics id="S1.p1.2.m2.1a"><mrow id="S1.p1.2.m2.1.1" xref="S1.p1.2.m2.1.1.cmml"><mn id="S1.p1.2.m2.1.1.2" xref="S1.p1.2.m2.1.1.2.cmml">4</mn><mo lspace="0em" rspace="0em" id="S1.p1.2.m2.1.1.1" xref="S1.p1.2.m2.1.1.1.cmml">​</mo><mi id="S1.p1.2.m2.1.1.3" xref="S1.p1.2.m2.1.1.3.cmml">T</mi></mrow><annotation-xml encoding="MathML-Content" id="S1.p1.2.m2.1b"><apply id="S1.p1.2.m2.1.1.cmml" xref="S1.p1.2.m2.1.1"><times id="S1.p1.2.m2.1.1.1.cmml" xref="S1.p1.2.m2.1.1.1"></times><cn type="integer" id="S1.p1.2.m2.1.1.2.cmml" xref="S1.p1.2.m2.1.1.2">4</cn><ci id="S1.p1.2.m2.1.1.3.cmml" xref="S1.p1.2.m2.1.1.3">𝑇</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.p1.2.m2.1c">4T</annotation></semantics></math> tokens.
Our models are pre-trained on high quality Web data consisting of primarily English tokens in three stages with different data mixes before final supervised tuning for chat version.
The models exhibit highly competitive metrics across a multitude of academic, chat, and fine-tuning benchmarks.
Thanks to its compact architecture, <span id="S1.p1.2.4" class="ltx_text ltx_font_italic">H2O-Danube3</span> can be efficiently run on a modern smartphone, enabling local inference and rapid processing capabilities even on mobile devices.
We make all models openly available under Apache 2.0 license further democratizing LLMs to a wider audience economically.</p>
</div>
<div id="S1.p2" class="ltx_para">
<p id="S1.p2.1" class="ltx_p"><span id="S1.p2.1.1" class="ltx_text ltx_font_bold">Danube3 model collection:
<br class="ltx_break"><a target="_blank" href="https://huggingface.co/collections/h2oai/h2o-danube3-6687a993641452457854c609" title="" class="ltx_ref ltx_url ltx_font_typewriter ltx_font_medium">https://huggingface.co/collections/h2oai/h2o-danube3-6687a993641452457854c609</a></span></p>
</div>
</section>
<section id="S2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">2 </span>Introduction</h2>

<div id="S2.p1" class="ltx_para">
<p id="S2.p1.1" class="ltx_p">Small language models have taken a pivotal place in today’s open source language model landscape particularly aiming at efficient inference on consumer hardware and edge devices also allowing for full offline applications. Additionally, smaller models have proven to be particularly useful after fine-tuning them for specific tasks, such as sequence classification, question answering, or token classification even outpacing previously used encoder/decoder models such as those stemming from BERT and its derivatives <cite class="ltx_cite ltx_citemacro_cite"><a href="#bib.bib4" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">devlin2018bert</span> </a>; <a href="#bib.bib7" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">he2020deberta</span> </a></cite>.</p>
</div>
<div id="S2.p2" class="ltx_para">
<p id="S2.p2.2" class="ltx_p">We extend previous research in this area <cite class="ltx_cite ltx_citemacro_cite"><a href="#bib.bib10" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">liu2024mobilellm</span> </a>; <a href="#bib.bib3" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">biderman2023pythia</span> </a>; <a href="#bib.bib14" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">zhang2024tinyllama</span> </a>; <a href="#bib.bib15" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">zhang2022opt</span> </a>; <a href="#bib.bib2" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">bai2023qwen</span> </a>; <a href="#bib.bib12" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">stablelm</span> </a>; <a href="#bib.bib11" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">h2odanube18b</span> </a>; <a href="#bib.bib5" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">phi3_2024</span> </a></cite> and present <span id="S2.p2.2.1" class="ltx_text ltx_font_italic">H2O-Danube3</span>, a series of small language models consisting of <span id="S2.p2.2.2" class="ltx_text ltx_font_italic">H2O-Danube3-4B</span>, trained on <math id="S2.p2.1.m1.1" class="ltx_Math" alttext="6T" display="inline"><semantics id="S2.p2.1.m1.1a"><mrow id="S2.p2.1.m1.1.1" xref="S2.p2.1.m1.1.1.cmml"><mn id="S2.p2.1.m1.1.1.2" xref="S2.p2.1.m1.1.1.2.cmml">6</mn><mo lspace="0em" rspace="0em" id="S2.p2.1.m1.1.1.1" xref="S2.p2.1.m1.1.1.1.cmml">​</mo><mi id="S2.p2.1.m1.1.1.3" xref="S2.p2.1.m1.1.1.3.cmml">T</mi></mrow><annotation-xml encoding="MathML-Content" id="S2.p2.1.m1.1b"><apply id="S2.p2.1.m1.1.1.cmml" xref="S2.p2.1.m1.1.1"><times id="S2.p2.1.m1.1.1.1.cmml" xref="S2.p2.1.m1.1.1.1"></times><cn type="integer" id="S2.p2.1.m1.1.1.2.cmml" xref="S2.p2.1.m1.1.1.2">6</cn><ci id="S2.p2.1.m1.1.1.3.cmml" xref="S2.p2.1.m1.1.1.3">𝑇</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.p2.1.m1.1c">6T</annotation></semantics></math> tokens and <span id="S2.p2.2.3" class="ltx_text ltx_font_italic">H2O-Danube3-500M</span>, trained on <math id="S2.p2.2.m2.1" class="ltx_Math" alttext="4T" display="inline"><semantics id="S2.p2.2.m2.1a"><mrow id="S2.p2.2.m2.1.1" xref="S2.p2.2.m2.1.1.cmml"><mn id="S2.p2.2.m2.1.1.2" xref="S2.p2.2.m2.1.1.2.cmml">4</mn><mo lspace="0em" rspace="0em" id="S2.p2.2.m2.1.1.1" xref="S2.p2.2.m2.1.1.1.cmml">​</mo><mi id="S2.p2.2.m2.1.1.3" xref="S2.p2.2.m2.1.1.3.cmml">T</mi></mrow><annotation-xml encoding="MathML-Content" id="S2.p2.2.m2.1b"><apply id="S2.p2.2.m2.1.1.cmml" xref="S2.p2.2.m2.1.1"><times id="S2.p2.2.m2.1.1.1.cmml" xref="S2.p2.2.m2.1.1.1"></times><cn type="integer" id="S2.p2.2.m2.1.1.2.cmml" xref="S2.p2.2.m2.1.1.2">4</cn><ci id="S2.p2.2.m2.1.1.3.cmml" xref="S2.p2.2.m2.1.1.3">𝑇</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.p2.2.m2.1c">4T</annotation></semantics></math> tokens based on incremental research and training efforts <cite class="ltx_cite ltx_citemacro_cite"><a href="#bib.bib11" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">h2odanube18b</span> </a></cite>.
In this report, we present an overview of the models, detailing their architecture, training procedures, and fine-tuning processes. We offer extensive evaluations using a diverse range of benchmarks, encompassing both standard academic metrics, chat benchmarks, and fine-tuning benchmarks.</p>
</div>
<div id="S2.p3" class="ltx_para">
<p id="S2.p3.1" class="ltx_p">Results show that <span id="S2.p3.1.1" class="ltx_text ltx_font_italic">H2O-Danube3</span> exhibits competitive benchmarks across all dimensions, expanding the repertoire of open source small language models. We hope our work can further democratize language models to a wider audience and that our models can play a pivotal role for various use cases such as (1) chatbot applications, (2) RAG applications, (3) fine-tuning for specific use cases such as classification, (4) research or (5) on-device offline applications. To demonstrate the potential, we also present <em id="S2.p3.1.2" class="ltx_emph ltx_font_italic">H2O AI Personal GPT</em><span id="footnote1" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">1</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">1</sup><span class="ltx_tag ltx_tag_note">1</span><a target="_blank" href="https://h2o.ai/platform/danube/personal-gpt/" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://h2o.ai/platform/danube/personal-gpt/</a></span></span></span>, an iOS application allowing to run <span id="S2.p3.1.3" class="ltx_text ltx_font_italic">H2O-Danube3</span> fully offline on a modern phone device.</p>
</div>
<figure id="S2.F1" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2"><img src="/html/2407.09276/assets/stage1.png" id="S2.F1.g1" class="ltx_graphics ltx_centering ltx_figure_panel ltx_img_landscape" width="287" height="178" alt="Refer to caption"></div>
<div class="ltx_flex_cell ltx_flex_size_2"><img src="/html/2407.09276/assets/stage2.png" id="S2.F1.g2" class="ltx_graphics ltx_centering ltx_figure_panel ltx_img_landscape" width="287" height="178" alt="Refer to caption"></div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1"><img src="/html/2407.09276/assets/stage3.png" id="S2.F1.g3" class="ltx_graphics ltx_centering ltx_figure_panel ltx_img_landscape" width="287" height="178" alt="Refer to caption"></div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 1: </span><span id="S2.F1.2.1" class="ltx_text ltx_font_bold">Data stages for <span id="S2.F1.2.1.1" class="ltx_text ltx_font_italic">H2O-Danube3-4B</span>.</span> The model is trained over three different stages with different data mixes. The first data stage consist of 90.6% of web data which is gradually decreasing to 81.7% at the second stage, and to 51.6% at the third stage. The first two stages include the majority of the tokens: 4.6T and 1.35T tokens respectively, while the third stage comprises of 0.05T tokens.</figcaption>
</figure>
</section>
<section id="S3" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">3 </span>Model architecture</h2>

<div id="S3.p1" class="ltx_para">
<p id="S3.p1.2" class="ltx_p"><span id="S3.p1.2.1" class="ltx_text ltx_font_italic">H2O-Danube3</span> is a family of decoder only LLM models that use the general Llama model architecture adopting core principles from Llama 2 <cite class="ltx_cite ltx_citemacro_cite"><a href="#bib.bib13" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">touvron2023llama</span> </a></cite> and Mistral <cite class="ltx_cite ltx_citemacro_cite"><a href="#bib.bib8" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">jiang2023mistral</span> </a></cite> with custom parameters determining the shape of each layer and total parameter count. We use the Mistral tokenizer with a vocabulary size of <math id="S3.p1.1.m1.2" class="ltx_Math" alttext="32,000" display="inline"><semantics id="S3.p1.1.m1.2a"><mrow id="S3.p1.1.m1.2.3.2" xref="S3.p1.1.m1.2.3.1.cmml"><mn id="S3.p1.1.m1.1.1" xref="S3.p1.1.m1.1.1.cmml">32</mn><mo id="S3.p1.1.m1.2.3.2.1" xref="S3.p1.1.m1.2.3.1.cmml">,</mo><mn id="S3.p1.1.m1.2.2" xref="S3.p1.1.m1.2.2.cmml">000</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.p1.1.m1.2b"><list id="S3.p1.1.m1.2.3.1.cmml" xref="S3.p1.1.m1.2.3.2"><cn type="integer" id="S3.p1.1.m1.1.1.cmml" xref="S3.p1.1.m1.1.1">32</cn><cn type="integer" id="S3.p1.1.m1.2.2.cmml" xref="S3.p1.1.m1.2.2">000</cn></list></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.1.m1.2c">32,000</annotation></semantics></math> and train our model up to a context length of <math id="S3.p1.2.m2.2" class="ltx_Math" alttext="8,192" display="inline"><semantics id="S3.p1.2.m2.2a"><mrow id="S3.p1.2.m2.2.3.2" xref="S3.p1.2.m2.2.3.1.cmml"><mn id="S3.p1.2.m2.1.1" xref="S3.p1.2.m2.1.1.cmml">8</mn><mo id="S3.p1.2.m2.2.3.2.1" xref="S3.p1.2.m2.2.3.1.cmml">,</mo><mn id="S3.p1.2.m2.2.2" xref="S3.p1.2.m2.2.2.cmml">192</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.p1.2.m2.2b"><list id="S3.p1.2.m2.2.3.1.cmml" xref="S3.p1.2.m2.2.3.2"><cn type="integer" id="S3.p1.2.m2.1.1.cmml" xref="S3.p1.2.m2.1.1">8</cn><cn type="integer" id="S3.p1.2.m2.2.2.cmml" xref="S3.p1.2.m2.2.2">192</cn></list></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.2.m2.2c">8,192</annotation></semantics></math>. We make use of Grouped Query Attention <cite class="ltx_cite ltx_citemacro_cite"><a href="#bib.bib1" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">ainslie2023gqa</span> </a></cite> and optimize towards parameter and compute efficiency resulting in a wide architecture (see Table <a href="#S3.T1" title="Table 1 ‣ 3 Model architecture ‣ H2O-Danube3 Technical Report" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>). In total, <span id="S3.p1.2.2" class="ltx_text ltx_font_italic">H2O-Danube3-4B</span> consists of 3.96B trainable parameters. In addition, we release <span id="S3.p1.2.3" class="ltx_text ltx_font_italic">H2O-Danube3-500M</span> with 500M trainable parameters for edge devices with limited compute or for custom fine-tuning tasks that require low memory footprint or high throughput at low cost.</p>
</div>
<figure id="S3.T1" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 1: </span>Key model parameters.</figcaption>
<table id="S3.T1.1" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S3.T1.1.1.1" class="ltx_tr">
<th id="S3.T1.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt">Parameters</th>
<th id="S3.T1.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">500M</th>
<th id="S3.T1.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">4B</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S3.T1.1.2.1" class="ltx_tr">
<th id="S3.T1.1.2.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">Layers</th>
<td id="S3.T1.1.2.1.2" class="ltx_td ltx_align_center ltx_border_t">16</td>
<td id="S3.T1.1.2.1.3" class="ltx_td ltx_align_center ltx_border_t">24</td>
</tr>
<tr id="S3.T1.1.3.2" class="ltx_tr">
<th id="S3.T1.1.3.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Hidden size</th>
<td id="S3.T1.1.3.2.2" class="ltx_td ltx_align_center">1536</td>
<td id="S3.T1.1.3.2.3" class="ltx_td ltx_align_center">3840</td>
</tr>
<tr id="S3.T1.1.4.3" class="ltx_tr">
<th id="S3.T1.1.4.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Intermediate size</th>
<td id="S3.T1.1.4.3.2" class="ltx_td ltx_align_center">4096</td>
<td id="S3.T1.1.4.3.3" class="ltx_td ltx_align_center">10240</td>
</tr>
<tr id="S3.T1.1.5.4" class="ltx_tr">
<th id="S3.T1.1.5.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Num heads</th>
<td id="S3.T1.1.5.4.2" class="ltx_td ltx_align_center">16</td>
<td id="S3.T1.1.5.4.3" class="ltx_td ltx_align_center">32</td>
</tr>
<tr id="S3.T1.1.6.5" class="ltx_tr">
<th id="S3.T1.1.6.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Num KV heads</th>
<td id="S3.T1.1.6.5.2" class="ltx_td ltx_align_center">8</td>
<td id="S3.T1.1.6.5.3" class="ltx_td ltx_align_center">8</td>
</tr>
<tr id="S3.T1.1.7.6" class="ltx_tr">
<th id="S3.T1.1.7.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Head size</th>
<td id="S3.T1.1.7.6.2" class="ltx_td ltx_align_center">96</td>
<td id="S3.T1.1.7.6.3" class="ltx_td ltx_align_center">120</td>
</tr>
<tr id="S3.T1.1.8.7" class="ltx_tr">
<th id="S3.T1.1.8.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Vocab size</th>
<td id="S3.T1.1.8.7.2" class="ltx_td ltx_align_center">32000</td>
<td id="S3.T1.1.8.7.3" class="ltx_td ltx_align_center">32000</td>
</tr>
<tr id="S3.T1.1.9.8" class="ltx_tr">
<th id="S3.T1.1.9.8.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">RoPE theta</th>
<td id="S3.T1.1.9.8.2" class="ltx_td ltx_align_center ltx_border_bb">100000</td>
<td id="S3.T1.1.9.8.3" class="ltx_td ltx_align_center ltx_border_bb">100000</td>
</tr>
</tbody>
</table>
</figure>
</section>
<section id="S4" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">4 </span>Training</h2>

<div id="S4.p1" class="ltx_para">
<p id="S4.p1.5" class="ltx_p">Models are primarily trained on English text in three stages with different data mixes. At each stage, we gradually decrease the percentage of noisy web data in favor of higher quality data. The first data stage consist of 90.6% of web data which is gradually decreasing to 81.7% at the second stage, and to 51.6% at the third stage. Simultaneously, the share of instruct data, Wikipedia, academic texts, synthetic texts and other higher quality textual data is increasing. The first two stages include the majority of the tokens: <math id="S4.p1.1.m1.1" class="ltx_Math" alttext="4.6T" display="inline"><semantics id="S4.p1.1.m1.1a"><mrow id="S4.p1.1.m1.1.1" xref="S4.p1.1.m1.1.1.cmml"><mn id="S4.p1.1.m1.1.1.2" xref="S4.p1.1.m1.1.1.2.cmml">4.6</mn><mo lspace="0em" rspace="0em" id="S4.p1.1.m1.1.1.1" xref="S4.p1.1.m1.1.1.1.cmml">​</mo><mi id="S4.p1.1.m1.1.1.3" xref="S4.p1.1.m1.1.1.3.cmml">T</mi></mrow><annotation-xml encoding="MathML-Content" id="S4.p1.1.m1.1b"><apply id="S4.p1.1.m1.1.1.cmml" xref="S4.p1.1.m1.1.1"><times id="S4.p1.1.m1.1.1.1.cmml" xref="S4.p1.1.m1.1.1.1"></times><cn type="float" id="S4.p1.1.m1.1.1.2.cmml" xref="S4.p1.1.m1.1.1.2">4.6</cn><ci id="S4.p1.1.m1.1.1.3.cmml" xref="S4.p1.1.m1.1.1.3">𝑇</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p1.1.m1.1c">4.6T</annotation></semantics></math> and <math id="S4.p1.2.m2.1" class="ltx_Math" alttext="1.35T" display="inline"><semantics id="S4.p1.2.m2.1a"><mrow id="S4.p1.2.m2.1.1" xref="S4.p1.2.m2.1.1.cmml"><mn id="S4.p1.2.m2.1.1.2" xref="S4.p1.2.m2.1.1.2.cmml">1.35</mn><mo lspace="0em" rspace="0em" id="S4.p1.2.m2.1.1.1" xref="S4.p1.2.m2.1.1.1.cmml">​</mo><mi id="S4.p1.2.m2.1.1.3" xref="S4.p1.2.m2.1.1.3.cmml">T</mi></mrow><annotation-xml encoding="MathML-Content" id="S4.p1.2.m2.1b"><apply id="S4.p1.2.m2.1.1.cmml" xref="S4.p1.2.m2.1.1"><times id="S4.p1.2.m2.1.1.1.cmml" xref="S4.p1.2.m2.1.1.1"></times><cn type="float" id="S4.p1.2.m2.1.1.2.cmml" xref="S4.p1.2.m2.1.1.2">1.35</cn><ci id="S4.p1.2.m2.1.1.3.cmml" xref="S4.p1.2.m2.1.1.3">𝑇</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p1.2.m2.1c">1.35T</annotation></semantics></math> tokens respectively (<math id="S4.p1.3.m3.1" class="ltx_Math" alttext="2.8T" display="inline"><semantics id="S4.p1.3.m3.1a"><mrow id="S4.p1.3.m3.1.1" xref="S4.p1.3.m3.1.1.cmml"><mn id="S4.p1.3.m3.1.1.2" xref="S4.p1.3.m3.1.1.2.cmml">2.8</mn><mo lspace="0em" rspace="0em" id="S4.p1.3.m3.1.1.1" xref="S4.p1.3.m3.1.1.1.cmml">​</mo><mi id="S4.p1.3.m3.1.1.3" xref="S4.p1.3.m3.1.1.3.cmml">T</mi></mrow><annotation-xml encoding="MathML-Content" id="S4.p1.3.m3.1b"><apply id="S4.p1.3.m3.1.1.cmml" xref="S4.p1.3.m3.1.1"><times id="S4.p1.3.m3.1.1.1.cmml" xref="S4.p1.3.m3.1.1.1"></times><cn type="float" id="S4.p1.3.m3.1.1.2.cmml" xref="S4.p1.3.m3.1.1.2">2.8</cn><ci id="S4.p1.3.m3.1.1.3.cmml" xref="S4.p1.3.m3.1.1.3">𝑇</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p1.3.m3.1c">2.8T</annotation></semantics></math> and <math id="S4.p1.4.m4.1" class="ltx_Math" alttext="1.15T" display="inline"><semantics id="S4.p1.4.m4.1a"><mrow id="S4.p1.4.m4.1.1" xref="S4.p1.4.m4.1.1.cmml"><mn id="S4.p1.4.m4.1.1.2" xref="S4.p1.4.m4.1.1.2.cmml">1.15</mn><mo lspace="0em" rspace="0em" id="S4.p1.4.m4.1.1.1" xref="S4.p1.4.m4.1.1.1.cmml">​</mo><mi id="S4.p1.4.m4.1.1.3" xref="S4.p1.4.m4.1.1.3.cmml">T</mi></mrow><annotation-xml encoding="MathML-Content" id="S4.p1.4.m4.1b"><apply id="S4.p1.4.m4.1.1.cmml" xref="S4.p1.4.m4.1.1"><times id="S4.p1.4.m4.1.1.1.cmml" xref="S4.p1.4.m4.1.1.1"></times><cn type="float" id="S4.p1.4.m4.1.1.2.cmml" xref="S4.p1.4.m4.1.1.2">1.15</cn><ci id="S4.p1.4.m4.1.1.3.cmml" xref="S4.p1.4.m4.1.1.3">𝑇</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p1.4.m4.1c">1.15T</annotation></semantics></math> tokens for <span id="S4.p1.5.1" class="ltx_text ltx_font_italic">H2O-Danube3-500M</span>), while third stage comprises of <math id="S4.p1.5.m5.1" class="ltx_Math" alttext="0.05T" display="inline"><semantics id="S4.p1.5.m5.1a"><mrow id="S4.p1.5.m5.1.1" xref="S4.p1.5.m5.1.1.cmml"><mn id="S4.p1.5.m5.1.1.2" xref="S4.p1.5.m5.1.1.2.cmml">0.05</mn><mo lspace="0em" rspace="0em" id="S4.p1.5.m5.1.1.1" xref="S4.p1.5.m5.1.1.1.cmml">​</mo><mi id="S4.p1.5.m5.1.1.3" xref="S4.p1.5.m5.1.1.3.cmml">T</mi></mrow><annotation-xml encoding="MathML-Content" id="S4.p1.5.m5.1b"><apply id="S4.p1.5.m5.1.1.cmml" xref="S4.p1.5.m5.1.1"><times id="S4.p1.5.m5.1.1.1.cmml" xref="S4.p1.5.m5.1.1.1"></times><cn type="float" id="S4.p1.5.m5.1.1.2.cmml" xref="S4.p1.5.m5.1.1.2">0.05</cn><ci id="S4.p1.5.m5.1.1.3.cmml" xref="S4.p1.5.m5.1.1.3">𝑇</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p1.5.m5.1c">0.05T</annotation></semantics></math> tokens. The data distribution across stages is presented in Figure <a href="#S2.F1" title="Figure 1 ‣ 2 Introduction ‣ H2O-Danube3 Technical Report" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>.</p>
</div>
<div id="S4.p2" class="ltx_para">
<p id="S4.p2.1" class="ltx_p">We also provide chat fine-tuned versions <span id="S4.p2.1.1" class="ltx_text ltx_font_italic">H2O-Danube3-4B-Chat</span> and <span id="S4.p2.1.2" class="ltx_text ltx_font_italic">H2O-Danube3-500M-Chat</span>. We utilize <em id="S4.p2.1.3" class="ltx_emph ltx_font_italic">H2O LLM Studio</em><span id="footnote2" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">2</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">2</sup><span class="ltx_tag ltx_tag_note">2</span><a target="_blank" href="https://github.com/h2oai/h2o-llmstudio" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/h2oai/h2o-llmstudio</a></span></span></span>, an Apache 2.0 open-source framework and no-code GUI for fine-tuning LLMs.
We tune the base model using supervised fine-tuning (SFT) on input/output conversational pairs. We mask the prompt loss, and use a custom prompt format. Hyperparameters were optimized iterating over multiple experiments.</p>
</div>
</section>
<section id="S5" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">5 </span>Evaluation</h2>

<div id="S5.p1" class="ltx_para">
<p id="S5.p1.1" class="ltx_p">In this section, we present evaluation of <span id="S5.p1.1.1" class="ltx_text ltx_font_italic">H2O-Danube3</span> across a variety of dimensions, focusing on (1) academic benchmarks, (2) chat benchmarks and (3) fine-tuning benchmarks.</p>
</div>
<div id="S5.p2" class="ltx_para ltx_noindent">
<p id="S5.p2.1" class="ltx_p"><span id="S5.p2.1.1" class="ltx_text ltx_font_bold">Academic benchmarks.</span> We evaluate <span id="S5.p2.1.2" class="ltx_text ltx_font_italic">H2O-Danube3</span> on a wide range of benchmarks and compare it with other existing open-source language models which have a similar number of parameters, specifically Qwen/Qwen1.5-4B-Chat, stabilityai/stablelm-zephyr-3b and microsoft/Phi-3-mini-4k-instruct. We also compare to our previous model h2oai/h2o-danube2-1.8b-chat. To evaluate the models, we use the Language Model Evaluation Harness framework<span id="footnote3" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">3</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">3</sup><span class="ltx_tag ltx_tag_note">3</span>commit e5e5ee0cb629c9c88165292d1b4bf34623392d33</span></span></span> <cite class="ltx_cite ltx_citemacro_cite"><a href="#bib.bib6" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">eval-harness</span> </a></cite>. <span id="S5.p2.1.3" class="ltx_text ltx_font_italic">H2O-Danube3-4B</span> shows very competitive and consistent results across all reported benchmarks (see Table <a href="#S5.T2" title="Table 2 ‣ 5 Evaluation ‣ H2O-Danube3 Technical Report" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>). It is the best-in-class model for the knowledge based CommonsenseQA benchmark and PhysicsQA and achieves a strong accuracy of 50.14% on the math centered benchmark GSM8K. In all other benchmarks, <span id="S5.p2.1.4" class="ltx_text ltx_font_italic">H2O-Danube3-4B</span> ranks second only after Phi-3-mini-4k-instruct which is well known for its outstanding reasoning capabilities and strong benchmark scores. Notably, <span id="S5.p2.1.5" class="ltx_text ltx_font_italic">H2O-Danube3-4B</span> scores over 80% on 10-shot hellaswag benchmark, closing the gap to much larger models. The smaller <span id="S5.p2.1.6" class="ltx_text ltx_font_italic">H2O-Danube3-500M</span> is evaluated against the same benchmarks and compared to similar sized Qwen2-0.5B-Instruct (see Table <a href="#S5.T3" title="Table 3 ‣ 5 Evaluation ‣ H2O-Danube3 Technical Report" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>). Our model scores highest in eight out of twelve benchmarks and we consider it a new well rounded model for this parameter count.</p>
</div>
<figure id="S5.T2" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 2: </span><span id="S5.T2.2.1" class="ltx_text ltx_font_bold">Academic benchmarks.</span> Academic benchmark results, compared to openly-available models of similar size and trained on general English text data. We compare the instruction fine-tuned models h2oai/h2o-danube2-1.8b-chat, h2oai/h2o-danube3-4b-chat, Qwen/Qwen1.5-4B-Chat, stabilityai/stablelm-zephyr-3b, microsoft/Phi-3-mini-4k-instruct. To evaluate the models, we use the Language Model Evaluation Harness framework <cite class="ltx_cite ltx_citemacro_cite"><a href="#bib.bib6" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">eval-harness</span> </a></cite>.</figcaption>
<table id="S5.T2.3" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S5.T2.3.1.1" class="ltx_tr">
<th id="S5.T2.3.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt">Benchmark</th>
<th id="S5.T2.3.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_tt">Metric</th>
<th id="S5.T2.3.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">Danube2</th>
<th id="S5.T2.3.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_tt">Danube3</th>
<th id="S5.T2.3.1.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">Qwen1.5</th>
<th id="S5.T2.3.1.1.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">StableLM</th>
<th id="S5.T2.3.1.1.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">Phi3</th>
</tr>
<tr id="S5.T2.3.2.2" class="ltx_tr">
<th id="S5.T2.3.2.2.1" class="ltx_td ltx_th ltx_th_column"></th>
<th id="S5.T2.3.2.2.2" class="ltx_td ltx_th ltx_th_column ltx_border_r"></th>
<th id="S5.T2.3.2.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_column">1.8B</th>
<th id="S5.T2.3.2.2.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r">4B</th>
<th id="S5.T2.3.2.2.5" class="ltx_td ltx_align_center ltx_th ltx_th_column">4B</th>
<th id="S5.T2.3.2.2.6" class="ltx_td ltx_align_center ltx_th ltx_th_column">3B</th>
<th id="S5.T2.3.2.2.7" class="ltx_td ltx_align_center ltx_th ltx_th_column">4B</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S5.T2.3.3.1" class="ltx_tr">
<td id="S5.T2.3.3.1.1" class="ltx_td ltx_align_left ltx_border_t">ARC-c</td>
<td id="S5.T2.3.3.1.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">25-shot</td>
<td id="S5.T2.3.3.1.3" class="ltx_td ltx_align_center ltx_border_t">43.69</td>
<td id="S5.T2.3.3.1.4" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">58.96</td>
<td id="S5.T2.3.3.1.5" class="ltx_td ltx_align_center ltx_border_t">42.15</td>
<td id="S5.T2.3.3.1.6" class="ltx_td ltx_align_center ltx_border_t">47.70</td>
<td id="S5.T2.3.3.1.7" class="ltx_td ltx_align_center ltx_border_t"><span id="S5.T2.3.3.1.7.1" class="ltx_text ltx_font_bold">63.91</span></td>
</tr>
<tr id="S5.T2.3.4.2" class="ltx_tr">
<td id="S5.T2.3.4.2.1" class="ltx_td ltx_align_left">Hellaswag</td>
<td id="S5.T2.3.4.2.2" class="ltx_td ltx_align_center ltx_border_r">10-shot</td>
<td id="S5.T2.3.4.2.3" class="ltx_td ltx_align_center">73.91</td>
<td id="S5.T2.3.4.2.4" class="ltx_td ltx_align_center ltx_border_r">80.36</td>
<td id="S5.T2.3.4.2.5" class="ltx_td ltx_align_center">69.46</td>
<td id="S5.T2.3.4.2.6" class="ltx_td ltx_align_center">73.71</td>
<td id="S5.T2.3.4.2.7" class="ltx_td ltx_align_center"><span id="S5.T2.3.4.2.7.1" class="ltx_text ltx_font_bold">80.62</span></td>
</tr>
<tr id="S5.T2.3.5.3" class="ltx_tr">
<td id="S5.T2.3.5.3.1" class="ltx_td ltx_align_left">MMLU</td>
<td id="S5.T2.3.5.3.2" class="ltx_td ltx_align_center ltx_border_r">5-shot</td>
<td id="S5.T2.3.5.3.3" class="ltx_td ltx_align_center">37.83</td>
<td id="S5.T2.3.5.3.4" class="ltx_td ltx_align_center ltx_border_r">54.74</td>
<td id="S5.T2.3.5.3.5" class="ltx_td ltx_align_center">54.03</td>
<td id="S5.T2.3.5.3.6" class="ltx_td ltx_align_center">44.98</td>
<td id="S5.T2.3.5.3.7" class="ltx_td ltx_align_center"><span id="S5.T2.3.5.3.7.1" class="ltx_text ltx_font_bold">69.43</span></td>
</tr>
<tr id="S5.T2.3.6.4" class="ltx_tr">
<td id="S5.T2.3.6.4.1" class="ltx_td ltx_align_left">TruthfulQA</td>
<td id="S5.T2.3.6.4.2" class="ltx_td ltx_align_center ltx_border_r">0-shot mc2</td>
<td id="S5.T2.3.6.4.3" class="ltx_td ltx_align_center">40.53</td>
<td id="S5.T2.3.6.4.4" class="ltx_td ltx_align_center ltx_border_r">47.79</td>
<td id="S5.T2.3.6.4.5" class="ltx_td ltx_align_center">44.88</td>
<td id="S5.T2.3.6.4.6" class="ltx_td ltx_align_center">46.40</td>
<td id="S5.T2.3.6.4.7" class="ltx_td ltx_align_center"><span id="S5.T2.3.6.4.7.1" class="ltx_text ltx_font_bold">57.72</span></td>
</tr>
<tr id="S5.T2.3.7.5" class="ltx_tr">
<td id="S5.T2.3.7.5.1" class="ltx_td ltx_align_left">Winogrande</td>
<td id="S5.T2.3.7.5.2" class="ltx_td ltx_align_center ltx_border_r">5-shot</td>
<td id="S5.T2.3.7.5.3" class="ltx_td ltx_align_center">69.30</td>
<td id="S5.T2.3.7.5.4" class="ltx_td ltx_align_center ltx_border_r"><span id="S5.T2.3.7.5.4.1" class="ltx_text ltx_font_bold">76.48</span></td>
<td id="S5.T2.3.7.5.5" class="ltx_td ltx_align_center">66.22</td>
<td id="S5.T2.3.7.5.6" class="ltx_td ltx_align_center">65.59</td>
<td id="S5.T2.3.7.5.7" class="ltx_td ltx_align_center">70.80</td>
</tr>
<tr id="S5.T2.3.8.6" class="ltx_tr">
<td id="S5.T2.3.8.6.1" class="ltx_td ltx_align_left">GSM8K</td>
<td id="S5.T2.3.8.6.2" class="ltx_td ltx_align_center ltx_border_r">5-shot</td>
<td id="S5.T2.3.8.6.3" class="ltx_td ltx_align_center">32.30</td>
<td id="S5.T2.3.8.6.4" class="ltx_td ltx_align_center ltx_border_r">50.18</td>
<td id="S5.T2.3.8.6.5" class="ltx_td ltx_align_center">3.63</td>
<td id="S5.T2.3.8.6.6" class="ltx_td ltx_align_center">52.46</td>
<td id="S5.T2.3.8.6.7" class="ltx_td ltx_align_center"><span id="S5.T2.3.8.6.7.1" class="ltx_text ltx_font_bold">77.48</span></td>
</tr>
<tr id="S5.T2.3.9.7" class="ltx_tr">
<td id="S5.T2.3.9.7.1" class="ltx_td ltx_align_left">ARC-e</td>
<td id="S5.T2.3.9.7.2" class="ltx_td ltx_align_center ltx_border_r">25-shot</td>
<td id="S5.T2.3.9.7.3" class="ltx_td ltx_align_center">74.92</td>
<td id="S5.T2.3.9.7.4" class="ltx_td ltx_align_center ltx_border_r">83.84</td>
<td id="S5.T2.3.9.7.5" class="ltx_td ltx_align_center">73.44</td>
<td id="S5.T2.3.9.7.6" class="ltx_td ltx_align_center">72.10</td>
<td id="S5.T2.3.9.7.7" class="ltx_td ltx_align_center"><span id="S5.T2.3.9.7.7.1" class="ltx_text ltx_font_bold">87.29</span></td>
</tr>
<tr id="S5.T2.3.10.8" class="ltx_tr">
<td id="S5.T2.3.10.8.1" class="ltx_td ltx_align_left">BBH</td>
<td id="S5.T2.3.10.8.2" class="ltx_td ltx_align_center ltx_border_r">3-shot CoT</td>
<td id="S5.T2.3.10.8.3" class="ltx_td ltx_align_center">30.39</td>
<td id="S5.T2.3.10.8.4" class="ltx_td ltx_align_center ltx_border_r">38.92</td>
<td id="S5.T2.3.10.8.5" class="ltx_td ltx_align_center">21.03</td>
<td id="S5.T2.3.10.8.6" class="ltx_td ltx_align_center">36.77</td>
<td id="S5.T2.3.10.8.7" class="ltx_td ltx_align_center"><span id="S5.T2.3.10.8.7.1" class="ltx_text ltx_font_bold">71.42</span></td>
</tr>
<tr id="S5.T2.3.11.9" class="ltx_tr">
<td id="S5.T2.3.11.9.1" class="ltx_td ltx_align_left">CommonsenseQA</td>
<td id="S5.T2.3.11.9.2" class="ltx_td ltx_align_center ltx_border_r">3-shot</td>
<td id="S5.T2.3.11.9.3" class="ltx_td ltx_align_center">54.30</td>
<td id="S5.T2.3.11.9.4" class="ltx_td ltx_align_center ltx_border_r"><span id="S5.T2.3.11.9.4.1" class="ltx_text ltx_font_bold">79.52</span></td>
<td id="S5.T2.3.11.9.5" class="ltx_td ltx_align_center">76.09</td>
<td id="S5.T2.3.11.9.6" class="ltx_td ltx_align_center">75.76</td>
<td id="S5.T2.3.11.9.7" class="ltx_td ltx_align_center">77.81</td>
</tr>
<tr id="S5.T2.3.12.10" class="ltx_tr">
<td id="S5.T2.3.12.10.1" class="ltx_td ltx_align_left">CoQA</td>
<td id="S5.T2.3.12.10.2" class="ltx_td ltx_align_center ltx_border_r">0-shot F1</td>
<td id="S5.T2.3.12.10.3" class="ltx_td ltx_align_center">68.30</td>
<td id="S5.T2.3.12.10.4" class="ltx_td ltx_align_center ltx_border_r">77.23</td>
<td id="S5.T2.3.12.10.5" class="ltx_td ltx_align_center">61.94</td>
<td id="S5.T2.3.12.10.6" class="ltx_td ltx_align_center">70.86</td>
<td id="S5.T2.3.12.10.7" class="ltx_td ltx_align_center"><span id="S5.T2.3.12.10.7.1" class="ltx_text ltx_font_bold">79.75</span></td>
</tr>
<tr id="S5.T2.3.13.11" class="ltx_tr">
<td id="S5.T2.3.13.11.1" class="ltx_td ltx_align_left">PIQA</td>
<td id="S5.T2.3.13.11.2" class="ltx_td ltx_align_center ltx_border_r">3-shot</td>
<td id="S5.T2.3.13.11.3" class="ltx_td ltx_align_center">78.67</td>
<td id="S5.T2.3.13.11.4" class="ltx_td ltx_align_center ltx_border_r"><span id="S5.T2.3.13.11.4.1" class="ltx_text ltx_font_bold">82.64</span></td>
<td id="S5.T2.3.13.11.5" class="ltx_td ltx_align_center">76.61</td>
<td id="S5.T2.3.13.11.6" class="ltx_td ltx_align_center">77.42</td>
<td id="S5.T2.3.13.11.7" class="ltx_td ltx_align_center">78.35</td>
</tr>
<tr id="S5.T2.3.14.12" class="ltx_tr">
<td id="S5.T2.3.14.12.1" class="ltx_td ltx_align_left">SciQ</td>
<td id="S5.T2.3.14.12.2" class="ltx_td ltx_align_center ltx_border_r">3-shot</td>
<td id="S5.T2.3.14.12.3" class="ltx_td ltx_align_center">95.70</td>
<td id="S5.T2.3.14.12.4" class="ltx_td ltx_align_center ltx_border_r">97.10</td>
<td id="S5.T2.3.14.12.5" class="ltx_td ltx_align_center">95.40</td>
<td id="S5.T2.3.14.12.6" class="ltx_td ltx_align_center">94.80</td>
<td id="S5.T2.3.14.12.7" class="ltx_td ltx_align_center"><span id="S5.T2.3.14.12.7.1" class="ltx_text ltx_font_bold">97.60</span></td>
</tr>
<tr id="S5.T2.3.15.13" class="ltx_tr">
<td id="S5.T2.3.15.13.1" class="ltx_td ltx_align_left ltx_border_bb ltx_border_t">Average</td>
<td id="S5.T2.3.15.13.2" class="ltx_td ltx_border_bb ltx_border_r ltx_border_t"></td>
<td id="S5.T2.3.15.13.3" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">58.32</td>
<td id="S5.T2.3.15.13.4" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r ltx_border_t">68.98</td>
<td id="S5.T2.3.15.13.5" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">57.07</td>
<td id="S5.T2.3.15.13.6" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">63.21</td>
<td id="S5.T2.3.15.13.7" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t"><span id="S5.T2.3.15.13.7.1" class="ltx_text ltx_font_bold">76.01</span></td>
</tr>
</tbody>
</table>
</figure>
<figure id="S5.T3" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 3: </span><span id="S5.T3.2.1" class="ltx_text ltx_font_bold">Academic benchmarks for smaller models.</span> Academic benchmark results, compared to openly-available models of similar size and trained on general English text data. We compare the instruction fine-tuned models h2oai/h2o-danube3-500m-chat and Qwen/Qwen2-0.5B-Instruct. To evaluate the models, we use the Language Model Evaluation Harness framework <cite class="ltx_cite ltx_citemacro_cite"><a href="#bib.bib6" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">eval-harness</span> </a></cite>.</figcaption>
<table id="S5.T3.3" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S5.T3.3.1.1" class="ltx_tr">
<th id="S5.T3.3.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt">Benchmark</th>
<th id="S5.T3.3.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">Metric</th>
<th id="S5.T3.3.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">Danube3</th>
<th id="S5.T3.3.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">Qwen2</th>
</tr>
<tr id="S5.T3.3.2.2" class="ltx_tr">
<th id="S5.T3.3.2.2.1" class="ltx_td ltx_th ltx_th_row"></th>
<th id="S5.T3.3.2.2.2" class="ltx_td ltx_th ltx_th_column"></th>
<th id="S5.T3.3.2.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_column">0.5B</th>
<th id="S5.T3.3.2.2.4" class="ltx_td ltx_align_center ltx_th ltx_th_column">0.5B</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S5.T3.3.3.1" class="ltx_tr">
<th id="S5.T3.3.3.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">ARC-c</th>
<td id="S5.T3.3.3.1.2" class="ltx_td ltx_align_center ltx_border_t">25-shot</td>
<td id="S5.T3.3.3.1.3" class="ltx_td ltx_align_center ltx_border_t"><span id="S5.T3.3.3.1.3.1" class="ltx_text ltx_font_bold">39.25</span></td>
<td id="S5.T3.3.3.1.4" class="ltx_td ltx_align_center ltx_border_t">32.00</td>
</tr>
<tr id="S5.T3.3.4.2" class="ltx_tr">
<th id="S5.T3.3.4.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Hellaswag</th>
<td id="S5.T3.3.4.2.2" class="ltx_td ltx_align_center">10-shot</td>
<td id="S5.T3.3.4.2.3" class="ltx_td ltx_align_center"><span id="S5.T3.3.4.2.3.1" class="ltx_text ltx_font_bold">61.02</span></td>
<td id="S5.T3.3.4.2.4" class="ltx_td ltx_align_center">49.11</td>
</tr>
<tr id="S5.T3.3.5.3" class="ltx_tr">
<th id="S5.T3.3.5.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">MMLU</th>
<td id="S5.T3.3.5.3.2" class="ltx_td ltx_align_center">5-shot</td>
<td id="S5.T3.3.5.3.3" class="ltx_td ltx_align_center">26.33</td>
<td id="S5.T3.3.5.3.4" class="ltx_td ltx_align_center"><span id="S5.T3.3.5.3.4.1" class="ltx_text ltx_font_bold">43.88</span></td>
</tr>
<tr id="S5.T3.3.6.4" class="ltx_tr">
<th id="S5.T3.3.6.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">TruthfulQA</th>
<td id="S5.T3.3.6.4.2" class="ltx_td ltx_align_center">0-shot mc2</td>
<td id="S5.T3.3.6.4.3" class="ltx_td ltx_align_center"><span id="S5.T3.3.6.4.3.1" class="ltx_text ltx_font_bold">39.96</span></td>
<td id="S5.T3.3.6.4.4" class="ltx_td ltx_align_center">39.28</td>
</tr>
<tr id="S5.T3.3.7.5" class="ltx_tr">
<th id="S5.T3.3.7.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Winogrande</th>
<td id="S5.T3.3.7.5.2" class="ltx_td ltx_align_center">5-shot</td>
<td id="S5.T3.3.7.5.3" class="ltx_td ltx_align_center"><span id="S5.T3.3.7.5.3.1" class="ltx_text ltx_font_bold">61.72</span></td>
<td id="S5.T3.3.7.5.4" class="ltx_td ltx_align_center">56.99</td>
</tr>
<tr id="S5.T3.3.8.6" class="ltx_tr">
<th id="S5.T3.3.8.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">GSM8K</th>
<td id="S5.T3.3.8.6.2" class="ltx_td ltx_align_center">5-shot</td>
<td id="S5.T3.3.8.6.3" class="ltx_td ltx_align_center">16.00</td>
<td id="S5.T3.3.8.6.4" class="ltx_td ltx_align_center"><span id="S5.T3.3.8.6.4.1" class="ltx_text ltx_font_bold">34.12</span></td>
</tr>
<tr id="S5.T3.3.9.7" class="ltx_tr">
<th id="S5.T3.3.9.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">ARC-e</th>
<td id="S5.T3.3.9.7.2" class="ltx_td ltx_align_center">25-shot</td>
<td id="S5.T3.3.9.7.3" class="ltx_td ltx_align_center"><span id="S5.T3.3.9.7.3.1" class="ltx_text ltx_font_bold">71.84</span></td>
<td id="S5.T3.3.9.7.4" class="ltx_td ltx_align_center">62.12</td>
</tr>
<tr id="S5.T3.3.10.8" class="ltx_tr">
<th id="S5.T3.3.10.8.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">BBH</th>
<td id="S5.T3.3.10.8.2" class="ltx_td ltx_align_center">3-shot CoT</td>
<td id="S5.T3.3.10.8.3" class="ltx_td ltx_align_center"><span id="S5.T3.3.10.8.3.1" class="ltx_text ltx_font_bold">25.14</span></td>
<td id="S5.T3.3.10.8.4" class="ltx_td ltx_align_center">18.98</td>
</tr>
<tr id="S5.T3.3.11.9" class="ltx_tr">
<th id="S5.T3.3.11.9.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">CommonsenseQA</th>
<td id="S5.T3.3.11.9.2" class="ltx_td ltx_align_center">3-shot</td>
<td id="S5.T3.3.11.9.3" class="ltx_td ltx_align_center">19.57</td>
<td id="S5.T3.3.11.9.4" class="ltx_td ltx_align_center"><span id="S5.T3.3.11.9.4.1" class="ltx_text ltx_font_bold">52.74</span></td>
</tr>
<tr id="S5.T3.3.12.10" class="ltx_tr">
<th id="S5.T3.3.12.10.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">CoQA</th>
<td id="S5.T3.3.12.10.2" class="ltx_td ltx_align_center">0-shot F1</td>
<td id="S5.T3.3.12.10.3" class="ltx_td ltx_align_center">48.02</td>
<td id="S5.T3.3.12.10.4" class="ltx_td ltx_align_center"><span id="S5.T3.3.12.10.4.1" class="ltx_text ltx_font_bold">54.89</span></td>
</tr>
<tr id="S5.T3.3.13.11" class="ltx_tr">
<th id="S5.T3.3.13.11.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">PIQA</th>
<td id="S5.T3.3.13.11.2" class="ltx_td ltx_align_center">3-shot</td>
<td id="S5.T3.3.13.11.3" class="ltx_td ltx_align_center"><span id="S5.T3.3.13.11.3.1" class="ltx_text ltx_font_bold">74.70</span></td>
<td id="S5.T3.3.13.11.4" class="ltx_td ltx_align_center">68.72</td>
</tr>
<tr id="S5.T3.3.14.12" class="ltx_tr">
<th id="S5.T3.3.14.12.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">SciQ</th>
<td id="S5.T3.3.14.12.2" class="ltx_td ltx_align_center ltx_border_bb">3-shot</td>
<td id="S5.T3.3.14.12.3" class="ltx_td ltx_align_center ltx_border_bb"><span id="S5.T3.3.14.12.3.1" class="ltx_text ltx_font_bold">95.40</span></td>
<td id="S5.T3.3.14.12.4" class="ltx_td ltx_align_center ltx_border_bb">92.60</td>
</tr>
</tbody>
</table>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
<div id="S5.p3" class="ltx_para ltx_noindent">
<p id="S5.p3.1" class="ltx_p"><span id="S5.p3.1.1" class="ltx_text ltx_font_bold">Chat benchmarks.</span>
Evaluating chat and instruct fine-tuned LLMs remains a critical challenge and can most reliably be conducted by large scale human assessment. In order to give an initial evaluation of our chat model, we resort to <em id="S5.p3.1.2" class="ltx_emph ltx_font_italic">MT-Bench</em> <cite class="ltx_cite ltx_citemacro_cite"><a href="#bib.bib16" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">zheng2023judging</span> </a></cite> and <em id="S5.p3.1.3" class="ltx_emph ltx_font_italic">WildBench-v2</em> <cite class="ltx_cite ltx_citemacro_cite"><a href="#bib.bib9" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">lin2024wildbench</span> </a></cite> benchmarks. They represent a collection of multi-turn questions across different categories followed by GPT-4 judgement which assigns a score from 1 to 10 for each model’s response. Results are presented in Table <a href="#S5.T4" title="Table 4 ‣ 5 Evaluation ‣ H2O-Danube3 Technical Report" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a> showing that <span id="S5.p3.1.4" class="ltx_text ltx_font_italic">H2O-Danube3-4B-Chat</span> is surpassing other similar sized models while Phi-3-mini takes the top spot. The 500M parameter version of the model <span id="S5.p3.1.5" class="ltx_text ltx_font_italic">H2O-Danube3-500M-Chat</span> shows results that are comparable to Qwen2-0.5B-Instruct (see Table <a href="#S5.T5" title="Table 5 ‣ 5 Evaluation ‣ H2O-Danube3 Technical Report" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>).</p>
</div>
<div id="S5.p4" class="ltx_para">
<p id="S5.p4.1" class="ltx_p">We additionally conducted multiple internal evaluations and show them in the same tables. First, we performed a blind evaluation of chat performance (excluding the 500M models) following the idea of Chat Arena<span id="footnote4" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">4</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">4</sup><span class="ltx_tag ltx_tag_note">4</span><a target="_blank" href="https://chat.lmsys.org/" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://chat.lmsys.org/</a></span></span></span>. This involved presenting users with random pairs of models and allowing them to prompt and vote on output preference (A better, B better, both bad, both good), followed by calculating an ELO score using MLE and bootstrapping. Second, we utilized an internal RAG (Retrieval-Augmented Generation) benchmark<span id="footnote5" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">5</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">5</sup><span class="ltx_tag ltx_tag_note">5</span><a target="_blank" href="https://github.com/h2oai/enterprise-h2ogpte/tree/main/rag_benchmark" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/h2oai/enterprise-h2ogpte/tree/main/rag_benchmark</a></span></span></span> to assess the models performance in question-answering tasks based on long PDF documents. We calculated an accuracy score for each model by comparing its generated responses to the ground truth answers.</p>
</div>
<figure id="S5.T4" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 4: </span><span id="S5.T4.3.1" class="ltx_text ltx_font_bold">Chat benchmarks.</span> <span id="S5.T4.4.2" class="ltx_text ltx_font_italic">H2O-Danube3-4B-Chat</span> consistently performs very well across all benchmarks, surpassing other similar sized models and outperforming our previous Danube2 release, while Phi-3-mini takes the top spot.</figcaption>
<table id="S5.T4.5" class="ltx_tabular ltx_centering ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="S5.T4.5.1.1" class="ltx_tr">
<td id="S5.T4.5.1.1.1" class="ltx_td ltx_align_left ltx_border_tt">Benchmark</td>
<td id="S5.T4.5.1.1.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt">Metric</td>
<td id="S5.T4.5.1.1.3" class="ltx_td ltx_align_center ltx_border_tt">Danube2</td>
<td id="S5.T4.5.1.1.4" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt">Danube3</td>
<td id="S5.T4.5.1.1.5" class="ltx_td ltx_align_center ltx_border_tt">Qwen1.5</td>
<td id="S5.T4.5.1.1.6" class="ltx_td ltx_align_center ltx_border_tt">StableLM</td>
<td id="S5.T4.5.1.1.7" class="ltx_td ltx_align_center ltx_border_tt">Phi3</td>
</tr>
<tr id="S5.T4.5.2.2" class="ltx_tr">
<td id="S5.T4.5.2.2.1" class="ltx_td"></td>
<td id="S5.T4.5.2.2.2" class="ltx_td ltx_border_r"></td>
<td id="S5.T4.5.2.2.3" class="ltx_td ltx_align_center">1.8B</td>
<td id="S5.T4.5.2.2.4" class="ltx_td ltx_align_center ltx_border_r">4B</td>
<td id="S5.T4.5.2.2.5" class="ltx_td ltx_align_center">4B</td>
<td id="S5.T4.5.2.2.6" class="ltx_td ltx_align_center">3B</td>
<td id="S5.T4.5.2.2.7" class="ltx_td ltx_align_center">4B</td>
</tr>
<tr id="S5.T4.5.3.3" class="ltx_tr">
<td id="S5.T4.5.3.3.1" class="ltx_td ltx_align_left ltx_border_t">MT-Bench</td>
<td id="S5.T4.5.3.3.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">Turn 1</td>
<td id="S5.T4.5.3.3.3" class="ltx_td ltx_align_center ltx_border_t">6.41</td>
<td id="S5.T4.5.3.3.4" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">7.28</td>
<td id="S5.T4.5.3.3.5" class="ltx_td ltx_align_center ltx_border_t">6.68</td>
<td id="S5.T4.5.3.3.6" class="ltx_td ltx_align_center ltx_border_t">7.10</td>
<td id="S5.T4.5.3.3.7" class="ltx_td ltx_align_center ltx_border_t"><span id="S5.T4.5.3.3.7.1" class="ltx_text ltx_font_bold">8.38</span></td>
</tr>
<tr id="S5.T4.5.4.4" class="ltx_tr">
<td id="S5.T4.5.4.4.1" class="ltx_td ltx_align_left">MT-Bench</td>
<td id="S5.T4.5.4.4.2" class="ltx_td ltx_align_center ltx_border_r">Turn 2</td>
<td id="S5.T4.5.4.4.3" class="ltx_td ltx_align_center">4.88</td>
<td id="S5.T4.5.4.4.4" class="ltx_td ltx_align_center ltx_border_r">5.69</td>
<td id="S5.T4.5.4.4.5" class="ltx_td ltx_align_center">5.33</td>
<td id="S5.T4.5.4.4.6" class="ltx_td ltx_align_center">5.74</td>
<td id="S5.T4.5.4.4.7" class="ltx_td ltx_align_center"><span id="S5.T4.5.4.4.7.1" class="ltx_text ltx_font_bold">7.58</span></td>
</tr>
<tr id="S5.T4.5.5.5" class="ltx_tr">
<td id="S5.T4.5.5.5.1" class="ltx_td ltx_align_left">MT-Bench</td>
<td id="S5.T4.5.5.5.2" class="ltx_td ltx_align_center ltx_border_r">Average</td>
<td id="S5.T4.5.5.5.3" class="ltx_td ltx_align_center">5.64</td>
<td id="S5.T4.5.5.5.4" class="ltx_td ltx_align_center ltx_border_r">6.49</td>
<td id="S5.T4.5.5.5.5" class="ltx_td ltx_align_center">6.00</td>
<td id="S5.T4.5.5.5.6" class="ltx_td ltx_align_center">6.42</td>
<td id="S5.T4.5.5.5.7" class="ltx_td ltx_align_center"><span id="S5.T4.5.5.5.7.1" class="ltx_text ltx_font_bold">7.98</span></td>
</tr>
<tr id="S5.T4.5.6.6" class="ltx_tr">
<td id="S5.T4.5.6.6.1" class="ltx_td ltx_align_left ltx_border_t">WildBench-v2</td>
<td id="S5.T4.5.6.6.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">Raw score</td>
<td id="S5.T4.5.6.6.3" class="ltx_td ltx_align_center ltx_border_t">4.65</td>
<td id="S5.T4.5.6.6.4" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">5.54</td>
<td id="S5.T4.5.6.6.5" class="ltx_td ltx_align_center ltx_border_t">4.87</td>
<td id="S5.T4.5.6.6.6" class="ltx_td ltx_align_center ltx_border_t">5.51</td>
<td id="S5.T4.5.6.6.7" class="ltx_td ltx_align_center ltx_border_t"><span id="S5.T4.5.6.6.7.1" class="ltx_text ltx_font_bold">6.47</span></td>
</tr>
<tr id="S5.T4.5.7.7" class="ltx_tr">
<td id="S5.T4.5.7.7.1" class="ltx_td ltx_align_left ltx_border_t">Internal Voting</td>
<td id="S5.T4.5.7.7.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">ELO</td>
<td id="S5.T4.5.7.7.3" class="ltx_td ltx_border_t"></td>
<td id="S5.T4.5.7.7.4" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">1531</td>
<td id="S5.T4.5.7.7.5" class="ltx_td ltx_align_center ltx_border_t">1466</td>
<td id="S5.T4.5.7.7.6" class="ltx_td ltx_align_center ltx_border_t">1435</td>
<td id="S5.T4.5.7.7.7" class="ltx_td ltx_align_center ltx_border_t"><span id="S5.T4.5.7.7.7.1" class="ltx_text ltx_font_bold">1564</span></td>
</tr>
<tr id="S5.T4.5.8.8" class="ltx_tr">
<td id="S5.T4.5.8.8.1" class="ltx_td ltx_align_left ltx_border_bb ltx_border_t">RAG Benchmark</td>
<td id="S5.T4.5.8.8.2" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r ltx_border_t">Accuracy</td>
<td id="S5.T4.5.8.8.3" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">66.88</td>
<td id="S5.T4.5.8.8.4" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r ltx_border_t"><span id="S5.T4.5.8.8.4.1" class="ltx_text ltx_font_bold">73.37</span></td>
<td id="S5.T4.5.8.8.5" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">67.53</td>
<td id="S5.T4.5.8.8.6" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">68.18</td>
<td id="S5.T4.5.8.8.7" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t"><span id="S5.T4.5.8.8.7.1" class="ltx_text ltx_font_bold">73.37</span></td>
</tr>
</tbody>
</table>
</figure>
<figure id="S5.T5" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 5: </span><span id="S5.T5.5.1" class="ltx_text ltx_font_bold">Chat benchmarks for smaller models.</span> The 500M parameter version of the model <span id="S5.T5.6.2" class="ltx_text ltx_font_italic">H2O-Danube3-500M-Chat</span> shows results that are comparable to Qwen2-0.5B-Instruct model. In particular, they achieve a close MT-Bench average score (<span id="S5.T5.7.3" class="ltx_text ltx_font_italic">H2O-Danube3-500M-Chat</span> being better in the 1st turn), while <span id="S5.T5.8.4" class="ltx_text ltx_font_italic">H2O-Danube3-500M-Chat</span> produces better results on Wild-Bench benchmark.</figcaption>
<table id="S5.T5.9" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S5.T5.9.1.1" class="ltx_tr">
<th id="S5.T5.9.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt">Benchmark</th>
<th id="S5.T5.9.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_r ltx_border_tt">Metric</th>
<th id="S5.T5.9.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">Danube3</th>
<th id="S5.T5.9.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">Qwen2</th>
</tr>
<tr id="S5.T5.9.2.2" class="ltx_tr">
<th id="S5.T5.9.2.2.1" class="ltx_td ltx_th ltx_th_column ltx_th_row"></th>
<th id="S5.T5.9.2.2.2" class="ltx_td ltx_th ltx_th_column ltx_th_row ltx_border_r"></th>
<th id="S5.T5.9.2.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_column">0.5B</th>
<th id="S5.T5.9.2.2.4" class="ltx_td ltx_align_center ltx_th ltx_th_column">0.5B</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S5.T5.9.3.1" class="ltx_tr">
<th id="S5.T5.9.3.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">MT-Bench</th>
<th id="S5.T5.9.3.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r ltx_border_t">Turn 1</th>
<td id="S5.T5.9.3.1.3" class="ltx_td ltx_align_center ltx_border_t"><span id="S5.T5.9.3.1.3.1" class="ltx_text ltx_font_bold">4.16</span></td>
<td id="S5.T5.9.3.1.4" class="ltx_td ltx_align_center ltx_border_t">3.78</td>
</tr>
<tr id="S5.T5.9.4.2" class="ltx_tr">
<th id="S5.T5.9.4.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">MT-Bench</th>
<th id="S5.T5.9.4.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">Turn 2</th>
<td id="S5.T5.9.4.2.3" class="ltx_td ltx_align_center">2.40</td>
<td id="S5.T5.9.4.2.4" class="ltx_td ltx_align_center"><span id="S5.T5.9.4.2.4.1" class="ltx_text ltx_font_bold">2.76</span></td>
</tr>
<tr id="S5.T5.9.5.3" class="ltx_tr">
<th id="S5.T5.9.5.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">MT-Bench</th>
<th id="S5.T5.9.5.3.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">Average</th>
<td id="S5.T5.9.5.3.3" class="ltx_td ltx_align_center"><span id="S5.T5.9.5.3.3.1" class="ltx_text ltx_font_bold">3.28</span></td>
<td id="S5.T5.9.5.3.4" class="ltx_td ltx_align_center">3.27</td>
</tr>
<tr id="S5.T5.9.6.4" class="ltx_tr">
<th id="S5.T5.9.6.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">WildBench-v2</th>
<th id="S5.T5.9.6.4.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r ltx_border_t">Raw score</th>
<td id="S5.T5.9.6.4.3" class="ltx_td ltx_align_center ltx_border_t"><span id="S5.T5.9.6.4.3.1" class="ltx_text ltx_font_bold">3.36</span></td>
<td id="S5.T5.9.6.4.4" class="ltx_td ltx_align_center ltx_border_t">3.11</td>
</tr>
<tr id="S5.T5.9.7.5" class="ltx_tr">
<th id="S5.T5.9.7.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb ltx_border_t">Internal RAG Benchmark</th>
<th id="S5.T5.9.7.5.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_bb ltx_border_r ltx_border_t">Accuracy</th>
<td id="S5.T5.9.7.5.3" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">44.16</td>
<td id="S5.T5.9.7.5.4" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t"><span id="S5.T5.9.7.5.4.1" class="ltx_text ltx_font_bold">50.00</span></td>
</tr>
</tbody>
</table>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
<div id="S5.p5" class="ltx_para ltx_noindent">
<p id="S5.p5.1" class="ltx_p"><span id="S5.p5.1.1" class="ltx_text ltx_font_bold">Fine-tuning benchmarks.</span> A common application of small language models is their fine-tuning for various use cases to optimize performance on specific tasks. To that end, we also evaluate the different models’ capability to be easily adaptable to new tasks, here focusing on text classification observed frequently across various use cases in businesses and applications.</p>
</div>
<div id="S5.p6" class="ltx_para">
<p id="S5.p6.7" class="ltx_p">We employ the following process for our fine-tuning benchmarks. We utilize <em id="S5.p6.7.1" class="ltx_emph ltx_font_italic">H2O LLM Studio</em> offering out-of-the-box for tuning language models for classification feeding the final token logit distribution in a custom head for classification. For all models and datasets, we use the same settings with LoRA (<math id="S5.p6.1.m1.1" class="ltx_Math" alttext="r=16" display="inline"><semantics id="S5.p6.1.m1.1a"><mrow id="S5.p6.1.m1.1.1" xref="S5.p6.1.m1.1.1.cmml"><mi id="S5.p6.1.m1.1.1.2" xref="S5.p6.1.m1.1.1.2.cmml">r</mi><mo id="S5.p6.1.m1.1.1.1" xref="S5.p6.1.m1.1.1.1.cmml">=</mo><mn id="S5.p6.1.m1.1.1.3" xref="S5.p6.1.m1.1.1.3.cmml">16</mn></mrow><annotation-xml encoding="MathML-Content" id="S5.p6.1.m1.1b"><apply id="S5.p6.1.m1.1.1.cmml" xref="S5.p6.1.m1.1.1"><eq id="S5.p6.1.m1.1.1.1.cmml" xref="S5.p6.1.m1.1.1.1"></eq><ci id="S5.p6.1.m1.1.1.2.cmml" xref="S5.p6.1.m1.1.1.2">𝑟</ci><cn type="integer" id="S5.p6.1.m1.1.1.3.cmml" xref="S5.p6.1.m1.1.1.3">16</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.p6.1.m1.1c">r=16</annotation></semantics></math>, <math id="S5.p6.2.m2.1" class="ltx_Math" alttext="\alpha=32" display="inline"><semantics id="S5.p6.2.m2.1a"><mrow id="S5.p6.2.m2.1.1" xref="S5.p6.2.m2.1.1.cmml"><mi id="S5.p6.2.m2.1.1.2" xref="S5.p6.2.m2.1.1.2.cmml">α</mi><mo id="S5.p6.2.m2.1.1.1" xref="S5.p6.2.m2.1.1.1.cmml">=</mo><mn id="S5.p6.2.m2.1.1.3" xref="S5.p6.2.m2.1.1.3.cmml">32</mn></mrow><annotation-xml encoding="MathML-Content" id="S5.p6.2.m2.1b"><apply id="S5.p6.2.m2.1.1.cmml" xref="S5.p6.2.m2.1.1"><eq id="S5.p6.2.m2.1.1.1.cmml" xref="S5.p6.2.m2.1.1.1"></eq><ci id="S5.p6.2.m2.1.1.2.cmml" xref="S5.p6.2.m2.1.1.2">𝛼</ci><cn type="integer" id="S5.p6.2.m2.1.1.3.cmml" xref="S5.p6.2.m2.1.1.3">32</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.p6.2.m2.1c">\alpha=32</annotation></semantics></math>) and same hyperparameters (<math id="S5.p6.3.m3.1" class="ltx_Math" alttext="bs=1" display="inline"><semantics id="S5.p6.3.m3.1a"><mrow id="S5.p6.3.m3.1.1" xref="S5.p6.3.m3.1.1.cmml"><mrow id="S5.p6.3.m3.1.1.2" xref="S5.p6.3.m3.1.1.2.cmml"><mi id="S5.p6.3.m3.1.1.2.2" xref="S5.p6.3.m3.1.1.2.2.cmml">b</mi><mo lspace="0em" rspace="0em" id="S5.p6.3.m3.1.1.2.1" xref="S5.p6.3.m3.1.1.2.1.cmml">​</mo><mi id="S5.p6.3.m3.1.1.2.3" xref="S5.p6.3.m3.1.1.2.3.cmml">s</mi></mrow><mo id="S5.p6.3.m3.1.1.1" xref="S5.p6.3.m3.1.1.1.cmml">=</mo><mn id="S5.p6.3.m3.1.1.3" xref="S5.p6.3.m3.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="S5.p6.3.m3.1b"><apply id="S5.p6.3.m3.1.1.cmml" xref="S5.p6.3.m3.1.1"><eq id="S5.p6.3.m3.1.1.1.cmml" xref="S5.p6.3.m3.1.1.1"></eq><apply id="S5.p6.3.m3.1.1.2.cmml" xref="S5.p6.3.m3.1.1.2"><times id="S5.p6.3.m3.1.1.2.1.cmml" xref="S5.p6.3.m3.1.1.2.1"></times><ci id="S5.p6.3.m3.1.1.2.2.cmml" xref="S5.p6.3.m3.1.1.2.2">𝑏</ci><ci id="S5.p6.3.m3.1.1.2.3.cmml" xref="S5.p6.3.m3.1.1.2.3">𝑠</ci></apply><cn type="integer" id="S5.p6.3.m3.1.1.3.cmml" xref="S5.p6.3.m3.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.p6.3.m3.1c">bs=1</annotation></semantics></math>, <math id="S5.p6.4.m4.1" class="ltx_Math" alttext="epochs=1" display="inline"><semantics id="S5.p6.4.m4.1a"><mrow id="S5.p6.4.m4.1.1" xref="S5.p6.4.m4.1.1.cmml"><mrow id="S5.p6.4.m4.1.1.2" xref="S5.p6.4.m4.1.1.2.cmml"><mi id="S5.p6.4.m4.1.1.2.2" xref="S5.p6.4.m4.1.1.2.2.cmml">e</mi><mo lspace="0em" rspace="0em" id="S5.p6.4.m4.1.1.2.1" xref="S5.p6.4.m4.1.1.2.1.cmml">​</mo><mi id="S5.p6.4.m4.1.1.2.3" xref="S5.p6.4.m4.1.1.2.3.cmml">p</mi><mo lspace="0em" rspace="0em" id="S5.p6.4.m4.1.1.2.1a" xref="S5.p6.4.m4.1.1.2.1.cmml">​</mo><mi id="S5.p6.4.m4.1.1.2.4" xref="S5.p6.4.m4.1.1.2.4.cmml">o</mi><mo lspace="0em" rspace="0em" id="S5.p6.4.m4.1.1.2.1b" xref="S5.p6.4.m4.1.1.2.1.cmml">​</mo><mi id="S5.p6.4.m4.1.1.2.5" xref="S5.p6.4.m4.1.1.2.5.cmml">c</mi><mo lspace="0em" rspace="0em" id="S5.p6.4.m4.1.1.2.1c" xref="S5.p6.4.m4.1.1.2.1.cmml">​</mo><mi id="S5.p6.4.m4.1.1.2.6" xref="S5.p6.4.m4.1.1.2.6.cmml">h</mi><mo lspace="0em" rspace="0em" id="S5.p6.4.m4.1.1.2.1d" xref="S5.p6.4.m4.1.1.2.1.cmml">​</mo><mi id="S5.p6.4.m4.1.1.2.7" xref="S5.p6.4.m4.1.1.2.7.cmml">s</mi></mrow><mo id="S5.p6.4.m4.1.1.1" xref="S5.p6.4.m4.1.1.1.cmml">=</mo><mn id="S5.p6.4.m4.1.1.3" xref="S5.p6.4.m4.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="S5.p6.4.m4.1b"><apply id="S5.p6.4.m4.1.1.cmml" xref="S5.p6.4.m4.1.1"><eq id="S5.p6.4.m4.1.1.1.cmml" xref="S5.p6.4.m4.1.1.1"></eq><apply id="S5.p6.4.m4.1.1.2.cmml" xref="S5.p6.4.m4.1.1.2"><times id="S5.p6.4.m4.1.1.2.1.cmml" xref="S5.p6.4.m4.1.1.2.1"></times><ci id="S5.p6.4.m4.1.1.2.2.cmml" xref="S5.p6.4.m4.1.1.2.2">𝑒</ci><ci id="S5.p6.4.m4.1.1.2.3.cmml" xref="S5.p6.4.m4.1.1.2.3">𝑝</ci><ci id="S5.p6.4.m4.1.1.2.4.cmml" xref="S5.p6.4.m4.1.1.2.4">𝑜</ci><ci id="S5.p6.4.m4.1.1.2.5.cmml" xref="S5.p6.4.m4.1.1.2.5">𝑐</ci><ci id="S5.p6.4.m4.1.1.2.6.cmml" xref="S5.p6.4.m4.1.1.2.6">ℎ</ci><ci id="S5.p6.4.m4.1.1.2.7.cmml" xref="S5.p6.4.m4.1.1.2.7">𝑠</ci></apply><cn type="integer" id="S5.p6.4.m4.1.1.3.cmml" xref="S5.p6.4.m4.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.p6.4.m4.1c">epochs=1</annotation></semantics></math>, <math id="S5.p6.5.m5.1" class="ltx_Math" alttext="lr=1e-4" display="inline"><semantics id="S5.p6.5.m5.1a"><mrow id="S5.p6.5.m5.1.1" xref="S5.p6.5.m5.1.1.cmml"><mrow id="S5.p6.5.m5.1.1.2" xref="S5.p6.5.m5.1.1.2.cmml"><mi id="S5.p6.5.m5.1.1.2.2" xref="S5.p6.5.m5.1.1.2.2.cmml">l</mi><mo lspace="0em" rspace="0em" id="S5.p6.5.m5.1.1.2.1" xref="S5.p6.5.m5.1.1.2.1.cmml">​</mo><mi id="S5.p6.5.m5.1.1.2.3" xref="S5.p6.5.m5.1.1.2.3.cmml">r</mi></mrow><mo id="S5.p6.5.m5.1.1.1" xref="S5.p6.5.m5.1.1.1.cmml">=</mo><mrow id="S5.p6.5.m5.1.1.3" xref="S5.p6.5.m5.1.1.3.cmml"><mrow id="S5.p6.5.m5.1.1.3.2" xref="S5.p6.5.m5.1.1.3.2.cmml"><mn id="S5.p6.5.m5.1.1.3.2.2" xref="S5.p6.5.m5.1.1.3.2.2.cmml">1</mn><mo lspace="0em" rspace="0em" id="S5.p6.5.m5.1.1.3.2.1" xref="S5.p6.5.m5.1.1.3.2.1.cmml">​</mo><mi id="S5.p6.5.m5.1.1.3.2.3" xref="S5.p6.5.m5.1.1.3.2.3.cmml">e</mi></mrow><mo id="S5.p6.5.m5.1.1.3.1" xref="S5.p6.5.m5.1.1.3.1.cmml">−</mo><mn id="S5.p6.5.m5.1.1.3.3" xref="S5.p6.5.m5.1.1.3.3.cmml">4</mn></mrow></mrow><annotation-xml encoding="MathML-Content" id="S5.p6.5.m5.1b"><apply id="S5.p6.5.m5.1.1.cmml" xref="S5.p6.5.m5.1.1"><eq id="S5.p6.5.m5.1.1.1.cmml" xref="S5.p6.5.m5.1.1.1"></eq><apply id="S5.p6.5.m5.1.1.2.cmml" xref="S5.p6.5.m5.1.1.2"><times id="S5.p6.5.m5.1.1.2.1.cmml" xref="S5.p6.5.m5.1.1.2.1"></times><ci id="S5.p6.5.m5.1.1.2.2.cmml" xref="S5.p6.5.m5.1.1.2.2">𝑙</ci><ci id="S5.p6.5.m5.1.1.2.3.cmml" xref="S5.p6.5.m5.1.1.2.3">𝑟</ci></apply><apply id="S5.p6.5.m5.1.1.3.cmml" xref="S5.p6.5.m5.1.1.3"><minus id="S5.p6.5.m5.1.1.3.1.cmml" xref="S5.p6.5.m5.1.1.3.1"></minus><apply id="S5.p6.5.m5.1.1.3.2.cmml" xref="S5.p6.5.m5.1.1.3.2"><times id="S5.p6.5.m5.1.1.3.2.1.cmml" xref="S5.p6.5.m5.1.1.3.2.1"></times><cn type="integer" id="S5.p6.5.m5.1.1.3.2.2.cmml" xref="S5.p6.5.m5.1.1.3.2.2">1</cn><ci id="S5.p6.5.m5.1.1.3.2.3.cmml" xref="S5.p6.5.m5.1.1.3.2.3">𝑒</ci></apply><cn type="integer" id="S5.p6.5.m5.1.1.3.3.cmml" xref="S5.p6.5.m5.1.1.3.3">4</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.p6.5.m5.1c">lr=1e-4</annotation></semantics></math>, <math id="S5.p6.6.m6.1" class="ltx_Math" alttext="diff\_lr=1e-05" display="inline"><semantics id="S5.p6.6.m6.1a"><mrow id="S5.p6.6.m6.1.1" xref="S5.p6.6.m6.1.1.cmml"><mrow id="S5.p6.6.m6.1.1.2" xref="S5.p6.6.m6.1.1.2.cmml"><mi id="S5.p6.6.m6.1.1.2.2" xref="S5.p6.6.m6.1.1.2.2.cmml">d</mi><mo lspace="0em" rspace="0em" id="S5.p6.6.m6.1.1.2.1" xref="S5.p6.6.m6.1.1.2.1.cmml">​</mo><mi id="S5.p6.6.m6.1.1.2.3" xref="S5.p6.6.m6.1.1.2.3.cmml">i</mi><mo lspace="0em" rspace="0em" id="S5.p6.6.m6.1.1.2.1a" xref="S5.p6.6.m6.1.1.2.1.cmml">​</mo><mi id="S5.p6.6.m6.1.1.2.4" xref="S5.p6.6.m6.1.1.2.4.cmml">f</mi><mo lspace="0em" rspace="0em" id="S5.p6.6.m6.1.1.2.1b" xref="S5.p6.6.m6.1.1.2.1.cmml">​</mo><mi id="S5.p6.6.m6.1.1.2.5" xref="S5.p6.6.m6.1.1.2.5.cmml">f</mi><mo lspace="0em" rspace="0em" id="S5.p6.6.m6.1.1.2.1c" xref="S5.p6.6.m6.1.1.2.1.cmml">​</mo><mi mathvariant="normal" id="S5.p6.6.m6.1.1.2.6" xref="S5.p6.6.m6.1.1.2.6.cmml">_</mi><mo lspace="0em" rspace="0em" id="S5.p6.6.m6.1.1.2.1d" xref="S5.p6.6.m6.1.1.2.1.cmml">​</mo><mi id="S5.p6.6.m6.1.1.2.7" xref="S5.p6.6.m6.1.1.2.7.cmml">l</mi><mo lspace="0em" rspace="0em" id="S5.p6.6.m6.1.1.2.1e" xref="S5.p6.6.m6.1.1.2.1.cmml">​</mo><mi id="S5.p6.6.m6.1.1.2.8" xref="S5.p6.6.m6.1.1.2.8.cmml">r</mi></mrow><mo id="S5.p6.6.m6.1.1.1" xref="S5.p6.6.m6.1.1.1.cmml">=</mo><mrow id="S5.p6.6.m6.1.1.3" xref="S5.p6.6.m6.1.1.3.cmml"><mrow id="S5.p6.6.m6.1.1.3.2" xref="S5.p6.6.m6.1.1.3.2.cmml"><mn id="S5.p6.6.m6.1.1.3.2.2" xref="S5.p6.6.m6.1.1.3.2.2.cmml">1</mn><mo lspace="0em" rspace="0em" id="S5.p6.6.m6.1.1.3.2.1" xref="S5.p6.6.m6.1.1.3.2.1.cmml">​</mo><mi id="S5.p6.6.m6.1.1.3.2.3" xref="S5.p6.6.m6.1.1.3.2.3.cmml">e</mi></mrow><mo id="S5.p6.6.m6.1.1.3.1" xref="S5.p6.6.m6.1.1.3.1.cmml">−</mo><mn id="S5.p6.6.m6.1.1.3.3" xref="S5.p6.6.m6.1.1.3.3.cmml">05</mn></mrow></mrow><annotation-xml encoding="MathML-Content" id="S5.p6.6.m6.1b"><apply id="S5.p6.6.m6.1.1.cmml" xref="S5.p6.6.m6.1.1"><eq id="S5.p6.6.m6.1.1.1.cmml" xref="S5.p6.6.m6.1.1.1"></eq><apply id="S5.p6.6.m6.1.1.2.cmml" xref="S5.p6.6.m6.1.1.2"><times id="S5.p6.6.m6.1.1.2.1.cmml" xref="S5.p6.6.m6.1.1.2.1"></times><ci id="S5.p6.6.m6.1.1.2.2.cmml" xref="S5.p6.6.m6.1.1.2.2">𝑑</ci><ci id="S5.p6.6.m6.1.1.2.3.cmml" xref="S5.p6.6.m6.1.1.2.3">𝑖</ci><ci id="S5.p6.6.m6.1.1.2.4.cmml" xref="S5.p6.6.m6.1.1.2.4">𝑓</ci><ci id="S5.p6.6.m6.1.1.2.5.cmml" xref="S5.p6.6.m6.1.1.2.5">𝑓</ci><ci id="S5.p6.6.m6.1.1.2.6.cmml" xref="S5.p6.6.m6.1.1.2.6">_</ci><ci id="S5.p6.6.m6.1.1.2.7.cmml" xref="S5.p6.6.m6.1.1.2.7">𝑙</ci><ci id="S5.p6.6.m6.1.1.2.8.cmml" xref="S5.p6.6.m6.1.1.2.8">𝑟</ci></apply><apply id="S5.p6.6.m6.1.1.3.cmml" xref="S5.p6.6.m6.1.1.3"><minus id="S5.p6.6.m6.1.1.3.1.cmml" xref="S5.p6.6.m6.1.1.3.1"></minus><apply id="S5.p6.6.m6.1.1.3.2.cmml" xref="S5.p6.6.m6.1.1.3.2"><times id="S5.p6.6.m6.1.1.3.2.1.cmml" xref="S5.p6.6.m6.1.1.3.2.1"></times><cn type="integer" id="S5.p6.6.m6.1.1.3.2.2.cmml" xref="S5.p6.6.m6.1.1.3.2.2">1</cn><ci id="S5.p6.6.m6.1.1.3.2.3.cmml" xref="S5.p6.6.m6.1.1.3.2.3">𝑒</ci></apply><cn type="integer" id="S5.p6.6.m6.1.1.3.3.cmml" xref="S5.p6.6.m6.1.1.3.3">05</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.p6.6.m6.1c">diff\_lr=1e-05</annotation></semantics></math>, <math id="S5.p6.7.m7.1" class="ltx_Math" alttext="max\_length=8192" display="inline"><semantics id="S5.p6.7.m7.1a"><mrow id="S5.p6.7.m7.1.1" xref="S5.p6.7.m7.1.1.cmml"><mrow id="S5.p6.7.m7.1.1.2" xref="S5.p6.7.m7.1.1.2.cmml"><mi id="S5.p6.7.m7.1.1.2.2" xref="S5.p6.7.m7.1.1.2.2.cmml">m</mi><mo lspace="0em" rspace="0em" id="S5.p6.7.m7.1.1.2.1" xref="S5.p6.7.m7.1.1.2.1.cmml">​</mo><mi id="S5.p6.7.m7.1.1.2.3" xref="S5.p6.7.m7.1.1.2.3.cmml">a</mi><mo lspace="0em" rspace="0em" id="S5.p6.7.m7.1.1.2.1a" xref="S5.p6.7.m7.1.1.2.1.cmml">​</mo><mi id="S5.p6.7.m7.1.1.2.4" xref="S5.p6.7.m7.1.1.2.4.cmml">x</mi><mo lspace="0em" rspace="0em" id="S5.p6.7.m7.1.1.2.1b" xref="S5.p6.7.m7.1.1.2.1.cmml">​</mo><mi mathvariant="normal" id="S5.p6.7.m7.1.1.2.5" xref="S5.p6.7.m7.1.1.2.5.cmml">_</mi><mo lspace="0em" rspace="0em" id="S5.p6.7.m7.1.1.2.1c" xref="S5.p6.7.m7.1.1.2.1.cmml">​</mo><mi id="S5.p6.7.m7.1.1.2.6" xref="S5.p6.7.m7.1.1.2.6.cmml">l</mi><mo lspace="0em" rspace="0em" id="S5.p6.7.m7.1.1.2.1d" xref="S5.p6.7.m7.1.1.2.1.cmml">​</mo><mi id="S5.p6.7.m7.1.1.2.7" xref="S5.p6.7.m7.1.1.2.7.cmml">e</mi><mo lspace="0em" rspace="0em" id="S5.p6.7.m7.1.1.2.1e" xref="S5.p6.7.m7.1.1.2.1.cmml">​</mo><mi id="S5.p6.7.m7.1.1.2.8" xref="S5.p6.7.m7.1.1.2.8.cmml">n</mi><mo lspace="0em" rspace="0em" id="S5.p6.7.m7.1.1.2.1f" xref="S5.p6.7.m7.1.1.2.1.cmml">​</mo><mi id="S5.p6.7.m7.1.1.2.9" xref="S5.p6.7.m7.1.1.2.9.cmml">g</mi><mo lspace="0em" rspace="0em" id="S5.p6.7.m7.1.1.2.1g" xref="S5.p6.7.m7.1.1.2.1.cmml">​</mo><mi id="S5.p6.7.m7.1.1.2.10" xref="S5.p6.7.m7.1.1.2.10.cmml">t</mi><mo lspace="0em" rspace="0em" id="S5.p6.7.m7.1.1.2.1h" xref="S5.p6.7.m7.1.1.2.1.cmml">​</mo><mi id="S5.p6.7.m7.1.1.2.11" xref="S5.p6.7.m7.1.1.2.11.cmml">h</mi></mrow><mo id="S5.p6.7.m7.1.1.1" xref="S5.p6.7.m7.1.1.1.cmml">=</mo><mn id="S5.p6.7.m7.1.1.3" xref="S5.p6.7.m7.1.1.3.cmml">8192</mn></mrow><annotation-xml encoding="MathML-Content" id="S5.p6.7.m7.1b"><apply id="S5.p6.7.m7.1.1.cmml" xref="S5.p6.7.m7.1.1"><eq id="S5.p6.7.m7.1.1.1.cmml" xref="S5.p6.7.m7.1.1.1"></eq><apply id="S5.p6.7.m7.1.1.2.cmml" xref="S5.p6.7.m7.1.1.2"><times id="S5.p6.7.m7.1.1.2.1.cmml" xref="S5.p6.7.m7.1.1.2.1"></times><ci id="S5.p6.7.m7.1.1.2.2.cmml" xref="S5.p6.7.m7.1.1.2.2">𝑚</ci><ci id="S5.p6.7.m7.1.1.2.3.cmml" xref="S5.p6.7.m7.1.1.2.3">𝑎</ci><ci id="S5.p6.7.m7.1.1.2.4.cmml" xref="S5.p6.7.m7.1.1.2.4">𝑥</ci><ci id="S5.p6.7.m7.1.1.2.5.cmml" xref="S5.p6.7.m7.1.1.2.5">_</ci><ci id="S5.p6.7.m7.1.1.2.6.cmml" xref="S5.p6.7.m7.1.1.2.6">𝑙</ci><ci id="S5.p6.7.m7.1.1.2.7.cmml" xref="S5.p6.7.m7.1.1.2.7">𝑒</ci><ci id="S5.p6.7.m7.1.1.2.8.cmml" xref="S5.p6.7.m7.1.1.2.8">𝑛</ci><ci id="S5.p6.7.m7.1.1.2.9.cmml" xref="S5.p6.7.m7.1.1.2.9">𝑔</ci><ci id="S5.p6.7.m7.1.1.2.10.cmml" xref="S5.p6.7.m7.1.1.2.10">𝑡</ci><ci id="S5.p6.7.m7.1.1.2.11.cmml" xref="S5.p6.7.m7.1.1.2.11">ℎ</ci></apply><cn type="integer" id="S5.p6.7.m7.1.1.3.cmml" xref="S5.p6.7.m7.1.1.3">8192</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.p6.7.m7.1c">max\_length=8192</annotation></semantics></math>). These settings are commonly used default settings in the field and we aim at particularly evaluating the default performance of models after tuning.
We look at the following datasets, that all can be found on Hugging Face:</p>
</div>
<div id="S5.p7" class="ltx_para">
<ul id="S5.I1" class="ltx_itemize">
<li id="S5.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S5.I1.i1.p1" class="ltx_para">
<p id="S5.I1.i1.p1.1" class="ltx_p">stanfordnlp/imdb: binary sentence classification of imdb movie reviews</p>
</div>
</li>
<li id="S5.I1.i2" class="ltx_item" style="list-style-type:none;padding-top:1.0pt;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S5.I1.i2.p1" class="ltx_para">
<p id="S5.I1.i2.p1.1" class="ltx_p">knowledgator/Scientific-text-classification: classification of scientific texts into 10 most frequent classes, random 50-50 split</p>
</div>
</li>
<li id="S5.I1.i3" class="ltx_item" style="list-style-type:none;padding-top:1.0pt;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S5.I1.i3.p1" class="ltx_para">
<p id="S5.I1.i3.p1.1" class="ltx_p">ccdv/arxiv-classification: long context classification of arxiv papers into 11 classes</p>
</div>
</li>
<li id="S5.I1.i4" class="ltx_item" style="list-style-type:none;padding-top:1.0pt;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S5.I1.i4.p1" class="ltx_para">
<p id="S5.I1.i4.p1.1" class="ltx_p">ccdv/patent-classification: long context classification of patents into 9 classes</p>
</div>
</li>
</ul>
</div>
<div id="S5.p8" class="ltx_para">
<p id="S5.p8.1" class="ltx_p">Table <a href="#S5.T6" title="Table 6 ‣ 5 Evaluation ‣ H2O-Danube3 Technical Report" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a> highlights the results for individual datasets and models, we always report the accuracy taking the highest probability class. We can see, that all small language models show excellent performance on text classification tasks after fine-tuning. Even small 500M parameter models can be highly competitive, exemplifying the utility of fine-tuning such models for specific use cases. Overall, <span id="S5.p8.1.1" class="ltx_text ltx_font_italic">H2O-Danube3-4B</span> takes a leading spot in all benchmarks. These results can be seen as baseline results based on default hyperparameter settings. More extensive parameter sweeps would potentially improve results of all models at hand, and might also alter the order of performance. We plan on investigating such fine-tuning performance more extensively in the future.</p>
</div>
<figure id="S5.T6" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 6: </span><span id="S5.T6.3.1" class="ltx_text ltx_font_bold">Fine-tuning benchmarks.</span> All models show excellent performance on various classification tasks after fine-tuning with <span id="S5.T6.4.2" class="ltx_text ltx_font_italic">H2O-Danube3-4B</span> taking a top spot in most benchmarks.</figcaption>
<table id="S5.T6.5" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S5.T6.5.1.1" class="ltx_tr">
<th id="S5.T6.5.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_r ltx_border_tt">Dataset</th>
<th id="S5.T6.5.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">Danube2</th>
<th id="S5.T6.5.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">Danube3</th>
<th id="S5.T6.5.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_tt">Danube3</th>
<th id="S5.T6.5.1.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">Qwen1.5</th>
<th id="S5.T6.5.1.1.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">Qwen2</th>
<th id="S5.T6.5.1.1.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">StableLM</th>
<th id="S5.T6.5.1.1.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">Phi3</th>
</tr>
<tr id="S5.T6.5.2.2" class="ltx_tr">
<th id="S5.T6.5.2.2.1" class="ltx_td ltx_th ltx_th_column ltx_border_r"></th>
<th id="S5.T6.5.2.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_column">1.8B</th>
<th id="S5.T6.5.2.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_column">0.5B</th>
<th id="S5.T6.5.2.2.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r">4B</th>
<th id="S5.T6.5.2.2.5" class="ltx_td ltx_align_center ltx_th ltx_th_column">4B</th>
<th id="S5.T6.5.2.2.6" class="ltx_td ltx_align_center ltx_th ltx_th_column">0.5B</th>
<th id="S5.T6.5.2.2.7" class="ltx_td ltx_align_center ltx_th ltx_th_column">3B</th>
<th id="S5.T6.5.2.2.8" class="ltx_td ltx_align_center ltx_th ltx_th_column">4B</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S5.T6.5.3.1" class="ltx_tr">
<td id="S5.T6.5.3.1.1" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">Arxiv</td>
<td id="S5.T6.5.3.1.2" class="ltx_td ltx_align_center ltx_border_t">0.864</td>
<td id="S5.T6.5.3.1.3" class="ltx_td ltx_align_center ltx_border_t">0.863</td>
<td id="S5.T6.5.3.1.4" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">0.873</td>
<td id="S5.T6.5.3.1.5" class="ltx_td ltx_align_center ltx_border_t"><span id="S5.T6.5.3.1.5.1" class="ltx_text ltx_font_bold">0.877</span></td>
<td id="S5.T6.5.3.1.6" class="ltx_td ltx_align_center ltx_border_t">0.874</td>
<td id="S5.T6.5.3.1.7" class="ltx_td ltx_align_center ltx_border_t">0.865</td>
<td id="S5.T6.5.3.1.8" class="ltx_td ltx_align_center ltx_border_t">0.869</td>
</tr>
<tr id="S5.T6.5.4.2" class="ltx_tr">
<td id="S5.T6.5.4.2.1" class="ltx_td ltx_align_left ltx_border_r">Imdb</td>
<td id="S5.T6.5.4.2.2" class="ltx_td ltx_align_center">0.968</td>
<td id="S5.T6.5.4.2.3" class="ltx_td ltx_align_center">0.959</td>
<td id="S5.T6.5.4.2.4" class="ltx_td ltx_align_center ltx_border_r"><span id="S5.T6.5.4.2.4.1" class="ltx_text ltx_font_bold">0.971</span></td>
<td id="S5.T6.5.4.2.5" class="ltx_td ltx_align_center">0.970</td>
<td id="S5.T6.5.4.2.6" class="ltx_td ltx_align_center">0.959</td>
<td id="S5.T6.5.4.2.7" class="ltx_td ltx_align_center">0.969</td>
<td id="S5.T6.5.4.2.8" class="ltx_td ltx_align_center">0.967</td>
</tr>
<tr id="S5.T6.5.5.3" class="ltx_tr">
<td id="S5.T6.5.5.3.1" class="ltx_td ltx_align_left ltx_border_r">Patent</td>
<td id="S5.T6.5.5.3.2" class="ltx_td ltx_align_center">0.721</td>
<td id="S5.T6.5.5.3.3" class="ltx_td ltx_align_center">0.708</td>
<td id="S5.T6.5.5.3.4" class="ltx_td ltx_align_center ltx_border_r"><span id="S5.T6.5.5.3.4.1" class="ltx_text ltx_font_bold">0.727</span></td>
<td id="S5.T6.5.5.3.5" class="ltx_td ltx_align_center">0.717</td>
<td id="S5.T6.5.5.3.6" class="ltx_td ltx_align_center">0.707</td>
<td id="S5.T6.5.5.3.7" class="ltx_td ltx_align_center">0.719</td>
<td id="S5.T6.5.5.3.8" class="ltx_td ltx_align_center">0.712</td>
</tr>
<tr id="S5.T6.5.6.4" class="ltx_tr">
<td id="S5.T6.5.6.4.1" class="ltx_td ltx_align_left ltx_border_r">Scientific</td>
<td id="S5.T6.5.6.4.2" class="ltx_td ltx_align_center">0.868</td>
<td id="S5.T6.5.6.4.3" class="ltx_td ltx_align_center">0.846</td>
<td id="S5.T6.5.6.4.4" class="ltx_td ltx_align_center ltx_border_r">0.872</td>
<td id="S5.T6.5.6.4.5" class="ltx_td ltx_align_center"><span id="S5.T6.5.6.4.5.1" class="ltx_text ltx_font_bold">0.875</span></td>
<td id="S5.T6.5.6.4.6" class="ltx_td ltx_align_center">0.855</td>
<td id="S5.T6.5.6.4.7" class="ltx_td ltx_align_center">0.867</td>
<td id="S5.T6.5.6.4.8" class="ltx_td ltx_align_center">0.870</td>
</tr>
<tr id="S5.T6.5.7.5" class="ltx_tr">
<td id="S5.T6.5.7.5.1" class="ltx_td ltx_align_left ltx_border_bb ltx_border_r ltx_border_t">Average</td>
<td id="S5.T6.5.7.5.2" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">0.855</td>
<td id="S5.T6.5.7.5.3" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">0.844</td>
<td id="S5.T6.5.7.5.4" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r ltx_border_t"><span id="S5.T6.5.7.5.4.1" class="ltx_text ltx_font_bold">0.861</span></td>
<td id="S5.T6.5.7.5.5" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">0.86</td>
<td id="S5.T6.5.7.5.6" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">0.849</td>
<td id="S5.T6.5.7.5.7" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">0.855</td>
<td id="S5.T6.5.7.5.8" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">0.854</td>
</tr>
</tbody>
</table>
</figure>
<figure id="S5.T7" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 7: </span><span id="S5.T7.3.1" class="ltx_text ltx_font_bold">Model quantization.</span> This table summarizes different quantized versions of <span id="S5.T7.4.2" class="ltx_text ltx_font_italic">H2O-Danube3-4B-Chat</span> showing the trade-off between size and quality of the models. Results indicate that quantized models reduced to 4 bits exhibit minimal loss in benchmark performance.</figcaption>
<table id="S5.T7.5" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S5.T7.5.1.1" class="ltx_tr">
<th id="S5.T7.5.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt">Quant method</th>
<th id="S5.T7.5.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">Model size</th>
<th id="S5.T7.5.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">MT-Bench</th>
<th id="S5.T7.5.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">Perplexity</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S5.T7.5.2.1" class="ltx_tr">
<th id="S5.T7.5.2.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">F16</th>
<td id="S5.T7.5.2.1.2" class="ltx_td ltx_align_center ltx_border_t">7.92 GB</td>
<td id="S5.T7.5.2.1.3" class="ltx_td ltx_align_center ltx_border_t">6.43</td>
<td id="S5.T7.5.2.1.4" class="ltx_td ltx_align_center ltx_border_t">6.17</td>
</tr>
<tr id="S5.T7.5.3.2" class="ltx_tr">
<th id="S5.T7.5.3.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Q8_0</th>
<td id="S5.T7.5.3.2.2" class="ltx_td ltx_align_center">4.21 GB</td>
<td id="S5.T7.5.3.2.3" class="ltx_td ltx_align_center">6.49</td>
<td id="S5.T7.5.3.2.4" class="ltx_td ltx_align_center">6.17</td>
</tr>
<tr id="S5.T7.5.4.3" class="ltx_tr">
<th id="S5.T7.5.4.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Q6_K</th>
<td id="S5.T7.5.4.3.2" class="ltx_td ltx_align_center">3.25 GB</td>
<td id="S5.T7.5.4.3.3" class="ltx_td ltx_align_center">6.37</td>
<td id="S5.T7.5.4.3.4" class="ltx_td ltx_align_center">6.20</td>
</tr>
<tr id="S5.T7.5.5.4" class="ltx_tr">
<th id="S5.T7.5.5.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Q5_K_M</th>
<td id="S5.T7.5.5.4.2" class="ltx_td ltx_align_center">2.81 GB</td>
<td id="S5.T7.5.5.4.3" class="ltx_td ltx_align_center">6.25</td>
<td id="S5.T7.5.5.4.4" class="ltx_td ltx_align_center">6.24</td>
</tr>
<tr id="S5.T7.5.6.5" class="ltx_tr">
<th id="S5.T7.5.6.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Q4_K_M</th>
<td id="S5.T7.5.6.5.2" class="ltx_td ltx_align_center">2.39 GB</td>
<td id="S5.T7.5.6.5.3" class="ltx_td ltx_align_center">6.31</td>
<td id="S5.T7.5.6.5.4" class="ltx_td ltx_align_center">6.37</td>
</tr>
<tr id="S5.T7.5.7.6" class="ltx_tr">
<th id="S5.T7.5.7.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Q3_K_M</th>
<td id="S5.T7.5.7.6.2" class="ltx_td ltx_align_center">1.94 GB</td>
<td id="S5.T7.5.7.6.3" class="ltx_td ltx_align_center">5.87</td>
<td id="S5.T7.5.7.6.4" class="ltx_td ltx_align_center">6.99</td>
</tr>
<tr id="S5.T7.5.8.7" class="ltx_tr">
<th id="S5.T7.5.8.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">Q2_K</th>
<td id="S5.T7.5.8.7.2" class="ltx_td ltx_align_center ltx_border_bb">1.51 GB</td>
<td id="S5.T7.5.8.7.3" class="ltx_td ltx_align_center ltx_border_bb">3.71</td>
<td id="S5.T7.5.8.7.4" class="ltx_td ltx_align_center ltx_border_bb">9.42</td>
</tr>
</tbody>
</table>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
<section id="S6" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">6 </span>Model quantization</h2>

<div id="S6.p1" class="ltx_para">
<p id="S6.p1.1" class="ltx_p">To facilitate the use of our models on edge devices, we introduce quantized versions of <span id="S6.p1.1.1" class="ltx_text ltx_font_italic">H2O-Danube3-4B-Chat</span> and <span id="S6.p1.1.2" class="ltx_text ltx_font_italic">H2O-Danube3-500M-Chat</span>. They are available in the <span id="S6.p1.1.3" class="ltx_text ltx_font_italic">H2O-Danube3</span> Hugging Face collection and contain GGUF format model files that were quantized using the llama.cpp<span id="footnote6" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">6</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">6</sup><span class="ltx_tag ltx_tag_note">6</span><a target="_blank" href="https://github.com/ggerganov/llama.cpp" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/ggerganov/llama.cpp</a></span></span></span> framework.</p>
</div>
<div id="S6.p2" class="ltx_para">
<p id="S6.p2.1" class="ltx_p">Table <a href="#S5.T7" title="Table 7 ‣ 5 Evaluation ‣ H2O-Danube3 Technical Report" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a> summarizes different quantized versions of <span id="S6.p2.1.1" class="ltx_text ltx_font_italic">H2O-Danube3-4B-Chat</span>. It shows the trade-off between size and quality of different quantization methods. Columns in the table represent quantization method, size of the model in gigabytes, MT-Bench <cite class="ltx_cite ltx_citemacro_cite"><a href="#bib.bib16" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">zheng2023judging</span> </a></cite> benchmark score, and perplexity metric on WikiText-2 dataset (as reported in a perplexity test from llama.cpp). Results suggest that we can reduce the model size by a factor of 3.3 (4-bit quantization) keeping the quality of the model almost the same, but going to 3-bit quantization already decreases the performance significantly.</p>
</div>
</section>
<section id="S7" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">7 </span>Conclusions</h2>

<div id="S7.p1" class="ltx_para">
<p id="S7.p1.1" class="ltx_p">We introduce <span id="S7.p1.1.1" class="ltx_text ltx_font_italic">H2O-Danube3</span>, a series of small language models consisting of <span id="S7.p1.1.2" class="ltx_text ltx_font_italic">H2O-Danube3-4B</span> and <span id="S7.p1.1.3" class="ltx_text ltx_font_italic">H2O-Danube3-500M</span> released open source under Apache 2.0. Our models show competitive performance compared to popular models of similar size across a wide variety of benchmarks including (1) academic benchmarks, (2) chat benchmarks, as well as (3) fine-tuning benchmarks. <span id="S7.p1.1.4" class="ltx_text ltx_font_italic">H2O-Danube3</span> is built on our continuous efforts to contribute to the growing ecosystem of open source small language models. We are confident that our models can play a pivotal role in a wide range of applications, from typical chatting and fine-tuning for specific use cases to on-device offline applications on mobile phones or edge devices.</p>
</div>
</section>
<section id="bib" class="ltx_bibliography">
<h2 class="ltx_title ltx_title_bibliography" style="font-size:90%;">References</h2>

<ul class="ltx_biblist">
<li id="bib.bib1" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib1.2.2.1" class="ltx_text" style="font-size:90%;">[1]</span></span>
<span class="ltx_bibblock"><span id="bib.bib1.4.1" class="ltx_text" style="font-size:90%;">
Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib1.5.1" class="ltx_text" style="font-size:90%;">Gqa: Training generalized multi-query transformer models from multi-head checkpoints.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib1.6.1" class="ltx_text ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2305.13245</span><span id="bib.bib1.7.2" class="ltx_text" style="font-size:90%;">, 2023.
</span>
</span>
</li>
<li id="bib.bib2" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib2.2.2.1" class="ltx_text" style="font-size:90%;">[2]</span></span>
<span class="ltx_bibblock"><span id="bib.bib2.4.1" class="ltx_text" style="font-size:90%;">
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib2.5.1" class="ltx_text" style="font-size:90%;">Qwen technical report.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib2.6.1" class="ltx_text ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2309.16609</span><span id="bib.bib2.7.2" class="ltx_text" style="font-size:90%;">, 2023.
</span>
</span>
</li>
<li id="bib.bib3" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib3.2.2.1" class="ltx_text" style="font-size:90%;">[3]</span></span>
<span class="ltx_bibblock"><span id="bib.bib3.4.1" class="ltx_text" style="font-size:90%;">
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib3.5.1" class="ltx_text" style="font-size:90%;">Pythia: A suite for analyzing large language models across training and scaling.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib3.6.1" class="ltx_text" style="font-size:90%;">In </span><span id="bib.bib3.7.2" class="ltx_text ltx_font_italic" style="font-size:90%;">International Conference on Machine Learning</span><span id="bib.bib3.8.3" class="ltx_text" style="font-size:90%;">, pages 2397–2430. PMLR, 2023.
</span>
</span>
</li>
<li id="bib.bib4" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib4.2.2.1" class="ltx_text" style="font-size:90%;">[4]</span></span>
<span class="ltx_bibblock"><span id="bib.bib4.4.1" class="ltx_text" style="font-size:90%;">
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib4.5.1" class="ltx_text" style="font-size:90%;">Bert: Pre-training of deep bidirectional transformers for language understanding.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib4.6.1" class="ltx_text ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:1810.04805</span><span id="bib.bib4.7.2" class="ltx_text" style="font-size:90%;">, 2018.
</span>
</span>
</li>
<li id="bib.bib5" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib5.2.2.1" class="ltx_text" style="font-size:90%;">[5]</span></span>
<span class="ltx_bibblock"><span id="bib.bib5.4.1" class="ltx_text" style="font-size:90%;">
Marah Abdin et al.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib5.5.1" class="ltx_text" style="font-size:90%;">Phi-3 technical report: A highly capable language model locally on your phone, 2024.
</span>
</span>
</li>
<li id="bib.bib6" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib6.2.2.1" class="ltx_text" style="font-size:90%;">[6]</span></span>
<span class="ltx_bibblock"><span id="bib.bib6.4.1" class="ltx_text" style="font-size:90%;">
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib6.5.1" class="ltx_text" style="font-size:90%;">A framework for few-shot language model evaluation, September 2021.
</span>
</span>
</li>
<li id="bib.bib7" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib7.2.2.1" class="ltx_text" style="font-size:90%;">[7]</span></span>
<span class="ltx_bibblock"><span id="bib.bib7.4.1" class="ltx_text" style="font-size:90%;">
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib7.5.1" class="ltx_text" style="font-size:90%;">Deberta: Decoding-enhanced bert with disentangled attention.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib7.6.1" class="ltx_text ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2006.03654</span><span id="bib.bib7.7.2" class="ltx_text" style="font-size:90%;">, 2020.
</span>
</span>
</li>
<li id="bib.bib8" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib8.2.2.1" class="ltx_text" style="font-size:90%;">[8]</span></span>
<span class="ltx_bibblock"><span id="bib.bib8.4.1" class="ltx_text" style="font-size:90%;">
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib8.5.1" class="ltx_text" style="font-size:90%;">Mistral 7b.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib8.6.1" class="ltx_text ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2310.06825</span><span id="bib.bib8.7.2" class="ltx_text" style="font-size:90%;">, 2023.
</span>
</span>
</li>
<li id="bib.bib9" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib9.2.2.1" class="ltx_text" style="font-size:90%;">[9]</span></span>
<span class="ltx_bibblock"><span id="bib.bib9.4.1" class="ltx_text" style="font-size:90%;">
Bill Yuchen Lin, Yuntian Deng, Khyathi Chandu, Faeze Brahman, Abhilasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan Le Bras, and Yejin Choi.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib9.5.1" class="ltx_text" style="font-size:90%;">Wildbench: Benchmarking llms with challenging tasks from real users in the wild, 2024.
</span>
</span>
</li>
<li id="bib.bib10" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib10.2.2.1" class="ltx_text" style="font-size:90%;">[10]</span></span>
<span class="ltx_bibblock"><span id="bib.bib10.4.1" class="ltx_text" style="font-size:90%;">
Zechun Liu, Changsheng Zhao, Forrest Iandola, Chen Lai, Yuandong Tian, Igor Fedorov, Yunyang Xiong, Ernie Chang, Yangyang Shi, Raghuraman Krishnamoorthi, et al.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib10.5.1" class="ltx_text" style="font-size:90%;">Mobilellm: Optimizing sub-billion parameter language models for on-device use cases.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib10.6.1" class="ltx_text ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2402.14905</span><span id="bib.bib10.7.2" class="ltx_text" style="font-size:90%;">, 2024.
</span>
</span>
</li>
<li id="bib.bib11" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib11.2.2.1" class="ltx_text" style="font-size:90%;">[11]</span></span>
<span class="ltx_bibblock"><span id="bib.bib11.4.1" class="ltx_text" style="font-size:90%;">
Philipp Singer, Pascal Pfeiffer, Yauhen Babakhin, Maximilian Jeblick, Nischay Dhankhar, Gabor Fodor, and Sri Satish Ambati.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib11.5.1" class="ltx_text" style="font-size:90%;">H2o-danube-1.8b technical report, 2024.
</span>
</span>
</li>
<li id="bib.bib12" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib12.2.2.1" class="ltx_text" style="font-size:90%;">[12]</span></span>
<span class="ltx_bibblock"><span id="bib.bib12.4.1" class="ltx_text" style="font-size:90%;">
Stability AI Language Team.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib12.5.1" class="ltx_text" style="font-size:90%;">Introducing stable lm 2 1.6b.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib12.6.1" class="ltx_text" style="font-size:90%;">Last accessed on 2024-01-22. </span><a target="_blank" href="https://stability.ai/news/introducing-stable-lm-2" title="" class="ltx_ref ltx_url ltx_font_typewriter" style="font-size:90%;">https://stability.ai/news/introducing-stable-lm-2</a><span id="bib.bib12.7.2" class="ltx_text" style="font-size:90%;">.
</span>
</span>
</li>
<li id="bib.bib13" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib13.2.2.1" class="ltx_text" style="font-size:90%;">[13]</span></span>
<span class="ltx_bibblock"><span id="bib.bib13.4.1" class="ltx_text" style="font-size:90%;">
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib13.5.1" class="ltx_text" style="font-size:90%;">Llama 2: Open foundation and fine-tuned chat models.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib13.6.1" class="ltx_text ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2307.09288</span><span id="bib.bib13.7.2" class="ltx_text" style="font-size:90%;">, 2023.
</span>
</span>
</li>
<li id="bib.bib14" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib14.2.2.1" class="ltx_text" style="font-size:90%;">[14]</span></span>
<span class="ltx_bibblock"><span id="bib.bib14.4.1" class="ltx_text" style="font-size:90%;">
Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib14.5.1" class="ltx_text" style="font-size:90%;">Tinyllama: An open-source small language model.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib14.6.1" class="ltx_text ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2401.02385</span><span id="bib.bib14.7.2" class="ltx_text" style="font-size:90%;">, 2024.
</span>
</span>
</li>
<li id="bib.bib15" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib15.2.2.1" class="ltx_text" style="font-size:90%;">[15]</span></span>
<span class="ltx_bibblock"><span id="bib.bib15.4.1" class="ltx_text" style="font-size:90%;">
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib15.5.1" class="ltx_text" style="font-size:90%;">Opt: Open pre-trained transformer language models.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib15.6.1" class="ltx_text ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2205.01068</span><span id="bib.bib15.7.2" class="ltx_text" style="font-size:90%;">, 2022.
</span>
</span>
</li>
<li id="bib.bib16" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib16.2.2.1" class="ltx_text" style="font-size:90%;">[16]</span></span>
<span class="ltx_bibblock"><span id="bib.bib16.4.1" class="ltx_text" style="font-size:90%;">
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib16.5.1" class="ltx_text" style="font-size:90%;">Judging llm-as-a-judge with mt-bench and chatbot arena.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib16.6.1" class="ltx_text ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2306.05685</span><span id="bib.bib16.7.2" class="ltx_text" style="font-size:90%;">, 2023.
</span>
</span>
</li>
</ul>
</section>
</article>
</div>
<div class="ar5iv-footer"><a href="/html/2407.09275" class="ar5iv-nav-button ar5iv-nav-button-prev">◄</a>
    <a class="ar5iv-home-button" href="/"><img height="40" alt="ar5iv homepage" src="/assets/ar5iv.png"></a>
    <a href="/feeling_lucky" class="ar5iv-text-button">Feeling<br>lucky?</a>
    <a href="/log/2407.09276" class="ar5iv-text-button ar5iv-severity-warning">Conversion<br>report</a>
    <a class="ar5iv-text-button" target="_blank" href="https://github.com/dginev/ar5iv/issues/new?template=improve-article--arxiv-id-.md&title=Improve+article+2407.09276">Report<br>an issue</a>
    <a href="https://arxiv.org/abs/2407.09276" class="ar5iv-text-button arxiv-ui-theme">View&nbsp;original<br>on&nbsp;arXiv</a><a href="/html/2407.09277" class="ar5iv-nav-button ar5iv-nav-button-next">►</a>
</div><footer class="ltx_page_footer">
<a class="ar5iv-toggle-color-scheme" href="javascript:toggleColorScheme()" title="Toggle ar5iv color scheme"><span class="color-scheme-icon"></span></a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/license" target="_blank">Copyright</a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/policies/privacy_policy" target="_blank">Privacy Policy</a>

<div class="ltx_page_logo">Generated  on Mon Aug  5 18:02:16 2024 by <a target="_blank" href="http://dlmf.nist.gov/LaTeXML/" class="ltx_LaTeXML_logo"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg==" alt="Mascot Sammy"></a>
</div></footer>
</div>

    <script>
      var canMathML = typeof(MathMLElement) == "function";
      if (!canMathML) {
        var body = document.querySelector("body");
        body.firstElementChild.setAttribute('style', 'opacity: 0;');
        var loading = document.createElement("div");
        loading.setAttribute("id", "mathjax-loading-spinner");
        var message = document.createElement("div");
        message.setAttribute("id", "mathjax-loading-message");
        message.innerText = "Typesetting Equations...";
        body.prepend(loading);
        body.prepend(message);

        var el = document.createElement("script");
        el.src = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js";
        document.querySelector("head").appendChild(el);

        window.MathJax = {
          startup: {
            pageReady: () => {
              return MathJax.startup.defaultPageReady().then(() => {
                body.removeChild(loading);
                body.removeChild(message);
                body.firstElementChild.removeAttribute('style');
              }); } } };
      }
    </script>
    <script>
    // Auxiliary function, building the preview feature when
    // an inline citation is clicked
    function clicked_cite(e) {
      e.preventDefault();
      let cite = this.closest('.ltx_cite');
      let next = cite.nextSibling;
      if (next && next.nodeType == Node.ELEMENT_NODE && next.getAttribute('class') == "ar5iv-bibitem-preview") {
        next.remove();
        return; }
      // Before adding a preview modal,
      // cleanup older previews, in case they're still open
      document.querySelectorAll('span.ar5iv-bibitem-preview').forEach(function(node) {
        node.remove();
      })

      // Create the preview
      preview = document.createElement('span');
      preview.setAttribute('class','ar5iv-bibitem-preview');
      let target = document.getElementById(this.getAttribute('href').slice(1));
      target.childNodes.forEach(function (child) {
        preview.append(child.cloneNode(true));
      });
      let close_x = document.createElement('button');
      close_x.setAttribute("aria-label","Close modal for bibliography item preview");
      close_x.textContent = "×";
      close_x.setAttribute('class', 'ar5iv-button-close-preview');
      close_x.setAttribute('onclick','this.parentNode.remove()');
      preview.append(close_x);
      preview.querySelectorAll('.ltx_tag_bibitem').forEach(function(node) {
        node.remove();
      });
      cite.parentNode.insertBefore(preview, cite.nextSibling);
      return;
    }
    // Global Document initialization:
    // - assign the preview feature to all inline citation links
    document.querySelectorAll(".ltx_cite .ltx_ref").forEach(function (link) {
      link.addEventListener("click", clicked_cite);
    });
    </script>
    </body>
</html>

2407.09276
1AbstractWe presentH2O-Danube3, a series of small language models consisting ofH2O-Danube3-4B, trained ontokens andH2O-Danube3-500M, trained ontokens.
Our models are pre-trained on high quality Web data consisting of primarily English tokens in three stages with different data mixes before final supervised tuning for chat version.
The models exhibit highly competitive metrics across a multitude of academic, chat, and fine-tuning benchmarks.
Thanks to its compact architecture,H2O-Danube3can be efficiently run on a modern smartphone, enabling local inference and rapid processing capabilities even on mobile devices.
We make all models openly available under Apache 2.0 license further democratizing LLMs to a wider audience economically.Danube3 model collection:https://huggingface.co/collections/h2oai/h2o-danube3-6687a993641452457854c609
introduction
H2O-Danube3 Technical Report
1Abstract
2407.09276
2IntroductionSmall language models have taken a pivotal place in today’s open source language model landscape particularly aiming at efficient inference on consumer hardware and edge devices also allowing for full offline applications. Additionally, smaller models have proven to be particularly useful after fine-tuning them for specific tasks, such as sequence classification, question answering, or token classification even outpacing previously used encoder/decoder models such as those stemming from BERT and its derivatives.We extend previous research in this areaand presentH2O-Danube3, a series of small language models consisting ofH2O-Danube3-4B, trained ontokens andH2O-Danube3-500M, trained ontokens based on incremental research and training efforts.
In this report, we present an overview of the models, detailing their architecture, training procedures, and fine-tuning processes. We offer extensive evaluations using a diverse range of benchmarks, encompassing both standard academic metrics, chat benchmarks, and fine-tuning benchmarks.Results show thatH2O-Danube3exhibits competitive benchmarks across all dimensions, expanding the repertoire of open source small language models. We hope our work can further democratize language models to a wider audience and that our models can play a pivotal role for various use cases such as (1)  chatbot applications, (2)  RAG applications, (3)  fine-tuning for specific use cases such as classification, (4)  research or (5)  on-device offline applications. To demonstrate the potential, we also presentH2O AI Personal GPT111https://h2o.ai/platform/danube/personal-gpt/, an iOS application allowing to runH2O-Danube3fully offline on a modern phone device.Figure 1:Data stages forH2O-Danube3-4B.The model is trained over three different stages with different data mixes. The first data stage consist of 90.6% of web data which is gradually decreasing to 81.7% at the second stage, and to 51.6% at the third stage. The first two stages include the majority of the tokens: 4.6T and 1.35T tokens respectively, while the third stage comprises of 0.05T tokens.
body
H2O-Danube3 Technical Report
2Introduction
2407.09276
3Model architectureH2O-Danube3is a family of decoder only LLM models that use the general Llama model architecture adopting core principles from Llama 2and Mistralwith custom parameters determining the shape of each layer and total parameter count. We use the Mistral tokenizer with a vocabulary size ofand train our model up to a context length of. We make use of Grouped Query Attentionand optimize towards parameter and compute efficiency resulting in a wide architecture (see Table1) . In total,H2O-Danube3-4Bconsists of 3.96B trainable parameters. In addition, we releaseH2O-Danube3-500Mwith 500M trainable parameters for edge devices with limited compute or for custom fine-tuning tasks that require low memory footprint or high throughput at low cost.Table 1:Key model parameters.Parameters500M4BLayers1624Hidden size15363840Intermediate size409610240Num heads1632Num KV heads88Head size96120Vocab size3200032000RoPE theta100000100000
body
H2O-Danube3 Technical Report
3Model architecture
2407.09276
4TrainingModels are primarily trained on English text in three stages with different data mixes. At each stage, we gradually decrease the percentage of noisy web data in favor of higher quality data. The first data stage consist of 90.6% of web data which is gradually decreasing to 81.7% at the second stage, and to 51.6% at the third stage. Simultaneously, the share of instruct data, Wikipedia, academic texts, synthetic texts and other higher quality textual data is increasing. The first two stages include the majority of the tokens:andtokens respectively (andtokens forH2O-Danube3-500M) , while third stage comprises oftokens. The data distribution across stages is presented in Figure1.We also provide chat fine-tuned versionsH2O-Danube3-4B-ChatandH2O-Danube3-500M-Chat. We utilizeH2O LLM Studio222https://github.com/h2oai/h2o-llmstudio, an Apache 2.0 open-source framework and no-code GUI for fine-tuning LLMs.
We tune the base model using supervised fine-tuning (SFT)  on input/output conversational pairs. We mask the prompt loss, and use a custom prompt format. Hyperparameters were optimized iterating over multiple experiments.
body
H2O-Danube3 Technical Report
4Training
2407.09276
5EvaluationIn this section, we present evaluation ofH2O-Danube3across a variety of dimensions, focusing on (1)  academic benchmarks, (2)  chat benchmarks and (3)  fine-tuning benchmarks.Academic benchmarks.We evaluateH2O-Danube3on a wide range of benchmarks and compare it with other existing open-source language models which have a similar number of parameters, specifically Qwen/Qwen1.5-4B-Chat, stabilityai/stablelm-zephyr-3b and microsoft/Phi-3-mini-4k-instruct. We also compare to our previous model h2oai/h2o-danube2-1.8b-chat. To evaluate the models, we use the Language Model Evaluation Harness framework333commit e5e5ee0cb629c9c88165292d1b4bf34623392d33.H2O-Danube3-4Bshows very competitive and consistent results across all reported benchmarks (see Table2) . It is the best-in-class model for the knowledge based CommonsenseQA benchmark and PhysicsQA and achieves a strong accuracy of 50.14% on the math centered benchmark GSM8K. In all other benchmarks,H2O-Danube3-4Branks second only after Phi-3-mini-4k-instruct which is well known for its outstanding reasoning capabilities and strong benchmark scores. Notably,H2O-Danube3-4Bscores over 80% on 10-shot hellaswag benchmark, closing the gap to much larger models. The smallerH2O-Danube3-500Mis evaluated against the same benchmarks and compared to similar sized Qwen2-0.5B-Instruct (see Table3) . Our model scores highest in eight out of twelve benchmarks and we consider it a new well rounded model for this parameter count.Table 2:Academic benchmarks.Academic benchmark results, compared to openly-available models of similar size and trained on general English text data. We compare the instruction fine-tuned models h2oai/h2o-danube2-1.8b-chat, h2oai/h2o-danube3-4b-chat, Qwen/Qwen1.5-4B-Chat, stabilityai/stablelm-zephyr-3b, microsoft/Phi-3-mini-4k-instruct. To evaluate the models, we use the Language Model Evaluation Harness framework.BenchmarkMetricDanube2Danube3Qwen1.5StableLMPhi31.8B4B4B3B4BARC-c25-shot43.6958.9642.1547.7063.91Hellaswag10-shot73.9180.3669.4673.7180.62MMLU5-shot37.8354.7454.0344.9869.43TruthfulQA0-shot mc240.5347.7944.8846.4057.72Winogrande5-shot69.3076.4866.2265.5970.80GSM8K5-shot32.3050.183.6352.4677.48ARC-e25-shot74.9283.8473.4472.1087.29BBH3-shot CoT30.3938.9221.0336.7771.42CommonsenseQA3-shot54.3079.5276.0975.7677.81CoQA0-shot F168.3077.2361.9470.8679.75PIQA3-shot78.6782.6476.6177.4278.35SciQ3-shot95.7097.1095.4094.8097.60Average58.3268.9857.0763.2176.01Table 3:Academic benchmarks for smaller models.Academic benchmark results, compared to openly-available models of similar size and trained on general English text data. We compare the instruction fine-tuned models h2oai/h2o-danube3-500m-chat and Qwen/Qwen2-0.5B-Instruct. To evaluate the models, we use the Language Model Evaluation Harness framework.BenchmarkMetricDanube3Qwen20.5B0.5BARC-c25-shot39.2532.00Hellaswag10-shot61.0249.11MMLU5-shot26.3343.88TruthfulQA0-shot mc239.9639.28Winogrande5-shot61.7256.99GSM8K5-shot16.0034.12ARC-e25-shot71.8462.12BBH3-shot CoT25.1418.98CommonsenseQA3-shot19.5752.74CoQA0-shot F148.0254.89PIQA3-shot74.7068.72SciQ3-shot95.4092.60Chat benchmarks.Evaluating chat and instruct fine-tuned LLMs remains a critical challenge and can most reliably be conducted by large scale human assessment. In order to give an initial evaluation of our chat model, we resort toMT-BenchandWildBench-v2benchmarks. They represent a collection of multi-turn questions across different categories followed by GPT-4 judgement which assigns a score from 1 to 10 for each model’s response. Results are presented in Table4showing thatH2O-Danube3-4B-Chatis surpassing other similar sized models while Phi-3-mini takes the top spot. The 500M parameter version of the modelH2O-Danube3-500M-Chatshows results that are comparable to Qwen2-0.5B-Instruct (see Table5) .We additionally conducted multiple internal evaluations and show them in the same tables. First, we performed a blind evaluation of chat performance (excluding the 500M models)  following the idea of Chat Arena444https://chat.lmsys.org/. This involved presenting users with random pairs of models and allowing them to prompt and vote on output preference (A better, B better, both bad, both good) , followed by calculating an ELO score using MLE and bootstrapping. Second, we utilized an internal RAG (Retrieval-Augmented Generation)  benchmark555https://github.com/h2oai/enterprise-h2ogpte/tree/main/rag_benchmarkto assess the models performance in question-answering tasks based on long PDF documents. We calculated an accuracy score for each model by comparing its generated responses to the ground truth answers.Table 4:Chat benchmarks.H2O-Danube3-4B-Chatconsistently performs very well across all benchmarks, surpassing other similar sized models and outperforming our previous Danube2 release, while Phi-3-mini takes the top spot.BenchmarkMetricDanube2Danube3Qwen1.5StableLMPhi31.8B4B4B3B4BMT-BenchTurn 16.417.286.687.108.38MT-BenchTurn 24.885.695.335.747.58MT-BenchAverage5.646.496.006.427.98WildBench-v2Raw score4.655.544.875.516.47Internal VotingELO1531146614351564RAG BenchmarkAccuracy66.8873.3767.5368.1873.37Table 5:Chat benchmarks for smaller models.The 500M parameter version of the modelH2O-Danube3-500M-Chatshows results that are comparable to Qwen2-0.5B-Instruct model. In particular, they achieve a close MT-Bench average score (H2O-Danube3-500M-Chatbeing better in the 1st turn) , whileH2O-Danube3-500M-Chatproduces better results on Wild-Bench benchmark.BenchmarkMetricDanube3Qwen20.5B0.5BMT-BenchTurn 14.163.78MT-BenchTurn 22.402.76MT-BenchAverage3.283.27WildBench-v2Raw score3.363.11Internal RAG BenchmarkAccuracy44.1650.00Fine-tuning benchmarks.A common application of small language models is their fine-tuning for various use cases to optimize performance on specific tasks. To that end, we also evaluate the different models’ capability to be easily adaptable to new tasks, here focusing on text classification observed frequently across various use cases in businesses and applications.We employ the following process for our fine-tuning benchmarks. We utilizeH2O LLM Studiooffering out-of-the-box for tuning language models for classification feeding the final token logit distribution in a custom head for classification. For all models and datasets, we use the same settings with LoRA (,)  and same hyperparameters (,,,,) . These settings are commonly used default settings in the field and we aim at particularly evaluating the default performance of models after tuning.
We look at the following datasets, that all can be found on Hugging Face:•stanfordnlp/imdb: binary sentence classification of imdb movie reviews•knowledgator/Scientific-text-classification: classification of scientific texts into 10 most frequent classes, random 50-50 split•ccdv/arxiv-classification: long context classification of arxiv papers into 11 classes•ccdv/patent-classification: long context classification of patents into 9 classesTable6highlights the results for individual datasets and models, we always report the accuracy taking the highest probability class. We can see, that all small language models show excellent performance on text classification tasks after fine-tuning. Even small 500M parameter models can be highly competitive, exemplifying the utility of fine-tuning such models for specific use cases. Overall,H2O-Danube3-4Btakes a leading spot in all benchmarks. These results can be seen as baseline results based on default hyperparameter settings. More extensive parameter sweeps would potentially improve results of all models at hand, and might also alter the order of performance. We plan on investigating such fine-tuning performance more extensively in the future.Table 6:Fine-tuning benchmarks.All models show excellent performance on various classification tasks after fine-tuning withH2O-Danube3-4Btaking a top spot in most benchmarks.DatasetDanube2Danube3Danube3Qwen1.5Qwen2StableLMPhi31.8B0.5B4B4B0.5B3B4BArxiv0.8640.8630.8730.8770.8740.8650.869Imdb0.9680.9590.9710.9700.9590.9690.967Patent0.7210.7080.7270.7170.7070.7190.712Scientific0.8680.8460.8720.8750.8550.8670.870Average0.8550.8440.8610.860.8490.8550.854Table 7:Model quantization.This table summarizes different quantized versions ofH2O-Danube3-4B-Chatshowing the trade-off between size and quality of the models. Results indicate that quantized models reduced to 4 bits exhibit minimal loss in benchmark performance.Quant methodModel sizeMT-BenchPerplexityF167.92 GB6.436.17Q8_04.21 GB6.496.17Q6_K3.25 GB6.376.20Q5_K_M2.81 GB6.256.24Q4_K_M2.39 GB6.316.37Q3_K_M1.94 GB5.876.99Q2_K1.51 GB3.719.42
body
H2O-Danube3 Technical Report
5Evaluation
2407.09276
6Model quantizationTo facilitate the use of our models on edge devices, we introduce quantized versions ofH2O-Danube3-4B-ChatandH2O-Danube3-500M-Chat. They are available in theH2O-Danube3Hugging Face collection and contain GGUF format model files that were quantized using the llama.cpp666https://github.com/ggerganov/llama.cppframework.Table7summarizes different quantized versions ofH2O-Danube3-4B-Chat. It shows the trade-off between size and quality of different quantization methods. Columns in the table represent quantization method, size of the model in gigabytes, MT-Benchbenchmark score, and perplexity metric on WikiText-2 dataset (as reported in a perplexity test from llama.cpp) . Results suggest that we can reduce the model size by a factor of 3.3 (4-bit quantization)  keeping the quality of the model almost the same, but going to 3-bit quantization already decreases the performance significantly.
body
H2O-Danube3 Technical Report
6Model quantization
2407.09276
7ConclusionsWe introduceH2O-Danube3, a series of small language models consisting ofH2O-Danube3-4BandH2O-Danube3-500Mreleased open source under Apache 2.0. Our models show competitive performance compared to popular models of similar size across a wide variety of benchmarks including (1)  academic benchmarks, (2)  chat benchmarks, as well as (3)  fine-tuning benchmarks.H2O-Danube3is built on our continuous efforts to contribute to the growing ecosystem of open source small language models. We are confident that our models can play a pivotal role in a wide range of applications, from typical chatting and fine-tuning for specific use cases to on-device offline applications on mobile phones or edge devices.
conclusion
H2O-Danube3 Technical Report
7Conclusions