Update index.html

index.html

@@ -32,15 +32,15 @@
           <h1 class="title is-1 publication-title">Atla Selene Mini:<br>A General Purpose Evaluation Model</h1>
           <div class="is-size-5 publication-authors">
             <span class="author-block">
-              <b>Andrei Alexandru</b><sup>1</sup>,</span>
+              Andrei Alexandru<sup>1</sup>,</span>
             <span class="author-block">
-              <b>Antonia Calvi</b><sup>1</sup>,</span>
+              Antonia Calvi<sup>1</sup>,</span>
             <span class="author-block">
-              <b>Henry Broomfield</b><sup>1</sup>,</span>
+              Henry Broomfield<sup>1</sup>,</span>
             <span class="author-block">
-              <b>Jackson Golden</b><sup>1</sup>,</span>
+              Jackson Golden<sup>1</sup>,</span>
             <span class="author-block">
-              <b>Kyle Dai</b><sup>1</sup>,</span>
+              Kyle Dai<sup>1</sup>,</span>
           </div>
           <div class="is-size-5 publication-authors">
             <span class="author-block">
@@ -69,10 +69,23 @@
 
           <div class="column has-text-centered">
             <div class="publication-links">
-              <!-- Model Link -->
+              <!-- PDF Link -->
+              <span class="link-block">
+                <a href="arxiv_submitted.pdf" target="_blank"
+                   class="external-link button is-normal is-rounded is-dark">
+                  <span class="icon">
+                      <i class="fas fa-file-pdf"></i>
+                  </span>
+                  <span>Paper</span>
+                </a>
+              </span>
+              <!-- HuggingFace Link -->
               <span class="link-block">
                 <a href="https://hf.co/AtlaAI/Selene-1-Mini-Llama-3.1-8B" target="_blank"
                    class="external-link button is-normal is-rounded is-dark">
+                  <span class="icon">
+                      <i class="fab fa-github"></i>
+                  </span>
                   <span>HuggingFace</span>
                 </a>
               </span>
@@ -80,6 +93,9 @@
               <span class="link-block">
                 <a href="https://ollama.com/atla/selene-mini" target="_blank"
                    class="external-link button is-normal is-rounded is-dark">
+                  <span class="icon">
+                      <i class="fas fa-code"></i>
+                  </span>
                   <span>Ollama</span>
                 </a>
               </span>
@@ -93,6 +109,13 @@
 
 <section class="section">
   <div class="container is-max-desktop">
+    <!-- Logo -->
+    <div class="columns is-centered has-text-centered">
+      <div class="column is-4">
+        <img src="figs/atla-logo.png" alt="Atla Logo">
+      </div>
+    </div>
+    
     <!-- Abstract -->
     <div class="columns is-centered has-text-centered">
       <div class="column is-four-fifths">
@@ -111,12 +134,19 @@
       </div>
     </div>
 
-    <!-- Figure 1 -->
-    <div class="columns is-centered has-text-centered">
+    <!-- Introduction -->
+    <div class="columns is-centered">
       <div class="column is-four-fifths">
-        <div class="content">
-          <figure>
-            <img src="/api/placeholder/800/400" alt="Performance comparison">
+        <h2 class="title is-3">Introduction</h2>
+        <div class="content has-text-justified">
+          <p>
+            Automated evaluation of large language models (LLMs) is an increasingly pertinent task as LLMs demonstrate their value across a growing array of real-world use cases. Reliable evaluation is critical to ensure that LLMs are aligned with human objectives, i.e. that these models do what they are intended to do.
+          </p>
+          <p>
+            Human evaluation is time-consuming and expensive, and scales poorly with volume and complexity – hence the need for scalable, automated techniques. As generative models have become more capable, the field has addressed this need by using LLMs themselves to evaluate other LLMs' responses, producing judgments and natural language critiques without humans in the loop – an approach also known as "LLM-as-a-judge" (LLMJ).
+          </p>
+          <figure class="image">
+            <img src="figs/Fig1.png" alt="Performance comparison">
             <figcaption>
               <b>Figure 1:</b> Atla Selene Mini outperforms current state-of-the-art SLMJs: a) Overall task-average performance, comparing Atla Selene Mini (black) with the best and most widely used SLMJs. b) Breakdown of performance by task type and benchmark.
             </figcaption>
@@ -133,12 +163,33 @@
           <p>
             Selene Mini is optimized for fast inference, high performance, and promptability. It is a general-purpose evaluator, and is trained to respond with both critiques and judgments in order to deliver actionable insights. To achieve this, we fine-tuned a Llama 3.1 8B Instruct model on a curated mixture of 16 publicly available datasets, totaling 577k data points.
           </p>
-          <figure>
-            <img src="/api/placeholder/800/400" alt="Data curation strategy">
+
+          <figure class="image">
+            <img src="figs/Fig2.png" alt="Data curation strategy">
             <figcaption>
               <b>Figure 2:</b> Data curation strategy: The process of transforming a candidate dataset (left) into the final training mix (right). Yellow boxes indicate filtering steps, purple represents synthetic generation of chosen and rejected pairs for preference optimization.
             </figcaption>
           </figure>
+
+          <h3 class="title is-4">Datasets</h3>
+          <p>
+            We took inspiration from the datasets used to train Foundational Large Autorater Models (FLAMe), which spanned a mix of pairwise, absolute scoring, and classification tasks. Each data point in these three task types was structured slightly differently.
+          </p>
+
+          <h3 class="title is-4">Synthetic augmentation</h3>
+          <p>
+            To construct pairs of contrasting evaluations, we generated rejected judgments that differed from the chosen ground-truth judgments in the data. For each judgment, we synthetically generated chosen and rejected chain-of-thought critiques by prompting a generation model to argue for the respective judgments.
+          </p>
+
+          <h3 class="title is-4">Filtering for quality</h3>
+          <p>
+            We used filtering strategies on both raw and synthetic data to ensure high quality. For raw data, we used ArmoRM, an off-the-shelf reward model, to score and filter four of our largest datasets that we hypothesized to contain high-variance in data quality.
+          </p>
+
+          <h3 class="title is-4">Training</h3>
+          <p>
+            We fine-tuned a Llama 3.1 8B Instruct model using the variant of DPO introduced in [citation], and refer readers to that paper for the full derivation.
+          </p>
         </div>
       </div>
     </div>
@@ -152,12 +203,107 @@
           <p>
             We assess the performance of Selene Mini on 11 out-of-distribution benchmarks, spanning three different types of evaluation tasks: absolute scoring, classification, and pairwise preference.
           </p>
-          <figure>
-            <img src="/api/placeholder/800/400" alt="Real-world evaluation">
+
+          <div class="table-container">
+            <table class="table is-bordered is-striped is-narrow is-hoverable is-fullwidth">
+              <caption>Table 1: Detailed performance breakdown across model sizes</caption>
+              <thead>
+                <tr>
+                  <th>Model</th>
+                  <th colspan="2">Overall (average)</th>
+                  <th colspan="3">Absolute scoring tasks</th>
+                  <th colspan="6">Pairwise preference tasks</th>
+                  <th colspan="2">Classification tasks</th>
+                </tr>
+                <tr>
+                  <th></th>
+                  <th>Tasks</th>
+                  <th>Benchmarks</th>
+                  <th>MT-Bench</th>
+                  <th>FLASK</th>
+                  <th>BiGGen</th>
+                  <th>RewardB</th>
+                  <th>LFQA</th>
+                  <th>HHH</th>
+                  <th>EvalBias</th>
+                  <th>InstruSum</th>
+                  <th>Auto-J</th>
+                  <th>InfoBench</th>
+                  <th>AggreFact</th>
+                </tr>
+              </thead>
+              <tbody>
+                <tr>
+                  <td>Atla-Selene-Mini</td>
+                  <td>0.756</td>
+                  <td>0.753</td>
+                  <td>0.746</td>
+                  <td>0.613</td>
+                  <td>0.584</td>
+                  <td>0.891</td>
+                  <td>0.688</td>
+                  <td>0.900</td>
+                  <td>0.863</td>
+                  <td>0.732</td>
+                  <td>0.576</td>
+                  <td>0.915</td>
+                  <td>0.778</td>
+                </tr>
+                <!-- Add other rows from the LaTeX table -->
+              </tbody>
+            </table>
+          </div>
+
+          <h3 class="title is-4">Real-world evaluation</h3>
+          <p>
+            While the performance of our SLMJ across a wide range of benchmarks offers an indication of its strong general-purpose evaluation capabilities, such benchmarks are often not entirely representative of realistic evaluation use cases.
+          </p>
+
+          <figure class="image">
+            <img src="figs/Fig3.png" alt="Real-world evaluation">
             <figcaption>
               <b>Figure 3:</b> Real-world evaluation: a) Performance on domain-specific industry benchmarks b) Performance on RewardBench with different prompt formats c) Performance measured by ELO scores in Judge Arena.
             </figcaption>
           </figure>
+
+          <div class="table-container">
+            <table class="table is-bordered is-striped is-narrow is-hoverable is-fullwidth">
+              <caption>Table 2: Industry benchmarks</caption>
+              <thead>
+                <tr>
+                  <th>Model</th>
+                  <th colspan="4">CRAFT-MD</th>
+                  <th>Finance</th>
+                </tr>
+                <tr>
+                  <th></th>
+                  <th>Medical terminology</th>
+                  <th>Most likely diagnosis</th>
+                  <th>Relevant med. hist.</th>
+                  <th>Overall</th>
+                  <th>Bench</th>
+                </tr>
+              </thead>
+              <tbody>
+                <tr>
+                  <td>Atla-Selene-Mini</td>
+                  <td>0.92</td>
+                  <td>0.62</td>
+                  <td>0.68</td>
+                  <td>0.74</td>
+                  <td>0.717</td>
+                </tr>
+                <tr>
+                  <td>LLama-3.1-8B-Instruct</td>
+                  <td>0.79</td>
+                  <td>0.51</td>
+                  <td>0.62</td>
+                  <td>0.64</td>
+                  <td>0.664</td>
+                </tr>
+              </tbody>
+            </table>
+          </div>
         </div>
       </div>
     </div>
@@ -171,7 +317,22 @@
             In this work, we introduce Atla Selene Mini, demonstrating that effective general-purpose evaluation can be achieved in smaller model architectures through principled data curation and a hybrid training objective (DPO + SFT). The model's strong performance across benchmarks, particularly on absolute scoring tasks – which represent the most common and useful form of evaluation in practice – suggests that careful attention to training data quality can be as impactful as increased model size for evaluation capabilities.
           </p>
           <p>
-            Looking ahead, we anticipate two emerging frontiers that will shape the future of AI evaluation. First is the rise of agent-based systems that combine language models with external tools and APIs, creating more powerful and versatile AI systems. Second is the increasing use of inference-time compute – systems that perform additional reasoning steps during inference to generate higher-quality outputs.
+            Looking ahead, we anticipate two emerging frontiers that will shape the future of AI evaluation. First is the rise of agent-based systems that combine language models with external tools and APIs, creating more powerful and versatile AI systems. Second is the increasing use of inference-time compute – systems that perform additional reasoning steps during inference to generate higher-quality outputs. These developments will require new evaluation frameworks and capabilities. Future research could explore how evaluator models can assess not just language outputs, but entire chains of reasoning, tool usage, and multi-step processes.
+          </p>
+          <p>
+            In conclusion, Atla Selene Mini represents a significant step forward in making reliable, general-purpose LLM evaluation more accessible to the broader community. Its combination of strong performance, domain generalization, and practical usability in an open-weights model provides a valuable tool for researchers and practitioners working to improve language model capabilities and safety.
+          </p>
+        </div>
+      </div>
+    </div>
+
+    <!-- Acknowledgments -->
+    <div class="columns is-centered">
+      <div class="column is-four-fifths">
+        <h2 class="title is-3">Acknowledgments</h2>
+        <div class="content has-text-justified">
+          <p>
+            We thank Clémentine Fourrier and the HuggingFace team for their help in setting up Judge Arena. We are grateful to Juan Felipe Cerón Uribe, Seungone Kim, Shreya Shankar, Eugene Yan, Yifan Mai, Austin Xu, Peifeng Wang and the team at SalesForce for helpful discussions around evaluations. We thank Zongheng Yang, Romil Bhardwaj and the Skypilot team for their assistance with our training infrastructure.
           </p>
         </div>
       </div>
@@ -179,6 +340,22 @@
   </div>
 </section>
 
+<footer class="footer">
+  <div class="container">
+    <div class="columns is-centered">
+      <div class="column is-8">
+        <div class="content">
+          <p class="has-text-centered">
+            © 2025 Atla AI
+          </p>
+        </div>
+      </div>
+    </div>
+  </div>
+</footer>
+  </div>
+</section>
+
 <footer class="footer">
   <div class="container">
     <div class="content has-text-centered">