<!DOCTYPE html><html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>[2310.15147] S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models</title><meta property="og:description" content="The rapid development of Large Language Models (LLMs) has led to great strides in model capabilities like reasoning and long-context understanding.
However, as LLMs are able to process longer contexts, it becomes more …">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models">
<meta name="twitter:image:src" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta name="twitter:image:alt" content="ar5iv logo">
<meta property="og:title" content="S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models">
<meta property="og:site_name" content="ar5iv">
<meta property="og:image" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta property="og:type" content="article">
<meta property="og:url" content="https://ar5iv.labs.arxiv.org/html/2310.15147">

<!--Generated on Tue Feb 27 23:36:40 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.-->
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

<script>
  function detectColorScheme(){
    var theme="light";
    var current_theme = localStorage.getItem("ar5iv_theme");
    if(current_theme){
      if(current_theme == "dark"){
        theme = "dark";
      } }
    else if(!window.matchMedia) { return false; }
    else if(window.matchMedia("(prefers-color-scheme: dark)").matches) {
      theme = "dark"; }
    if (theme=="dark") {
      document.documentElement.setAttribute("data-theme", "dark");
    } else {
      document.documentElement.setAttribute("data-theme", "light"); } }

  detectColorScheme();

  function toggleColorScheme(){
    var current_theme = localStorage.getItem("ar5iv_theme");
    if (current_theme) {
      if (current_theme == "light") {
        localStorage.setItem("ar5iv_theme", "dark"); }
      else {
        localStorage.setItem("ar5iv_theme", "light"); } }
    else {
        localStorage.setItem("ar5iv_theme", "dark"); }
    detectColorScheme(); }
</script>
<link media="all" rel="stylesheet" href="/assets/ar5iv-fonts.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv-site.0.2.2.css">
</head>
<body>
<div class="ltx_page_main">
<div class="ltx_page_content">
<article class="ltx_document ltx_authors_1line">
<div id="p1" class="ltx_para">
<span id="p1.1" class="ltx_ERROR undefined">\etocdepthtag</span>
<p id="p1.2" class="ltx_p">.tocchapter
<span id="p1.2.1" class="ltx_ERROR undefined">\etocsettagdepth</span>chapternone
<span id="p1.2.2" class="ltx_ERROR undefined">\etocsettagdepth</span>appendixnone</p>
</div>
<h1 class="ltx_title ltx_title_document">
<span id="id1.1" class="ltx_text" style="position:relative; bottom:-1.0pt;"><img src="/html/2310.15147/assets/x1.png" id="id1.1.g1" class="ltx_graphics ltx_img_square" width="12" height="10" alt="[Uncaptioned image]"></span> S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models</h1>
<div class="ltx_authors">
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">
<span id="id11.10.id1" class="ltx_text"></span> <span id="id7.6.6" class="ltx_text">
<span id="id7.6.6.6" class="ltx_tabular ltx_align_middle">
<span id="id4.3.3.3.3" class="ltx_tr">
<span id="id4.3.3.3.3.3" class="ltx_td ltx_nopad_r ltx_align_center">Fangyu Lei <sup id="id4.3.3.3.3.3.1" class="ltx_sup"><span id="id4.3.3.3.3.3.1.1" class="ltx_text ltx_font_italic">1,2</span></sup>  , Qian Liu <sup id="id4.3.3.3.3.3.2" class="ltx_sup"><span id="id4.3.3.3.3.3.2.1" class="ltx_text ltx_font_italic">3∗</span></sup>, Yiming Huang <sup id="id4.3.3.3.3.3.3" class="ltx_sup"><span id="id4.3.3.3.3.3.3.1" class="ltx_text ltx_font_italic">1∗</span></sup>,</span></span>
<span id="id7.6.6.6.6" class="ltx_tr">
<span id="id7.6.6.6.6.3" class="ltx_td ltx_nopad_r ltx_align_center">Shizhu He <sup id="id7.6.6.6.6.3.1" class="ltx_sup"><span id="id7.6.6.6.6.3.1.1" class="ltx_text ltx_font_italic">1,2</span></sup>, Jun Zhao <sup id="id7.6.6.6.6.3.2" class="ltx_sup"><span id="id7.6.6.6.6.3.2.1" class="ltx_text ltx_font_italic">1,2</span></sup>, Kang Liu <sup id="id7.6.6.6.6.3.3" class="ltx_sup"><span id="id7.6.6.6.6.3.3.1" class="ltx_text ltx_font_italic">1,2</span></sup></span></span>
</span></span> <span id="id12.11.id2" class="ltx_text"></span>


<br class="ltx_break"><span id="id10.9.9" class="ltx_text ltx_align_center"><sup id="id10.9.9.4" class="ltx_sup">1</sup>Institute of Automation, CAS  <sup id="id10.9.9.5" class="ltx_sup">2</sup>University of Chinese Academy of Sciences  <sup id="id10.9.9.6" class="ltx_sup">3</sup>Sea AI Lab</span>
<br class="ltx_break">
<br class="ltx_break">
</span><span class="ltx_author_notes">  Equal Contributions. Contact <a href="leifangyu2022@ia.ac.cn" title="" class="ltx_ref ltx_url ltx_font_typewriter">leifangyu2022@ia.ac.cn</a> and <a href="liuqian@sea.com" title="" class="ltx_ref ltx_url ltx_font_typewriter">liuqian@sea.com</a>.</span></span>
</div>

<div class="ltx_abstract">
<h6 class="ltx_title ltx_title_abstract">Abstract</h6>
<p id="id13.id1" class="ltx_p">The rapid development of Large Language Models (LLMs) has led to great strides in model capabilities like reasoning and long-context understanding.
However, as LLMs are able to process longer contexts, it becomes more challenging to evaluate whether they have acquired certain capabilities, since the length of text (e.g., 100K tokens) they can process far exceeds what humans can reliably assess in a reasonable duration.
In this paper, we propose using complex synthetic tasks as a proxy evaluation method, and present <span id="id13.id1.1" class="ltx_text ltx_font_smallcaps">S3Eval</span>, a <span id="id13.id1.2" class="ltx_text ltx_font_bold">S</span>ynthetic, <span id="id13.id1.3" class="ltx_text ltx_font_bold">S</span>calable, <span id="id13.id1.4" class="ltx_text ltx_font_bold">S</span>ystematic evaluation suite for LLMs evaluation.
As a synthetic benchmark, <span id="id13.id1.5" class="ltx_text ltx_font_smallcaps">S3Eval</span> enables the creation of any number of evaluation examples that are theoretically invisible to LLMs, mitigating the test set contamination issue.
The synthetic nature of <span id="id13.id1.6" class="ltx_text ltx_font_smallcaps">S3Eval</span> provides users full control over the dataset, allowing them to systematically probe LLM capabilities by scaling text length and varying task difficulty across diverse scenarios.
The strong correlation between <span id="id13.id1.7" class="ltx_text ltx_font_smallcaps">S3Eval</span> performance and scores of real-world benchmarks like Big-Bench Hard (BBH) demonstrates the soundness of using <span id="id13.id1.8" class="ltx_text ltx_font_smallcaps">S3Eval</span> for evaluation of LLMs.
The in-depth analysis also uncover additional insights, including performance drop when the answer is sparsely distributed or located in the middle context, as well as some counter-intuitive trends of model performance. Our code is available at <a target="_blank" href="https://github.com/lfy79001/SQLEval" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/lfy79001/SQLEval</a>.</p>
</div>
<section id="S1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">1 </span>Introduction</h2>

<div id="S1.p1" class="ltx_para ltx_noindent">
<p id="S1.p1.1" class="ltx_p">Large Language Models (LLMs) have greatly propelled significant advancements in Natural Language Processing (NLP), such as OpenAI GPT <cite class="ltx_cite ltx_citemacro_citep">(Brown et al., <a href="#bib.bib9" title="" class="ltx_ref">2020</a>)</cite>, Llama <cite class="ltx_cite ltx_citemacro_citep">(Touvron et al., <a href="#bib.bib39" title="" class="ltx_ref">2023a</a>; <a href="#bib.bib40" title="" class="ltx_ref">b</a>)</cite>, StarCoder <cite class="ltx_cite ltx_citemacro_citep">(Li et al., <a href="#bib.bib20" title="" class="ltx_ref">2023a</a>)</cite>, and others. These models perform well in many NLP tasks and claim to have made progress in advanced capabilities such as reasoning, long-context understanding, and so on. However, existing benchmarks <cite class="ltx_cite ltx_citemacro_citep">(Chang et al., <a href="#bib.bib10" title="" class="ltx_ref">2023</a>)</cite> often fail when it comes to evaluating extremely long-context LLMs or analysing the controllable characteristics and limitations of LLMs.</p>
</div>
<div id="S1.p2" class="ltx_para ltx_noindent">
<p id="S1.p2.1" class="ltx_p">For long-context understanding, previous work has often evaluated LLMs using the scope of language modeling metrics (i.e., perplexity) <cite class="ltx_cite ltx_citemacro_citep">(Sun et al., <a href="#bib.bib36" title="" class="ltx_ref">2021</a>; Peng et al., <a href="#bib.bib32" title="" class="ltx_ref">2023</a>)</cite> or the performance on simple artificial tasks <cite class="ltx_cite ltx_citemacro_citep">(Li &amp; Roth, <a href="#bib.bib21" title="" class="ltx_ref">2002</a>; Berant et al., <a href="#bib.bib6" title="" class="ltx_ref">2013</a>; Mohtashami &amp; Jaggi, <a href="#bib.bib26" title="" class="ltx_ref">2023</a>)</cite>.
However, these evaluation tasks tend to lack complexity and are narrowly focused on simple comprehension, which is misaligned with the sophistication required for real-world downstream applications.
While recent work has made great progress on building evaluation benchmarks at longer context lengths with real-world use cases (e.g, question answering) <cite class="ltx_cite ltx_citemacro_citep">(Bai et al., <a href="#bib.bib5" title="" class="ltx_ref">2023b</a>; An et al., <a href="#bib.bib1" title="" class="ltx_ref">2023</a>)</cite>, these manually annotated datasets often lack the scale and diversity to thoroughly assess performance on extended context lengths.
For example, existing benchmarks struggle to effectively evaluate LLMs that claim an ability to process contexts up to 100K tokens, due to the limited capacity of human annotation for very long text.
Developing more scalable and diverse evaluation datasets, potentially leveraging automated supervision, remains an open challenge.</p>
</div>
<div id="S1.p3" class="ltx_para ltx_noindent">
<p id="S1.p3.1" class="ltx_p">For reasoning analysis <cite class="ltx_cite ltx_citemacro_citep">(Hendrycks et al., <a href="#bib.bib15" title="" class="ltx_ref">2021b</a>; Chen et al., <a href="#bib.bib11" title="" class="ltx_ref">2021a</a>; Suzgun et al., <a href="#bib.bib37" title="" class="ltx_ref">2023</a>; Zhong et al., <a href="#bib.bib44" title="" class="ltx_ref">2023</a>)</cite>, conducting both qualitative and quantitative analysis of answers and reasoning processes provides important insights.
However, existing benchmarks lack the ability to precisely control the distribution of the dataset, limiting their utility for in-depth research analysis.
In other words, the nature of these benchmarks makes it challenging for developers to identify the specific weaknesses of their LLMs.
More configurable and granular benchmarks are needed to enable detailed analysis of model performance.
In addition, these benchmarks often draw their evaluation data from NLP tasks that have been extensively studied and are likely to be used in the training corpus of LLMs.
The potential data leakage makes the evaluation less convincing.</p>
</div>
<figure id="S1.F1" class="ltx_figure"><img src="/html/2310.15147/assets/x2.png" id="S1.F1.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="423" height="161" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 1: </span>The illustration demonstrates the <span id="S1.F1.2.1" class="ltx_text ltx_font_smallcaps">S3Eval</span> pipeline, where the capabilities of LLMs are assessed by evaluating their ability to execute SQL queries over randomly generated tables.</figcaption>
</figure>
<div id="S1.p4" class="ltx_para ltx_noindent">
<p id="S1.p4.1" class="ltx_p">In this paper, we propose a new evaluation suite called <span id="S1.p4.1.1" class="ltx_text ltx_font_smallcaps">S3Eval</span>, which addresses the aforementioned issues by using a complex synthetic task - SQL execution - as a proxy for the performance of LLMs on realistic reasoning tasks.
As shown in Figure <a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>, inspired by the work of <span id="S1.p4.1.2" class="ltx_text ltx_font_smallcaps">TaPEx</span> <cite class="ltx_cite ltx_citemacro_citep">(Liu et al., <a href="#bib.bib25" title="" class="ltx_ref">2022</a>)</cite>, <span id="S1.p4.1.3" class="ltx_text ltx_font_smallcaps">S3Eval</span> is based on the SQL execution task.
Specially, given a randomly generated table and a random SQL query, <span id="S1.p4.1.4" class="ltx_text ltx_font_smallcaps">S3Eval</span> evaluates whether LLMs can return the correct execution results.
<span id="S1.p4.1.5" class="ltx_text ltx_font_smallcaps">S3Eval</span> has three notable characteristics: (1) It is synthetic, with no table or SQL query present in the LLM training corpus. The tasks use complex, grammatically correct SQL syntax, making them very challenging.
(2) It is scalable, allowing users to customize the benchmark to any length and difficulty.
(3) It is systematic, containing diverse reasoning types and operations. This enables comprehensive evaluation of LLM capabilities.</p>
</div>
<div id="S1.p5" class="ltx_para ltx_noindent">
<p id="S1.p5.1" class="ltx_p">With these powerful features, developers can extend the context to really long lengths and generate meaningful SQL statements using <span id="S1.p5.1.1" class="ltx_text ltx_font_smallcaps">S3Eval</span>.
As demonstrated in Table <a href="#S2.T1" title="Table 1 ‣ 2 Synthetic Suite: Alignment with Realistic Benchmark ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>, developers can modify the configuration settings to conduct focused studies on particular reasoning abilities (e.g., aggregation) of LLMs.
We conducted comprehensive multi-perspective experiments on several popular LLMs using <span id="S1.p5.1.2" class="ltx_text ltx_font_smallcaps">S3Eval</span>.
Experimental results demonstrated that the performance of LLMs on <span id="S1.p5.1.3" class="ltx_text ltx_font_smallcaps">S3Eval</span> aligns closely with their performance on mainstream LLM benchmarks.
While LLMs have shown impressive capabilities, our work reveals limitations in their ability to leverage long contexts, since we observe performance degradation of almost all LLMs in long-context settings.
By carefully studying experimental results, we can work to pinpoint situations where LLMs tend to fail and summarize valuable insights. For example, LLMs often encounter challenges when the answer lies in the middle of the context, similar to findings from  <cite class="ltx_cite ltx_citemacro_citet">Liu et al. (<a href="#bib.bib24" title="" class="ltx_ref">2023b</a>)</cite>, or when the answers are sparsely distributed across the input. We believe that these observations from <span id="S1.p5.1.4" class="ltx_text ltx_font_smallcaps">S3Eval</span> provide valuable guidance for the development of LLMs and dynamic benchmarks.</p>
</div>
</section>
<section id="S2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">2 </span>Synthetic Suite: Alignment with Realistic Benchmark</h2>

<div id="S2.p1" class="ltx_para ltx_noindent">
<p id="S2.p1.1" class="ltx_p">In this section, we describe the details of synthesizing the evaluation data (Section <a href="#S2.SS1" title="2.1 Suite Construction ‣ 2 Synthetic Suite: Alignment with Realistic Benchmark ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2.1</span></a>) and verify the alignment between our synthetic suite <span id="S2.p1.1.1" class="ltx_text ltx_font_smallcaps">S3Eval</span> and real-world benchmark results.</p>
</div>
<figure id="S2.T1" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 1: </span>Our <span id="S2.T1.2.1" class="ltx_text ltx_font_smallcaps">S3Eval</span> method allows users to customize configuration settings and provides descriptions for each parameter that can be adjusted. More configurations can be found in Appendix <a href="#A4.SS1" title="D.1 Setting Description ‣ Appendix D Experiments Settings Details ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">D.1</span></a>.</figcaption>
<table id="S2.T1.3" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="S2.T1.3.1" class="ltx_tr">
<td id="S2.T1.3.1.1" class="ltx_td ltx_align_center ltx_border_tt" style="padding-left:5.7pt;padding-right:5.7pt;" colspan="2"><span id="S2.T1.3.1.1.1" class="ltx_text ltx_font_bold">Configuration</span></td>
<td id="S2.T1.3.1.2" class="ltx_td ltx_align_center ltx_border_tt" style="padding-left:5.7pt;padding-right:5.7pt;"><span id="S2.T1.3.1.2.1" class="ltx_text ltx_font_bold">Description</span></td>
</tr>
<tr id="S2.T1.3.2" class="ltx_tr">
<td id="S2.T1.3.2.1" class="ltx_td ltx_align_left ltx_border_t" style="padding-left:5.7pt;padding-right:5.7pt;" rowspan="5"><span id="S2.T1.3.2.1.1" class="ltx_text">Table Control</span></td>
<td id="S2.T1.3.2.2" class="ltx_td ltx_align_left ltx_border_t" style="padding-left:5.7pt;padding-right:5.7pt;"># of Rows</td>
<td id="S2.T1.3.2.3" class="ltx_td ltx_align_left ltx_border_t" style="padding-left:5.7pt;padding-right:5.7pt;">The number of rows in the generated tables</td>
</tr>
<tr id="S2.T1.3.3" class="ltx_tr">
<td id="S2.T1.3.3.1" class="ltx_td ltx_align_left" style="padding-left:5.7pt;padding-right:5.7pt;"># of Columns</td>
<td id="S2.T1.3.3.2" class="ltx_td ltx_align_left" style="padding-left:5.7pt;padding-right:5.7pt;">The number of columns in the generated tables</td>
</tr>
<tr id="S2.T1.3.4" class="ltx_tr">
<td id="S2.T1.3.4.1" class="ltx_td ltx_align_left" style="padding-left:5.7pt;padding-right:5.7pt;">Header Type Ratio</td>
<td id="S2.T1.3.4.2" class="ltx_td ltx_align_left" style="padding-left:5.7pt;padding-right:5.7pt;">The proportion of table column types that are TEXT, INT, DATE</td>
</tr>
<tr id="S2.T1.3.5" class="ltx_tr">
<td id="S2.T1.3.5.1" class="ltx_td ltx_align_left" style="padding-left:5.7pt;padding-right:5.7pt;">Cell Uniqueness</td>
<td id="S2.T1.3.5.2" class="ltx_td ltx_align_left" style="padding-left:5.7pt;padding-right:5.7pt;">The proportion of duplicate cells in each column</td>
</tr>
<tr id="S2.T1.3.6" class="ltx_tr">
<td id="S2.T1.3.6.1" class="ltx_td ltx_align_left" style="padding-left:5.7pt;padding-right:5.7pt;">String / Int Length</td>
<td id="S2.T1.3.6.2" class="ltx_td ltx_align_left" style="padding-left:5.7pt;padding-right:5.7pt;">The string length or numeric range of cell values</td>
</tr>
<tr id="S2.T1.3.7" class="ltx_tr">
<td id="S2.T1.3.7.1" class="ltx_td ltx_align_left ltx_border_t" style="padding-left:5.7pt;padding-right:5.7pt;" rowspan="8"><span id="S2.T1.3.7.1.1" class="ltx_text">
<span id="S2.T1.3.7.1.1.1" class="ltx_inline-block ltx_align_center">
<span id="S2.T1.3.7.1.1.1.1" class="ltx_p">Instruction</span>
<span id="S2.T1.3.7.1.1.1.2" class="ltx_p">Control</span>
</span></span></td>
<td id="S2.T1.3.7.2" class="ltx_td ltx_align_left ltx_border_t" style="padding-left:5.7pt;padding-right:5.7pt;">SQL Keywords</td>
<td id="S2.T1.3.7.3" class="ltx_td ltx_align_left ltx_border_t" style="padding-left:5.7pt;padding-right:5.7pt;">SELECT, WHERE, GROUP BY, HAVING, ORDER BY</td>
</tr>
<tr id="S2.T1.3.8" class="ltx_tr">
<td id="S2.T1.3.8.1" class="ltx_td ltx_align_left" style="padding-left:5.7pt;padding-right:5.7pt;">SQL Length</td>
<td id="S2.T1.3.8.2" class="ltx_td ltx_align_left" style="padding-left:5.7pt;padding-right:5.7pt;">The number of tokens after SQL split by space</td>
</tr>
<tr id="S2.T1.3.9" class="ltx_tr">
<td id="S2.T1.3.9.1" class="ltx_td ltx_align_left" style="padding-left:5.7pt;padding-right:5.7pt;">Column Coverage</td>
<td id="S2.T1.3.9.2" class="ltx_td ltx_align_left" style="padding-left:5.7pt;padding-right:5.7pt;">The ratio of columns involved in SQL execution to total columns.</td>
</tr>
<tr id="S2.T1.3.10" class="ltx_tr">
<td id="S2.T1.3.10.1" class="ltx_td ltx_align_left" style="padding-left:5.7pt;padding-right:5.7pt;">Row Coverage</td>
<td id="S2.T1.3.10.2" class="ltx_td ltx_align_left" style="padding-left:5.7pt;padding-right:5.7pt;">The ratio of rows involved in SQL execution to total rows</td>
</tr>
<tr id="S2.T1.3.11" class="ltx_tr">
<td id="S2.T1.3.11.1" class="ltx_td ltx_align_left" style="padding-left:5.7pt;padding-right:5.7pt;">Calculate Times</td>
<td id="S2.T1.3.11.2" class="ltx_td ltx_align_left" style="padding-left:5.7pt;padding-right:5.7pt;">The number of SQL numerical calculations.</td>
</tr>
<tr id="S2.T1.3.12" class="ltx_tr">
<td id="S2.T1.3.12.1" class="ltx_td ltx_align_left" style="padding-left:5.7pt;padding-right:5.7pt;">Filter Times</td>
<td id="S2.T1.3.12.2" class="ltx_td ltx_align_left" style="padding-left:5.7pt;padding-right:5.7pt;">The number of SQL filtering operations.</td>
</tr>
<tr id="S2.T1.3.13" class="ltx_tr">
<td id="S2.T1.3.13.1" class="ltx_td ltx_align_left" style="padding-left:5.7pt;padding-right:5.7pt;">Aggregator</td>
<td id="S2.T1.3.13.2" class="ltx_td ltx_align_left" style="padding-left:5.7pt;padding-right:5.7pt;">COUNT, MAX, MIN, SUM, AVG</td>
</tr>
<tr id="S2.T1.3.14" class="ltx_tr">
<td id="S2.T1.3.14.1" class="ltx_td ltx_align_left" style="padding-left:5.7pt;padding-right:5.7pt;">Filter Operator</td>
<td id="S2.T1.3.14.2" class="ltx_td ltx_align_left" style="padding-left:5.7pt;padding-right:5.7pt;">&gt;, &lt;, =, IN, LIKE</td>
</tr>
<tr id="S2.T1.3.15" class="ltx_tr">
<td id="S2.T1.3.15.1" class="ltx_td ltx_align_left ltx_border_bb ltx_border_t" style="padding-left:5.7pt;padding-right:5.7pt;" rowspan="3"><span id="S2.T1.3.15.1.1" class="ltx_text">Output Control</span></td>
<td id="S2.T1.3.15.2" class="ltx_td ltx_align_left ltx_border_t" style="padding-left:5.7pt;padding-right:5.7pt;">Answer Location</td>
<td id="S2.T1.3.15.3" class="ltx_td ltx_align_left ltx_border_t" style="padding-left:5.7pt;padding-right:5.7pt;">The location of SQL answers in the input table</td>
</tr>
<tr id="S2.T1.3.16" class="ltx_tr">
<td id="S2.T1.3.16.1" class="ltx_td ltx_align_left" style="padding-left:5.7pt;padding-right:5.7pt;"># of Answer Cells</td>
<td id="S2.T1.3.16.2" class="ltx_td ltx_align_left" style="padding-left:5.7pt;padding-right:5.7pt;">The number of selected cells in the answer</td>
</tr>
<tr id="S2.T1.3.17" class="ltx_tr">
<td id="S2.T1.3.17.1" class="ltx_td ltx_align_left ltx_border_bb" style="padding-left:5.7pt;padding-right:5.7pt;">Answer Length</td>
<td id="S2.T1.3.17.2" class="ltx_td ltx_align_left ltx_border_bb" style="padding-left:5.7pt;padding-right:5.7pt;">The total number of tokens in the answer</td>
</tr>
</table>
</figure>
<section id="S2.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.1 </span>Suite Construction</h3>

<section id="S2.SS1.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Task Formulation</h4>

<div id="S2.SS1.SSS0.Px1.p1" class="ltx_para ltx_noindent">
<p id="S2.SS1.SSS0.Px1.p1.18" class="ltx_p">Following previous work <cite class="ltx_cite ltx_citemacro_citep">(Liu et al., <a href="#bib.bib25" title="" class="ltx_ref">2022</a>)</cite>, each example in <span id="S2.SS1.SSS0.Px1.p1.18.1" class="ltx_text ltx_font_smallcaps">S3Eval</span> generally contain an SQL query and a (semi-)structured table <math id="S2.SS1.SSS0.Px1.p1.1.m1.1" class="ltx_Math" alttext="T" display="inline"><semantics id="S2.SS1.SSS0.Px1.p1.1.m1.1a"><mi id="S2.SS1.SSS0.Px1.p1.1.m1.1.1" xref="S2.SS1.SSS0.Px1.p1.1.m1.1.1.cmml">T</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.SSS0.Px1.p1.1.m1.1b"><ci id="S2.SS1.SSS0.Px1.p1.1.m1.1.1.cmml" xref="S2.SS1.SSS0.Px1.p1.1.m1.1.1">𝑇</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.SSS0.Px1.p1.1.m1.1c">T</annotation></semantics></math> as the input.
Each table <math id="S2.SS1.SSS0.Px1.p1.2.m2.1" class="ltx_Math" alttext="T" display="inline"><semantics id="S2.SS1.SSS0.Px1.p1.2.m2.1a"><mi id="S2.SS1.SSS0.Px1.p1.2.m2.1.1" xref="S2.SS1.SSS0.Px1.p1.2.m2.1.1.cmml">T</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.SSS0.Px1.p1.2.m2.1b"><ci id="S2.SS1.SSS0.Px1.p1.2.m2.1.1.cmml" xref="S2.SS1.SSS0.Px1.p1.2.m2.1.1">𝑇</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.SSS0.Px1.p1.2.m2.1c">T</annotation></semantics></math> consists of <math id="S2.SS1.SSS0.Px1.p1.3.m3.1" class="ltx_Math" alttext="M" display="inline"><semantics id="S2.SS1.SSS0.Px1.p1.3.m3.1a"><mi id="S2.SS1.SSS0.Px1.p1.3.m3.1.1" xref="S2.SS1.SSS0.Px1.p1.3.m3.1.1.cmml">M</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.SSS0.Px1.p1.3.m3.1b"><ci id="S2.SS1.SSS0.Px1.p1.3.m3.1.1.cmml" xref="S2.SS1.SSS0.Px1.p1.3.m3.1.1">𝑀</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.SSS0.Px1.p1.3.m3.1c">M</annotation></semantics></math> rows <math id="S2.SS1.SSS0.Px1.p1.4.m4.1" class="ltx_Math" alttext="\left\{r_{i}\right\}_{i=1}^{M}" display="inline"><semantics id="S2.SS1.SSS0.Px1.p1.4.m4.1a"><msubsup id="S2.SS1.SSS0.Px1.p1.4.m4.1.1" xref="S2.SS1.SSS0.Px1.p1.4.m4.1.1.cmml"><mrow id="S2.SS1.SSS0.Px1.p1.4.m4.1.1.1.1.1" xref="S2.SS1.SSS0.Px1.p1.4.m4.1.1.1.1.2.cmml"><mo id="S2.SS1.SSS0.Px1.p1.4.m4.1.1.1.1.1.2" xref="S2.SS1.SSS0.Px1.p1.4.m4.1.1.1.1.2.cmml">{</mo><msub id="S2.SS1.SSS0.Px1.p1.4.m4.1.1.1.1.1.1" xref="S2.SS1.SSS0.Px1.p1.4.m4.1.1.1.1.1.1.cmml"><mi id="S2.SS1.SSS0.Px1.p1.4.m4.1.1.1.1.1.1.2" xref="S2.SS1.SSS0.Px1.p1.4.m4.1.1.1.1.1.1.2.cmml">r</mi><mi id="S2.SS1.SSS0.Px1.p1.4.m4.1.1.1.1.1.1.3" xref="S2.SS1.SSS0.Px1.p1.4.m4.1.1.1.1.1.1.3.cmml">i</mi></msub><mo id="S2.SS1.SSS0.Px1.p1.4.m4.1.1.1.1.1.3" xref="S2.SS1.SSS0.Px1.p1.4.m4.1.1.1.1.2.cmml">}</mo></mrow><mrow id="S2.SS1.SSS0.Px1.p1.4.m4.1.1.1.3" xref="S2.SS1.SSS0.Px1.p1.4.m4.1.1.1.3.cmml"><mi id="S2.SS1.SSS0.Px1.p1.4.m4.1.1.1.3.2" xref="S2.SS1.SSS0.Px1.p1.4.m4.1.1.1.3.2.cmml">i</mi><mo id="S2.SS1.SSS0.Px1.p1.4.m4.1.1.1.3.1" xref="S2.SS1.SSS0.Px1.p1.4.m4.1.1.1.3.1.cmml">=</mo><mn id="S2.SS1.SSS0.Px1.p1.4.m4.1.1.1.3.3" xref="S2.SS1.SSS0.Px1.p1.4.m4.1.1.1.3.3.cmml">1</mn></mrow><mi id="S2.SS1.SSS0.Px1.p1.4.m4.1.1.3" xref="S2.SS1.SSS0.Px1.p1.4.m4.1.1.3.cmml">M</mi></msubsup><annotation-xml encoding="MathML-Content" id="S2.SS1.SSS0.Px1.p1.4.m4.1b"><apply id="S2.SS1.SSS0.Px1.p1.4.m4.1.1.cmml" xref="S2.SS1.SSS0.Px1.p1.4.m4.1.1"><csymbol cd="ambiguous" id="S2.SS1.SSS0.Px1.p1.4.m4.1.1.2.cmml" xref="S2.SS1.SSS0.Px1.p1.4.m4.1.1">superscript</csymbol><apply id="S2.SS1.SSS0.Px1.p1.4.m4.1.1.1.cmml" xref="S2.SS1.SSS0.Px1.p1.4.m4.1.1"><csymbol cd="ambiguous" id="S2.SS1.SSS0.Px1.p1.4.m4.1.1.1.2.cmml" xref="S2.SS1.SSS0.Px1.p1.4.m4.1.1">subscript</csymbol><set id="S2.SS1.SSS0.Px1.p1.4.m4.1.1.1.1.2.cmml" xref="S2.SS1.SSS0.Px1.p1.4.m4.1.1.1.1.1"><apply id="S2.SS1.SSS0.Px1.p1.4.m4.1.1.1.1.1.1.cmml" xref="S2.SS1.SSS0.Px1.p1.4.m4.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS1.SSS0.Px1.p1.4.m4.1.1.1.1.1.1.1.cmml" xref="S2.SS1.SSS0.Px1.p1.4.m4.1.1.1.1.1.1">subscript</csymbol><ci id="S2.SS1.SSS0.Px1.p1.4.m4.1.1.1.1.1.1.2.cmml" xref="S2.SS1.SSS0.Px1.p1.4.m4.1.1.1.1.1.1.2">𝑟</ci><ci id="S2.SS1.SSS0.Px1.p1.4.m4.1.1.1.1.1.1.3.cmml" xref="S2.SS1.SSS0.Px1.p1.4.m4.1.1.1.1.1.1.3">𝑖</ci></apply></set><apply id="S2.SS1.SSS0.Px1.p1.4.m4.1.1.1.3.cmml" xref="S2.SS1.SSS0.Px1.p1.4.m4.1.1.1.3"><eq id="S2.SS1.SSS0.Px1.p1.4.m4.1.1.1.3.1.cmml" xref="S2.SS1.SSS0.Px1.p1.4.m4.1.1.1.3.1"></eq><ci id="S2.SS1.SSS0.Px1.p1.4.m4.1.1.1.3.2.cmml" xref="S2.SS1.SSS0.Px1.p1.4.m4.1.1.1.3.2">𝑖</ci><cn type="integer" id="S2.SS1.SSS0.Px1.p1.4.m4.1.1.1.3.3.cmml" xref="S2.SS1.SSS0.Px1.p1.4.m4.1.1.1.3.3">1</cn></apply></apply><ci id="S2.SS1.SSS0.Px1.p1.4.m4.1.1.3.cmml" xref="S2.SS1.SSS0.Px1.p1.4.m4.1.1.3">𝑀</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.SSS0.Px1.p1.4.m4.1c">\left\{r_{i}\right\}_{i=1}^{M}</annotation></semantics></math>, in which each row <math id="S2.SS1.SSS0.Px1.p1.5.m5.1" class="ltx_Math" alttext="r_{i}" display="inline"><semantics id="S2.SS1.SSS0.Px1.p1.5.m5.1a"><msub id="S2.SS1.SSS0.Px1.p1.5.m5.1.1" xref="S2.SS1.SSS0.Px1.p1.5.m5.1.1.cmml"><mi id="S2.SS1.SSS0.Px1.p1.5.m5.1.1.2" xref="S2.SS1.SSS0.Px1.p1.5.m5.1.1.2.cmml">r</mi><mi id="S2.SS1.SSS0.Px1.p1.5.m5.1.1.3" xref="S2.SS1.SSS0.Px1.p1.5.m5.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS1.SSS0.Px1.p1.5.m5.1b"><apply id="S2.SS1.SSS0.Px1.p1.5.m5.1.1.cmml" xref="S2.SS1.SSS0.Px1.p1.5.m5.1.1"><csymbol cd="ambiguous" id="S2.SS1.SSS0.Px1.p1.5.m5.1.1.1.cmml" xref="S2.SS1.SSS0.Px1.p1.5.m5.1.1">subscript</csymbol><ci id="S2.SS1.SSS0.Px1.p1.5.m5.1.1.2.cmml" xref="S2.SS1.SSS0.Px1.p1.5.m5.1.1.2">𝑟</ci><ci id="S2.SS1.SSS0.Px1.p1.5.m5.1.1.3.cmml" xref="S2.SS1.SSS0.Px1.p1.5.m5.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.SSS0.Px1.p1.5.m5.1c">r_{i}</annotation></semantics></math> contains <math id="S2.SS1.SSS0.Px1.p1.6.m6.1" class="ltx_Math" alttext="N" display="inline"><semantics id="S2.SS1.SSS0.Px1.p1.6.m6.1a"><mi id="S2.SS1.SSS0.Px1.p1.6.m6.1.1" xref="S2.SS1.SSS0.Px1.p1.6.m6.1.1.cmml">N</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.SSS0.Px1.p1.6.m6.1b"><ci id="S2.SS1.SSS0.Px1.p1.6.m6.1.1.cmml" xref="S2.SS1.SSS0.Px1.p1.6.m6.1.1">𝑁</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.SSS0.Px1.p1.6.m6.1c">N</annotation></semantics></math> cell values <math id="S2.SS1.SSS0.Px1.p1.7.m7.3" class="ltx_Math" alttext="\left\{c_{\left\langle i,j\right\rangle}\right\}_{i=1}^{M}" display="inline"><semantics id="S2.SS1.SSS0.Px1.p1.7.m7.3a"><msubsup id="S2.SS1.SSS0.Px1.p1.7.m7.3.3" xref="S2.SS1.SSS0.Px1.p1.7.m7.3.3.cmml"><mrow id="S2.SS1.SSS0.Px1.p1.7.m7.3.3.1.1.1" xref="S2.SS1.SSS0.Px1.p1.7.m7.3.3.1.1.2.cmml"><mo id="S2.SS1.SSS0.Px1.p1.7.m7.3.3.1.1.1.2" xref="S2.SS1.SSS0.Px1.p1.7.m7.3.3.1.1.2.cmml">{</mo><msub id="S2.SS1.SSS0.Px1.p1.7.m7.3.3.1.1.1.1" xref="S2.SS1.SSS0.Px1.p1.7.m7.3.3.1.1.1.1.cmml"><mi id="S2.SS1.SSS0.Px1.p1.7.m7.3.3.1.1.1.1.2" xref="S2.SS1.SSS0.Px1.p1.7.m7.3.3.1.1.1.1.2.cmml">c</mi><mrow id="S2.SS1.SSS0.Px1.p1.7.m7.2.2.2.4" xref="S2.SS1.SSS0.Px1.p1.7.m7.2.2.2.3.cmml"><mo id="S2.SS1.SSS0.Px1.p1.7.m7.2.2.2.4.1" xref="S2.SS1.SSS0.Px1.p1.7.m7.2.2.2.3.cmml">⟨</mo><mi id="S2.SS1.SSS0.Px1.p1.7.m7.1.1.1.1" xref="S2.SS1.SSS0.Px1.p1.7.m7.1.1.1.1.cmml">i</mi><mo id="S2.SS1.SSS0.Px1.p1.7.m7.2.2.2.4.2" xref="S2.SS1.SSS0.Px1.p1.7.m7.2.2.2.3.cmml">,</mo><mi id="S2.SS1.SSS0.Px1.p1.7.m7.2.2.2.2" xref="S2.SS1.SSS0.Px1.p1.7.m7.2.2.2.2.cmml">j</mi><mo id="S2.SS1.SSS0.Px1.p1.7.m7.2.2.2.4.3" xref="S2.SS1.SSS0.Px1.p1.7.m7.2.2.2.3.cmml">⟩</mo></mrow></msub><mo id="S2.SS1.SSS0.Px1.p1.7.m7.3.3.1.1.1.3" xref="S2.SS1.SSS0.Px1.p1.7.m7.3.3.1.1.2.cmml">}</mo></mrow><mrow id="S2.SS1.SSS0.Px1.p1.7.m7.3.3.1.3" xref="S2.SS1.SSS0.Px1.p1.7.m7.3.3.1.3.cmml"><mi id="S2.SS1.SSS0.Px1.p1.7.m7.3.3.1.3.2" xref="S2.SS1.SSS0.Px1.p1.7.m7.3.3.1.3.2.cmml">i</mi><mo id="S2.SS1.SSS0.Px1.p1.7.m7.3.3.1.3.1" xref="S2.SS1.SSS0.Px1.p1.7.m7.3.3.1.3.1.cmml">=</mo><mn id="S2.SS1.SSS0.Px1.p1.7.m7.3.3.1.3.3" xref="S2.SS1.SSS0.Px1.p1.7.m7.3.3.1.3.3.cmml">1</mn></mrow><mi id="S2.SS1.SSS0.Px1.p1.7.m7.3.3.3" xref="S2.SS1.SSS0.Px1.p1.7.m7.3.3.3.cmml">M</mi></msubsup><annotation-xml encoding="MathML-Content" id="S2.SS1.SSS0.Px1.p1.7.m7.3b"><apply id="S2.SS1.SSS0.Px1.p1.7.m7.3.3.cmml" xref="S2.SS1.SSS0.Px1.p1.7.m7.3.3"><csymbol cd="ambiguous" id="S2.SS1.SSS0.Px1.p1.7.m7.3.3.2.cmml" xref="S2.SS1.SSS0.Px1.p1.7.m7.3.3">superscript</csymbol><apply id="S2.SS1.SSS0.Px1.p1.7.m7.3.3.1.cmml" xref="S2.SS1.SSS0.Px1.p1.7.m7.3.3"><csymbol cd="ambiguous" id="S2.SS1.SSS0.Px1.p1.7.m7.3.3.1.2.cmml" xref="S2.SS1.SSS0.Px1.p1.7.m7.3.3">subscript</csymbol><set id="S2.SS1.SSS0.Px1.p1.7.m7.3.3.1.1.2.cmml" xref="S2.SS1.SSS0.Px1.p1.7.m7.3.3.1.1.1"><apply id="S2.SS1.SSS0.Px1.p1.7.m7.3.3.1.1.1.1.cmml" xref="S2.SS1.SSS0.Px1.p1.7.m7.3.3.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS1.SSS0.Px1.p1.7.m7.3.3.1.1.1.1.1.cmml" xref="S2.SS1.SSS0.Px1.p1.7.m7.3.3.1.1.1.1">subscript</csymbol><ci id="S2.SS1.SSS0.Px1.p1.7.m7.3.3.1.1.1.1.2.cmml" xref="S2.SS1.SSS0.Px1.p1.7.m7.3.3.1.1.1.1.2">𝑐</ci><list id="S2.SS1.SSS0.Px1.p1.7.m7.2.2.2.3.cmml" xref="S2.SS1.SSS0.Px1.p1.7.m7.2.2.2.4"><ci id="S2.SS1.SSS0.Px1.p1.7.m7.1.1.1.1.cmml" xref="S2.SS1.SSS0.Px1.p1.7.m7.1.1.1.1">𝑖</ci><ci id="S2.SS1.SSS0.Px1.p1.7.m7.2.2.2.2.cmml" xref="S2.SS1.SSS0.Px1.p1.7.m7.2.2.2.2">𝑗</ci></list></apply></set><apply id="S2.SS1.SSS0.Px1.p1.7.m7.3.3.1.3.cmml" xref="S2.SS1.SSS0.Px1.p1.7.m7.3.3.1.3"><eq id="S2.SS1.SSS0.Px1.p1.7.m7.3.3.1.3.1.cmml" xref="S2.SS1.SSS0.Px1.p1.7.m7.3.3.1.3.1"></eq><ci id="S2.SS1.SSS0.Px1.p1.7.m7.3.3.1.3.2.cmml" xref="S2.SS1.SSS0.Px1.p1.7.m7.3.3.1.3.2">𝑖</ci><cn type="integer" id="S2.SS1.SSS0.Px1.p1.7.m7.3.3.1.3.3.cmml" xref="S2.SS1.SSS0.Px1.p1.7.m7.3.3.1.3.3">1</cn></apply></apply><ci id="S2.SS1.SSS0.Px1.p1.7.m7.3.3.3.cmml" xref="S2.SS1.SSS0.Px1.p1.7.m7.3.3.3">𝑀</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.SSS0.Px1.p1.7.m7.3c">\left\{c_{\left\langle i,j\right\rangle}\right\}_{i=1}^{M}</annotation></semantics></math>. Each cell <math id="S2.SS1.SSS0.Px1.p1.8.m8.2" class="ltx_Math" alttext="c_{\left\langle i,j\right\rangle}" display="inline"><semantics id="S2.SS1.SSS0.Px1.p1.8.m8.2a"><msub id="S2.SS1.SSS0.Px1.p1.8.m8.2.3" xref="S2.SS1.SSS0.Px1.p1.8.m8.2.3.cmml"><mi id="S2.SS1.SSS0.Px1.p1.8.m8.2.3.2" xref="S2.SS1.SSS0.Px1.p1.8.m8.2.3.2.cmml">c</mi><mrow id="S2.SS1.SSS0.Px1.p1.8.m8.2.2.2.4" xref="S2.SS1.SSS0.Px1.p1.8.m8.2.2.2.3.cmml"><mo id="S2.SS1.SSS0.Px1.p1.8.m8.2.2.2.4.1" xref="S2.SS1.SSS0.Px1.p1.8.m8.2.2.2.3.cmml">⟨</mo><mi id="S2.SS1.SSS0.Px1.p1.8.m8.1.1.1.1" xref="S2.SS1.SSS0.Px1.p1.8.m8.1.1.1.1.cmml">i</mi><mo id="S2.SS1.SSS0.Px1.p1.8.m8.2.2.2.4.2" xref="S2.SS1.SSS0.Px1.p1.8.m8.2.2.2.3.cmml">,</mo><mi id="S2.SS1.SSS0.Px1.p1.8.m8.2.2.2.2" xref="S2.SS1.SSS0.Px1.p1.8.m8.2.2.2.2.cmml">j</mi><mo id="S2.SS1.SSS0.Px1.p1.8.m8.2.2.2.4.3" xref="S2.SS1.SSS0.Px1.p1.8.m8.2.2.2.3.cmml">⟩</mo></mrow></msub><annotation-xml encoding="MathML-Content" id="S2.SS1.SSS0.Px1.p1.8.m8.2b"><apply id="S2.SS1.SSS0.Px1.p1.8.m8.2.3.cmml" xref="S2.SS1.SSS0.Px1.p1.8.m8.2.3"><csymbol cd="ambiguous" id="S2.SS1.SSS0.Px1.p1.8.m8.2.3.1.cmml" xref="S2.SS1.SSS0.Px1.p1.8.m8.2.3">subscript</csymbol><ci id="S2.SS1.SSS0.Px1.p1.8.m8.2.3.2.cmml" xref="S2.SS1.SSS0.Px1.p1.8.m8.2.3.2">𝑐</ci><list id="S2.SS1.SSS0.Px1.p1.8.m8.2.2.2.3.cmml" xref="S2.SS1.SSS0.Px1.p1.8.m8.2.2.2.4"><ci id="S2.SS1.SSS0.Px1.p1.8.m8.1.1.1.1.cmml" xref="S2.SS1.SSS0.Px1.p1.8.m8.1.1.1.1">𝑖</ci><ci id="S2.SS1.SSS0.Px1.p1.8.m8.2.2.2.2.cmml" xref="S2.SS1.SSS0.Px1.p1.8.m8.2.2.2.2">𝑗</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.SSS0.Px1.p1.8.m8.2c">c_{\left\langle i,j\right\rangle}</annotation></semantics></math> corresponds to a table header <math id="S2.SS1.SSS0.Px1.p1.9.m9.1" class="ltx_Math" alttext="h_{j}" display="inline"><semantics id="S2.SS1.SSS0.Px1.p1.9.m9.1a"><msub id="S2.SS1.SSS0.Px1.p1.9.m9.1.1" xref="S2.SS1.SSS0.Px1.p1.9.m9.1.1.cmml"><mi id="S2.SS1.SSS0.Px1.p1.9.m9.1.1.2" xref="S2.SS1.SSS0.Px1.p1.9.m9.1.1.2.cmml">h</mi><mi id="S2.SS1.SSS0.Px1.p1.9.m9.1.1.3" xref="S2.SS1.SSS0.Px1.p1.9.m9.1.1.3.cmml">j</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS1.SSS0.Px1.p1.9.m9.1b"><apply id="S2.SS1.SSS0.Px1.p1.9.m9.1.1.cmml" xref="S2.SS1.SSS0.Px1.p1.9.m9.1.1"><csymbol cd="ambiguous" id="S2.SS1.SSS0.Px1.p1.9.m9.1.1.1.cmml" xref="S2.SS1.SSS0.Px1.p1.9.m9.1.1">subscript</csymbol><ci id="S2.SS1.SSS0.Px1.p1.9.m9.1.1.2.cmml" xref="S2.SS1.SSS0.Px1.p1.9.m9.1.1.2">ℎ</ci><ci id="S2.SS1.SSS0.Px1.p1.9.m9.1.1.3.cmml" xref="S2.SS1.SSS0.Px1.p1.9.m9.1.1.3">𝑗</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.SSS0.Px1.p1.9.m9.1c">h_{j}</annotation></semantics></math>. Each SQL query consists of <math id="S2.SS1.SSS0.Px1.p1.10.m10.1" class="ltx_Math" alttext="K" display="inline"><semantics id="S2.SS1.SSS0.Px1.p1.10.m10.1a"><mi id="S2.SS1.SSS0.Px1.p1.10.m10.1.1" xref="S2.SS1.SSS0.Px1.p1.10.m10.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.SSS0.Px1.p1.10.m10.1b"><ci id="S2.SS1.SSS0.Px1.p1.10.m10.1.1.cmml" xref="S2.SS1.SSS0.Px1.p1.10.m10.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.SSS0.Px1.p1.10.m10.1c">K</annotation></semantics></math> tokens as <math id="S2.SS1.SSS0.Px1.p1.11.m11.4" class="ltx_Math" alttext="x=x_{1},x_{2},\cdots,x_{K}" display="inline"><semantics id="S2.SS1.SSS0.Px1.p1.11.m11.4a"><mrow id="S2.SS1.SSS0.Px1.p1.11.m11.4.4" xref="S2.SS1.SSS0.Px1.p1.11.m11.4.4.cmml"><mi id="S2.SS1.SSS0.Px1.p1.11.m11.4.4.5" xref="S2.SS1.SSS0.Px1.p1.11.m11.4.4.5.cmml">x</mi><mo id="S2.SS1.SSS0.Px1.p1.11.m11.4.4.4" xref="S2.SS1.SSS0.Px1.p1.11.m11.4.4.4.cmml">=</mo><mrow id="S2.SS1.SSS0.Px1.p1.11.m11.4.4.3.3" xref="S2.SS1.SSS0.Px1.p1.11.m11.4.4.3.4.cmml"><msub id="S2.SS1.SSS0.Px1.p1.11.m11.2.2.1.1.1" xref="S2.SS1.SSS0.Px1.p1.11.m11.2.2.1.1.1.cmml"><mi id="S2.SS1.SSS0.Px1.p1.11.m11.2.2.1.1.1.2" xref="S2.SS1.SSS0.Px1.p1.11.m11.2.2.1.1.1.2.cmml">x</mi><mn id="S2.SS1.SSS0.Px1.p1.11.m11.2.2.1.1.1.3" xref="S2.SS1.SSS0.Px1.p1.11.m11.2.2.1.1.1.3.cmml">1</mn></msub><mo id="S2.SS1.SSS0.Px1.p1.11.m11.4.4.3.3.4" xref="S2.SS1.SSS0.Px1.p1.11.m11.4.4.3.4.cmml">,</mo><msub id="S2.SS1.SSS0.Px1.p1.11.m11.3.3.2.2.2" xref="S2.SS1.SSS0.Px1.p1.11.m11.3.3.2.2.2.cmml"><mi id="S2.SS1.SSS0.Px1.p1.11.m11.3.3.2.2.2.2" xref="S2.SS1.SSS0.Px1.p1.11.m11.3.3.2.2.2.2.cmml">x</mi><mn id="S2.SS1.SSS0.Px1.p1.11.m11.3.3.2.2.2.3" xref="S2.SS1.SSS0.Px1.p1.11.m11.3.3.2.2.2.3.cmml">2</mn></msub><mo id="S2.SS1.SSS0.Px1.p1.11.m11.4.4.3.3.5" xref="S2.SS1.SSS0.Px1.p1.11.m11.4.4.3.4.cmml">,</mo><mi mathvariant="normal" id="S2.SS1.SSS0.Px1.p1.11.m11.1.1" xref="S2.SS1.SSS0.Px1.p1.11.m11.1.1.cmml">⋯</mi><mo id="S2.SS1.SSS0.Px1.p1.11.m11.4.4.3.3.6" xref="S2.SS1.SSS0.Px1.p1.11.m11.4.4.3.4.cmml">,</mo><msub id="S2.SS1.SSS0.Px1.p1.11.m11.4.4.3.3.3" xref="S2.SS1.SSS0.Px1.p1.11.m11.4.4.3.3.3.cmml"><mi id="S2.SS1.SSS0.Px1.p1.11.m11.4.4.3.3.3.2" xref="S2.SS1.SSS0.Px1.p1.11.m11.4.4.3.3.3.2.cmml">x</mi><mi id="S2.SS1.SSS0.Px1.p1.11.m11.4.4.3.3.3.3" xref="S2.SS1.SSS0.Px1.p1.11.m11.4.4.3.3.3.3.cmml">K</mi></msub></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS1.SSS0.Px1.p1.11.m11.4b"><apply id="S2.SS1.SSS0.Px1.p1.11.m11.4.4.cmml" xref="S2.SS1.SSS0.Px1.p1.11.m11.4.4"><eq id="S2.SS1.SSS0.Px1.p1.11.m11.4.4.4.cmml" xref="S2.SS1.SSS0.Px1.p1.11.m11.4.4.4"></eq><ci id="S2.SS1.SSS0.Px1.p1.11.m11.4.4.5.cmml" xref="S2.SS1.SSS0.Px1.p1.11.m11.4.4.5">𝑥</ci><list id="S2.SS1.SSS0.Px1.p1.11.m11.4.4.3.4.cmml" xref="S2.SS1.SSS0.Px1.p1.11.m11.4.4.3.3"><apply id="S2.SS1.SSS0.Px1.p1.11.m11.2.2.1.1.1.cmml" xref="S2.SS1.SSS0.Px1.p1.11.m11.2.2.1.1.1"><csymbol cd="ambiguous" id="S2.SS1.SSS0.Px1.p1.11.m11.2.2.1.1.1.1.cmml" xref="S2.SS1.SSS0.Px1.p1.11.m11.2.2.1.1.1">subscript</csymbol><ci id="S2.SS1.SSS0.Px1.p1.11.m11.2.2.1.1.1.2.cmml" xref="S2.SS1.SSS0.Px1.p1.11.m11.2.2.1.1.1.2">𝑥</ci><cn type="integer" id="S2.SS1.SSS0.Px1.p1.11.m11.2.2.1.1.1.3.cmml" xref="S2.SS1.SSS0.Px1.p1.11.m11.2.2.1.1.1.3">1</cn></apply><apply id="S2.SS1.SSS0.Px1.p1.11.m11.3.3.2.2.2.cmml" xref="S2.SS1.SSS0.Px1.p1.11.m11.3.3.2.2.2"><csymbol cd="ambiguous" id="S2.SS1.SSS0.Px1.p1.11.m11.3.3.2.2.2.1.cmml" xref="S2.SS1.SSS0.Px1.p1.11.m11.3.3.2.2.2">subscript</csymbol><ci id="S2.SS1.SSS0.Px1.p1.11.m11.3.3.2.2.2.2.cmml" xref="S2.SS1.SSS0.Px1.p1.11.m11.3.3.2.2.2.2">𝑥</ci><cn type="integer" id="S2.SS1.SSS0.Px1.p1.11.m11.3.3.2.2.2.3.cmml" xref="S2.SS1.SSS0.Px1.p1.11.m11.3.3.2.2.2.3">2</cn></apply><ci id="S2.SS1.SSS0.Px1.p1.11.m11.1.1.cmml" xref="S2.SS1.SSS0.Px1.p1.11.m11.1.1">⋯</ci><apply id="S2.SS1.SSS0.Px1.p1.11.m11.4.4.3.3.3.cmml" xref="S2.SS1.SSS0.Px1.p1.11.m11.4.4.3.3.3"><csymbol cd="ambiguous" id="S2.SS1.SSS0.Px1.p1.11.m11.4.4.3.3.3.1.cmml" xref="S2.SS1.SSS0.Px1.p1.11.m11.4.4.3.3.3">subscript</csymbol><ci id="S2.SS1.SSS0.Px1.p1.11.m11.4.4.3.3.3.2.cmml" xref="S2.SS1.SSS0.Px1.p1.11.m11.4.4.3.3.3.2">𝑥</ci><ci id="S2.SS1.SSS0.Px1.p1.11.m11.4.4.3.3.3.3.cmml" xref="S2.SS1.SSS0.Px1.p1.11.m11.4.4.3.3.3.3">𝐾</ci></apply></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.SSS0.Px1.p1.11.m11.4c">x=x_{1},x_{2},\cdots,x_{K}</annotation></semantics></math>. Each token <math id="S2.SS1.SSS0.Px1.p1.12.m12.1" class="ltx_Math" alttext="x_{i}" display="inline"><semantics id="S2.SS1.SSS0.Px1.p1.12.m12.1a"><msub id="S2.SS1.SSS0.Px1.p1.12.m12.1.1" xref="S2.SS1.SSS0.Px1.p1.12.m12.1.1.cmml"><mi id="S2.SS1.SSS0.Px1.p1.12.m12.1.1.2" xref="S2.SS1.SSS0.Px1.p1.12.m12.1.1.2.cmml">x</mi><mi id="S2.SS1.SSS0.Px1.p1.12.m12.1.1.3" xref="S2.SS1.SSS0.Px1.p1.12.m12.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS1.SSS0.Px1.p1.12.m12.1b"><apply id="S2.SS1.SSS0.Px1.p1.12.m12.1.1.cmml" xref="S2.SS1.SSS0.Px1.p1.12.m12.1.1"><csymbol cd="ambiguous" id="S2.SS1.SSS0.Px1.p1.12.m12.1.1.1.cmml" xref="S2.SS1.SSS0.Px1.p1.12.m12.1.1">subscript</csymbol><ci id="S2.SS1.SSS0.Px1.p1.12.m12.1.1.2.cmml" xref="S2.SS1.SSS0.Px1.p1.12.m12.1.1.2">𝑥</ci><ci id="S2.SS1.SSS0.Px1.p1.12.m12.1.1.3.cmml" xref="S2.SS1.SSS0.Px1.p1.12.m12.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.SSS0.Px1.p1.12.m12.1c">x_{i}</annotation></semantics></math> originates from SQL keywords <math id="S2.SS1.SSS0.Px1.p1.13.m13.1" class="ltx_Math" alttext="K" display="inline"><semantics id="S2.SS1.SSS0.Px1.p1.13.m13.1a"><mi id="S2.SS1.SSS0.Px1.p1.13.m13.1.1" xref="S2.SS1.SSS0.Px1.p1.13.m13.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.SSS0.Px1.p1.13.m13.1b"><ci id="S2.SS1.SSS0.Px1.p1.13.m13.1.1.cmml" xref="S2.SS1.SSS0.Px1.p1.13.m13.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.SSS0.Px1.p1.13.m13.1c">K</annotation></semantics></math>, table schema <math id="S2.SS1.SSS0.Px1.p1.14.m14.1" class="ltx_Math" alttext="H" display="inline"><semantics id="S2.SS1.SSS0.Px1.p1.14.m14.1a"><mi id="S2.SS1.SSS0.Px1.p1.14.m14.1.1" xref="S2.SS1.SSS0.Px1.p1.14.m14.1.1.cmml">H</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.SSS0.Px1.p1.14.m14.1b"><ci id="S2.SS1.SSS0.Px1.p1.14.m14.1.1.cmml" xref="S2.SS1.SSS0.Px1.p1.14.m14.1.1">𝐻</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.SSS0.Px1.p1.14.m14.1c">H</annotation></semantics></math>, or table cells <math id="S2.SS1.SSS0.Px1.p1.15.m15.1" class="ltx_Math" alttext="C" display="inline"><semantics id="S2.SS1.SSS0.Px1.p1.15.m15.1a"><mi id="S2.SS1.SSS0.Px1.p1.15.m15.1.1" xref="S2.SS1.SSS0.Px1.p1.15.m15.1.1.cmml">C</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.SSS0.Px1.p1.15.m15.1b"><ci id="S2.SS1.SSS0.Px1.p1.15.m15.1.1.cmml" xref="S2.SS1.SSS0.Px1.p1.15.m15.1.1">𝐶</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.SSS0.Px1.p1.15.m15.1c">C</annotation></semantics></math>. Each multi-step instruction is transformed from SQL query. The task prompts LLM <math id="S2.SS1.SSS0.Px1.p1.16.m16.1" class="ltx_Math" alttext="M" display="inline"><semantics id="S2.SS1.SSS0.Px1.p1.16.m16.1a"><mi id="S2.SS1.SSS0.Px1.p1.16.m16.1.1" xref="S2.SS1.SSS0.Px1.p1.16.m16.1.1.cmml">M</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.SSS0.Px1.p1.16.m16.1b"><ci id="S2.SS1.SSS0.Px1.p1.16.m16.1.1.cmml" xref="S2.SS1.SSS0.Px1.p1.16.m16.1.1">𝑀</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.SSS0.Px1.p1.16.m16.1c">M</annotation></semantics></math> to obtain the execution result <math id="S2.SS1.SSS0.Px1.p1.17.m17.1" class="ltx_Math" alttext="A" display="inline"><semantics id="S2.SS1.SSS0.Px1.p1.17.m17.1a"><mi id="S2.SS1.SSS0.Px1.p1.17.m17.1.1" xref="S2.SS1.SSS0.Px1.p1.17.m17.1.1.cmml">A</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.SSS0.Px1.p1.17.m17.1b"><ci id="S2.SS1.SSS0.Px1.p1.17.m17.1.1.cmml" xref="S2.SS1.SSS0.Px1.p1.17.m17.1.1">𝐴</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.SSS0.Px1.p1.17.m17.1c">A</annotation></semantics></math> of the SQL on the table <math id="S2.SS1.SSS0.Px1.p1.18.m18.1" class="ltx_Math" alttext="T" display="inline"><semantics id="S2.SS1.SSS0.Px1.p1.18.m18.1a"><mi id="S2.SS1.SSS0.Px1.p1.18.m18.1.1" xref="S2.SS1.SSS0.Px1.p1.18.m18.1.1.cmml">T</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.SSS0.Px1.p1.18.m18.1b"><ci id="S2.SS1.SSS0.Px1.p1.18.m18.1.1.cmml" xref="S2.SS1.SSS0.Px1.p1.18.m18.1.1">𝑇</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.SSS0.Px1.p1.18.m18.1c">T</annotation></semantics></math>. Our main focus is on analyzing the accuracy of LLM in executing SQL queries.</p>
</div>
</section>
<section id="S2.SS1.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Random Table Generation</h4>

<div id="S2.SS1.SSS0.Px2.p1" class="ltx_para ltx_noindent">
<p id="S2.SS1.SSS0.Px2.p1.4" class="ltx_p">All tables in <span id="S2.SS1.SSS0.Px2.p1.4.1" class="ltx_text ltx_font_smallcaps">S3Eval</span> are randomly generated and do not contain any real data or overlap with existing public tables. The tables have <math id="S2.SS1.SSS0.Px2.p1.1.m1.1" class="ltx_Math" alttext="M" display="inline"><semantics id="S2.SS1.SSS0.Px2.p1.1.m1.1a"><mi id="S2.SS1.SSS0.Px2.p1.1.m1.1.1" xref="S2.SS1.SSS0.Px2.p1.1.m1.1.1.cmml">M</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.SSS0.Px2.p1.1.m1.1b"><ci id="S2.SS1.SSS0.Px2.p1.1.m1.1.1.cmml" xref="S2.SS1.SSS0.Px2.p1.1.m1.1.1">𝑀</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.SSS0.Px2.p1.1.m1.1c">M</annotation></semantics></math> rows and <math id="S2.SS1.SSS0.Px2.p1.2.m2.1" class="ltx_Math" alttext="N" display="inline"><semantics id="S2.SS1.SSS0.Px2.p1.2.m2.1a"><mi id="S2.SS1.SSS0.Px2.p1.2.m2.1.1" xref="S2.SS1.SSS0.Px2.p1.2.m2.1.1.cmml">N</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.SSS0.Px2.p1.2.m2.1b"><ci id="S2.SS1.SSS0.Px2.p1.2.m2.1.1.cmml" xref="S2.SS1.SSS0.Px2.p1.2.m2.1.1">𝑁</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.SSS0.Px2.p1.2.m2.1c">N</annotation></semantics></math> columns, with adjustable parameters <math id="S2.SS1.SSS0.Px2.p1.3.m3.1" class="ltx_Math" alttext="M" display="inline"><semantics id="S2.SS1.SSS0.Px2.p1.3.m3.1a"><mi id="S2.SS1.SSS0.Px2.p1.3.m3.1.1" xref="S2.SS1.SSS0.Px2.p1.3.m3.1.1.cmml">M</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.SSS0.Px2.p1.3.m3.1b"><ci id="S2.SS1.SSS0.Px2.p1.3.m3.1.1.cmml" xref="S2.SS1.SSS0.Px2.p1.3.m3.1.1">𝑀</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.SSS0.Px2.p1.3.m3.1c">M</annotation></semantics></math> and <math id="S2.SS1.SSS0.Px2.p1.4.m4.1" class="ltx_Math" alttext="N" display="inline"><semantics id="S2.SS1.SSS0.Px2.p1.4.m4.1a"><mi id="S2.SS1.SSS0.Px2.p1.4.m4.1.1" xref="S2.SS1.SSS0.Px2.p1.4.m4.1.1.cmml">N</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.SSS0.Px2.p1.4.m4.1b"><ci id="S2.SS1.SSS0.Px2.p1.4.m4.1.1.cmml" xref="S2.SS1.SSS0.Px2.p1.4.m4.1.1">𝑁</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.SSS0.Px2.p1.4.m4.1c">N</annotation></semantics></math>. The column headers are sampled from English nouns <cite class="ltx_cite ltx_citemacro_citep">(Bird, <a href="#bib.bib8" title="" class="ltx_ref">2006</a>)</cite>, falling into three types: TEXT, INT, and DATE. INT columns contain random integers from 1 to 1000, which is an adjustable range. DATE columns have values in year-month-day format. TEXT columns have random strings of length 5 to 12 characters, which is also adjustable.
To simulate real-world data where the same value may recur in a column frequently, the data generator includes a parameter to set the probability of duplicating values within a specific column.</p>
</div>
</section>
<section id="S2.SS1.SSS0.Px3" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Random SQL Generation</h4>

<div id="S2.SS1.SSS0.Px3.p1" class="ltx_para ltx_noindent">
<p id="S2.SS1.SSS0.Px3.p1.1" class="ltx_p">The SQL language includes a variety of statements to query and manage data. <span id="S2.SS1.SSS0.Px3.p1.1.1" class="ltx_text ltx_font_smallcaps">S3Eval</span> use context free grammar to generate a specific number of examples with controllable attributes.
As Table <a href="#S2.T1" title="Table 1 ‣ 2 Synthetic Suite: Alignment with Realistic Benchmark ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> shows, the <span id="S2.SS1.SSS0.Px3.p1.1.2" class="ltx_text ltx_font_smallcaps">S3Eval</span> tool allows configuring several parameters of generated SQL statements, including nesting depth, keywords used, length, coverage of SQL features, computational complexity, and more.
For example, <span id="S2.SS1.SSS0.Px3.p1.1.3" class="ltx_text ltx_font_italic">calculate times</span> can be modified to control the complexity of numerical reasoning for each dataset. Except these configures, users can also manually write the specified SQL template to generate fine-grained evaluation data (Appendix <a href="#A3.SS2" title="C.2 SQL Template ‣ Appendix C Data Demostration ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">C.2</span></a>).</p>
</div>
</section>
<section id="S2.SS1.SSS0.Px4" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Evaluation Methods</h4>

<div id="S2.SS1.SSS0.Px4.p1" class="ltx_para ltx_noindent">
<p id="S2.SS1.SSS0.Px4.p1.2" class="ltx_p"><span id="S2.SS1.SSS0.Px4.p1.2.1" class="ltx_text ltx_font_smallcaps">S3Eval</span> includes both zero-shot and few-shot prompting methods. For each few-shot setting, all examples share one table. N-shot is formalized as <math id="S2.SS1.SSS0.Px4.p1.1.m1.5" class="ltx_Math" alttext="\mathrm{INPUT=[T;S_{1};A_{1};...;S_{n+1}]}" display="inline"><semantics id="S2.SS1.SSS0.Px4.p1.1.m1.5a"><mrow id="S2.SS1.SSS0.Px4.p1.1.m1.5.5" xref="S2.SS1.SSS0.Px4.p1.1.m1.5.5.cmml"><mi id="S2.SS1.SSS0.Px4.p1.1.m1.5.5.5" xref="S2.SS1.SSS0.Px4.p1.1.m1.5.5.5.cmml">INPUT</mi><mo id="S2.SS1.SSS0.Px4.p1.1.m1.5.5.4" xref="S2.SS1.SSS0.Px4.p1.1.m1.5.5.4.cmml">=</mo><mrow id="S2.SS1.SSS0.Px4.p1.1.m1.5.5.3.3" xref="S2.SS1.SSS0.Px4.p1.1.m1.5.5.3.4.cmml"><mo stretchy="false" id="S2.SS1.SSS0.Px4.p1.1.m1.5.5.3.3.4" xref="S2.SS1.SSS0.Px4.p1.1.m1.5.5.3.4.cmml">[</mo><mi mathvariant="normal" id="S2.SS1.SSS0.Px4.p1.1.m1.1.1" xref="S2.SS1.SSS0.Px4.p1.1.m1.1.1.cmml">T</mi><mo id="S2.SS1.SSS0.Px4.p1.1.m1.5.5.3.3.5" xref="S2.SS1.SSS0.Px4.p1.1.m1.5.5.3.4.cmml">;</mo><msub id="S2.SS1.SSS0.Px4.p1.1.m1.3.3.1.1.1" xref="S2.SS1.SSS0.Px4.p1.1.m1.3.3.1.1.1.cmml"><mi mathvariant="normal" id="S2.SS1.SSS0.Px4.p1.1.m1.3.3.1.1.1.2" xref="S2.SS1.SSS0.Px4.p1.1.m1.3.3.1.1.1.2.cmml">S</mi><mn id="S2.SS1.SSS0.Px4.p1.1.m1.3.3.1.1.1.3" xref="S2.SS1.SSS0.Px4.p1.1.m1.3.3.1.1.1.3.cmml">1</mn></msub><mo id="S2.SS1.SSS0.Px4.p1.1.m1.5.5.3.3.6" xref="S2.SS1.SSS0.Px4.p1.1.m1.5.5.3.4.cmml">;</mo><msub id="S2.SS1.SSS0.Px4.p1.1.m1.4.4.2.2.2" xref="S2.SS1.SSS0.Px4.p1.1.m1.4.4.2.2.2.cmml"><mi mathvariant="normal" id="S2.SS1.SSS0.Px4.p1.1.m1.4.4.2.2.2.2" xref="S2.SS1.SSS0.Px4.p1.1.m1.4.4.2.2.2.2.cmml">A</mi><mn id="S2.SS1.SSS0.Px4.p1.1.m1.4.4.2.2.2.3" xref="S2.SS1.SSS0.Px4.p1.1.m1.4.4.2.2.2.3.cmml">1</mn></msub><mo id="S2.SS1.SSS0.Px4.p1.1.m1.5.5.3.3.7" xref="S2.SS1.SSS0.Px4.p1.1.m1.5.5.3.4.cmml">;</mo><mi mathvariant="normal" id="S2.SS1.SSS0.Px4.p1.1.m1.2.2" xref="S2.SS1.SSS0.Px4.p1.1.m1.2.2.cmml">…</mi><mo id="S2.SS1.SSS0.Px4.p1.1.m1.5.5.3.3.8" xref="S2.SS1.SSS0.Px4.p1.1.m1.5.5.3.4.cmml">;</mo><msub id="S2.SS1.SSS0.Px4.p1.1.m1.5.5.3.3.3" xref="S2.SS1.SSS0.Px4.p1.1.m1.5.5.3.3.3.cmml"><mi mathvariant="normal" id="S2.SS1.SSS0.Px4.p1.1.m1.5.5.3.3.3.2" xref="S2.SS1.SSS0.Px4.p1.1.m1.5.5.3.3.3.2.cmml">S</mi><mrow id="S2.SS1.SSS0.Px4.p1.1.m1.5.5.3.3.3.3" xref="S2.SS1.SSS0.Px4.p1.1.m1.5.5.3.3.3.3.cmml"><mi mathvariant="normal" id="S2.SS1.SSS0.Px4.p1.1.m1.5.5.3.3.3.3.2" xref="S2.SS1.SSS0.Px4.p1.1.m1.5.5.3.3.3.3.2.cmml">n</mi><mo id="S2.SS1.SSS0.Px4.p1.1.m1.5.5.3.3.3.3.1" xref="S2.SS1.SSS0.Px4.p1.1.m1.5.5.3.3.3.3.1.cmml">+</mo><mn id="S2.SS1.SSS0.Px4.p1.1.m1.5.5.3.3.3.3.3" xref="S2.SS1.SSS0.Px4.p1.1.m1.5.5.3.3.3.3.3.cmml">1</mn></mrow></msub><mo stretchy="false" id="S2.SS1.SSS0.Px4.p1.1.m1.5.5.3.3.9" xref="S2.SS1.SSS0.Px4.p1.1.m1.5.5.3.4.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS1.SSS0.Px4.p1.1.m1.5b"><apply id="S2.SS1.SSS0.Px4.p1.1.m1.5.5.cmml" xref="S2.SS1.SSS0.Px4.p1.1.m1.5.5"><eq id="S2.SS1.SSS0.Px4.p1.1.m1.5.5.4.cmml" xref="S2.SS1.SSS0.Px4.p1.1.m1.5.5.4"></eq><ci id="S2.SS1.SSS0.Px4.p1.1.m1.5.5.5.cmml" xref="S2.SS1.SSS0.Px4.p1.1.m1.5.5.5">INPUT</ci><list id="S2.SS1.SSS0.Px4.p1.1.m1.5.5.3.4.cmml" xref="S2.SS1.SSS0.Px4.p1.1.m1.5.5.3.3"><ci id="S2.SS1.SSS0.Px4.p1.1.m1.1.1.cmml" xref="S2.SS1.SSS0.Px4.p1.1.m1.1.1">T</ci><apply id="S2.SS1.SSS0.Px4.p1.1.m1.3.3.1.1.1.cmml" xref="S2.SS1.SSS0.Px4.p1.1.m1.3.3.1.1.1"><csymbol cd="ambiguous" id="S2.SS1.SSS0.Px4.p1.1.m1.3.3.1.1.1.1.cmml" xref="S2.SS1.SSS0.Px4.p1.1.m1.3.3.1.1.1">subscript</csymbol><ci id="S2.SS1.SSS0.Px4.p1.1.m1.3.3.1.1.1.2.cmml" xref="S2.SS1.SSS0.Px4.p1.1.m1.3.3.1.1.1.2">S</ci><cn type="integer" id="S2.SS1.SSS0.Px4.p1.1.m1.3.3.1.1.1.3.cmml" xref="S2.SS1.SSS0.Px4.p1.1.m1.3.3.1.1.1.3">1</cn></apply><apply id="S2.SS1.SSS0.Px4.p1.1.m1.4.4.2.2.2.cmml" xref="S2.SS1.SSS0.Px4.p1.1.m1.4.4.2.2.2"><csymbol cd="ambiguous" id="S2.SS1.SSS0.Px4.p1.1.m1.4.4.2.2.2.1.cmml" xref="S2.SS1.SSS0.Px4.p1.1.m1.4.4.2.2.2">subscript</csymbol><ci id="S2.SS1.SSS0.Px4.p1.1.m1.4.4.2.2.2.2.cmml" xref="S2.SS1.SSS0.Px4.p1.1.m1.4.4.2.2.2.2">A</ci><cn type="integer" id="S2.SS1.SSS0.Px4.p1.1.m1.4.4.2.2.2.3.cmml" xref="S2.SS1.SSS0.Px4.p1.1.m1.4.4.2.2.2.3">1</cn></apply><ci id="S2.SS1.SSS0.Px4.p1.1.m1.2.2.cmml" xref="S2.SS1.SSS0.Px4.p1.1.m1.2.2">…</ci><apply id="S2.SS1.SSS0.Px4.p1.1.m1.5.5.3.3.3.cmml" xref="S2.SS1.SSS0.Px4.p1.1.m1.5.5.3.3.3"><csymbol cd="ambiguous" id="S2.SS1.SSS0.Px4.p1.1.m1.5.5.3.3.3.1.cmml" xref="S2.SS1.SSS0.Px4.p1.1.m1.5.5.3.3.3">subscript</csymbol><ci id="S2.SS1.SSS0.Px4.p1.1.m1.5.5.3.3.3.2.cmml" xref="S2.SS1.SSS0.Px4.p1.1.m1.5.5.3.3.3.2">S</ci><apply id="S2.SS1.SSS0.Px4.p1.1.m1.5.5.3.3.3.3.cmml" xref="S2.SS1.SSS0.Px4.p1.1.m1.5.5.3.3.3.3"><plus id="S2.SS1.SSS0.Px4.p1.1.m1.5.5.3.3.3.3.1.cmml" xref="S2.SS1.SSS0.Px4.p1.1.m1.5.5.3.3.3.3.1"></plus><ci id="S2.SS1.SSS0.Px4.p1.1.m1.5.5.3.3.3.3.2.cmml" xref="S2.SS1.SSS0.Px4.p1.1.m1.5.5.3.3.3.3.2">n</ci><cn type="integer" id="S2.SS1.SSS0.Px4.p1.1.m1.5.5.3.3.3.3.3.cmml" xref="S2.SS1.SSS0.Px4.p1.1.m1.5.5.3.3.3.3.3">1</cn></apply></apply></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.SSS0.Px4.p1.1.m1.5c">\mathrm{INPUT=[T;S_{1};A_{1};...;S_{n+1}]}</annotation></semantics></math>. For the input format of table <math id="S2.SS1.SSS0.Px4.p1.2.m2.1" class="ltx_Math" alttext="T" display="inline"><semantics id="S2.SS1.SSS0.Px4.p1.2.m2.1a"><mi id="S2.SS1.SSS0.Px4.p1.2.m2.1.1" xref="S2.SS1.SSS0.Px4.p1.2.m2.1.1.cmml">T</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.SSS0.Px4.p1.2.m2.1b"><ci id="S2.SS1.SSS0.Px4.p1.2.m2.1.1.cmml" xref="S2.SS1.SSS0.Px4.p1.2.m2.1.1">𝑇</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.SSS0.Px4.p1.2.m2.1c">T</annotation></semantics></math>, we designed several alternative ways, including markdown, flatten, tapex-style, etc.</p>
</div>
<div id="S2.SS1.SSS0.Px4.p2" class="ltx_para ltx_noindent">
<p id="S2.SS1.SSS0.Px4.p2.1" class="ltx_p">To evaluate the performance of LLMs, we use Exact Match (EM) as the evaluation metric.
Details are shown in Appendix <a href="#A3.SS3" title="C.3 Table Input Format ‣ Appendix C Data Demostration ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">C.3</span></a>.</p>
</div>
</section>
<section id="S2.SS1.SSS0.Px5" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Evaluation Settings</h4>

<div id="S2.SS1.SSS0.Px5.p1" class="ltx_para ltx_noindent">
<p id="S2.SS1.SSS0.Px5.p1.1" class="ltx_p">Models are evaluated on two settings: <span id="S2.SS1.SSS0.Px5.p1.1.1" class="ltx_text ltx_font_bold">Easy</span> and <span id="S2.SS1.SSS0.Px5.p1.1.2" class="ltx_text ltx_font_bold">General</span>. <span id="S2.SS1.SSS0.Px5.p1.1.3" class="ltx_text ltx_font_bold">Easy</span> is the simplest data that <span id="S2.SS1.SSS0.Px5.p1.1.4" class="ltx_text ltx_font_smallcaps">S3Eval</span> can generate and is used to evaluate LLM’s ability to understand the most basic instructions. It contains only one template, “SELECT &lt;col1&gt; WHERE &lt;col2&gt; &lt;op&gt; &lt;value&gt;”.
<span id="S2.SS1.SSS0.Px5.p1.1.5" class="ltx_text ltx_font_bold">General</span> is a more difficult setting, containing extensive SQL syntax, and its generating setting is described in Appendix <a href="#A4.SS2" title="D.2 General Setting ‣ Appendix D Experiments Settings Details ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">D.2</span></a>.</p>
</div>
<div id="S2.SS1.SSS0.Px5.p2" class="ltx_para ltx_noindent">
<p id="S2.SS1.SSS0.Px5.p2.9" class="ltx_p"><span id="S2.SS1.SSS0.Px5.p2.9.1" class="ltx_text ltx_font_bold">General</span> randomly generates SQL queries of varying difficulty from an extensive grammar and provides a comprehensive evaluation of LLM performance. All experiments were run for <math id="S2.SS1.SSS0.Px5.p2.1.m1.1" class="ltx_Math" alttext="3" display="inline"><semantics id="S2.SS1.SSS0.Px5.p2.1.m1.1a"><mn id="S2.SS1.SSS0.Px5.p2.1.m1.1.1" xref="S2.SS1.SSS0.Px5.p2.1.m1.1.1.cmml">3</mn><annotation-xml encoding="MathML-Content" id="S2.SS1.SSS0.Px5.p2.1.m1.1b"><cn type="integer" id="S2.SS1.SSS0.Px5.p2.1.m1.1.1.cmml" xref="S2.SS1.SSS0.Px5.p2.1.m1.1.1">3</cn></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.SSS0.Px5.p2.1.m1.1c">3</annotation></semantics></math> times, using 1000 randomly generated queries per trial, with tables of 15 rows and 8 columns and an average of 1200 tokens per input. Details on the LLMs are provided in Appendix <a href="#A4.SS3" title="D.3 LLMs Used In This Paper ‣ Appendix D Experiments Settings Details ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">D.3</span></a>.
Considering <span id="S2.SS1.SSS0.Px5.p2.9.2" class="ltx_text ltx_font_italic">SQL execution</span> is a difficult task, some models may have a poor understanding of symbolic language, which makes it difficult to execute SQL, so we propose an alternative task <span id="S2.SS1.SSS0.Px5.p2.9.3" class="ltx_text ltx_font_italic">SQL multi-step task</span> to remove this potential bias. Specifically, it converts an SQL query into a multi-step table operation instruction as shown in Appendix <a href="#A3.SS5" title="C.5 Multi-step Instruction (Few-shot) ‣ Appendix C Data Demostration ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">C.5</span></a>. SQL has a fixed execution flow for the query statement: FROM <math id="S2.SS1.SSS0.Px5.p2.2.m2.1" class="ltx_Math" alttext="\rightarrow" display="inline"><semantics id="S2.SS1.SSS0.Px5.p2.2.m2.1a"><mo stretchy="false" id="S2.SS1.SSS0.Px5.p2.2.m2.1.1" xref="S2.SS1.SSS0.Px5.p2.2.m2.1.1.cmml">→</mo><annotation-xml encoding="MathML-Content" id="S2.SS1.SSS0.Px5.p2.2.m2.1b"><ci id="S2.SS1.SSS0.Px5.p2.2.m2.1.1.cmml" xref="S2.SS1.SSS0.Px5.p2.2.m2.1.1">→</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.SSS0.Px5.p2.2.m2.1c">\rightarrow</annotation></semantics></math> ON <math id="S2.SS1.SSS0.Px5.p2.3.m3.1" class="ltx_Math" alttext="\rightarrow" display="inline"><semantics id="S2.SS1.SSS0.Px5.p2.3.m3.1a"><mo stretchy="false" id="S2.SS1.SSS0.Px5.p2.3.m3.1.1" xref="S2.SS1.SSS0.Px5.p2.3.m3.1.1.cmml">→</mo><annotation-xml encoding="MathML-Content" id="S2.SS1.SSS0.Px5.p2.3.m3.1b"><ci id="S2.SS1.SSS0.Px5.p2.3.m3.1.1.cmml" xref="S2.SS1.SSS0.Px5.p2.3.m3.1.1">→</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.SSS0.Px5.p2.3.m3.1c">\rightarrow</annotation></semantics></math> JOIN <math id="S2.SS1.SSS0.Px5.p2.4.m4.1" class="ltx_Math" alttext="\rightarrow" display="inline"><semantics id="S2.SS1.SSS0.Px5.p2.4.m4.1a"><mo stretchy="false" id="S2.SS1.SSS0.Px5.p2.4.m4.1.1" xref="S2.SS1.SSS0.Px5.p2.4.m4.1.1.cmml">→</mo><annotation-xml encoding="MathML-Content" id="S2.SS1.SSS0.Px5.p2.4.m4.1b"><ci id="S2.SS1.SSS0.Px5.p2.4.m4.1.1.cmml" xref="S2.SS1.SSS0.Px5.p2.4.m4.1.1">→</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.SSS0.Px5.p2.4.m4.1c">\rightarrow</annotation></semantics></math> WHERE <math id="S2.SS1.SSS0.Px5.p2.5.m5.1" class="ltx_Math" alttext="\rightarrow" display="inline"><semantics id="S2.SS1.SSS0.Px5.p2.5.m5.1a"><mo stretchy="false" id="S2.SS1.SSS0.Px5.p2.5.m5.1.1" xref="S2.SS1.SSS0.Px5.p2.5.m5.1.1.cmml">→</mo><annotation-xml encoding="MathML-Content" id="S2.SS1.SSS0.Px5.p2.5.m5.1b"><ci id="S2.SS1.SSS0.Px5.p2.5.m5.1.1.cmml" xref="S2.SS1.SSS0.Px5.p2.5.m5.1.1">→</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.SSS0.Px5.p2.5.m5.1c">\rightarrow</annotation></semantics></math> GROUP BY <math id="S2.SS1.SSS0.Px5.p2.6.m6.1" class="ltx_Math" alttext="\rightarrow" display="inline"><semantics id="S2.SS1.SSS0.Px5.p2.6.m6.1a"><mo stretchy="false" id="S2.SS1.SSS0.Px5.p2.6.m6.1.1" xref="S2.SS1.SSS0.Px5.p2.6.m6.1.1.cmml">→</mo><annotation-xml encoding="MathML-Content" id="S2.SS1.SSS0.Px5.p2.6.m6.1b"><ci id="S2.SS1.SSS0.Px5.p2.6.m6.1.1.cmml" xref="S2.SS1.SSS0.Px5.p2.6.m6.1.1">→</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.SSS0.Px5.p2.6.m6.1c">\rightarrow</annotation></semantics></math> HAVING <math id="S2.SS1.SSS0.Px5.p2.7.m7.1" class="ltx_Math" alttext="\rightarrow" display="inline"><semantics id="S2.SS1.SSS0.Px5.p2.7.m7.1a"><mo stretchy="false" id="S2.SS1.SSS0.Px5.p2.7.m7.1.1" xref="S2.SS1.SSS0.Px5.p2.7.m7.1.1.cmml">→</mo><annotation-xml encoding="MathML-Content" id="S2.SS1.SSS0.Px5.p2.7.m7.1b"><ci id="S2.SS1.SSS0.Px5.p2.7.m7.1.1.cmml" xref="S2.SS1.SSS0.Px5.p2.7.m7.1.1">→</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.SSS0.Px5.p2.7.m7.1c">\rightarrow</annotation></semantics></math> SELECT <math id="S2.SS1.SSS0.Px5.p2.8.m8.1" class="ltx_Math" alttext="\rightarrow" display="inline"><semantics id="S2.SS1.SSS0.Px5.p2.8.m8.1a"><mo stretchy="false" id="S2.SS1.SSS0.Px5.p2.8.m8.1.1" xref="S2.SS1.SSS0.Px5.p2.8.m8.1.1.cmml">→</mo><annotation-xml encoding="MathML-Content" id="S2.SS1.SSS0.Px5.p2.8.m8.1b"><ci id="S2.SS1.SSS0.Px5.p2.8.m8.1.1.cmml" xref="S2.SS1.SSS0.Px5.p2.8.m8.1.1">→</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.SSS0.Px5.p2.8.m8.1c">\rightarrow</annotation></semantics></math> ORDER BY <math id="S2.SS1.SSS0.Px5.p2.9.m9.1" class="ltx_Math" alttext="\rightarrow" display="inline"><semantics id="S2.SS1.SSS0.Px5.p2.9.m9.1a"><mo stretchy="false" id="S2.SS1.SSS0.Px5.p2.9.m9.1.1" xref="S2.SS1.SSS0.Px5.p2.9.m9.1.1.cmml">→</mo><annotation-xml encoding="MathML-Content" id="S2.SS1.SSS0.Px5.p2.9.m9.1b"><ci id="S2.SS1.SSS0.Px5.p2.9.m9.1.1.cmml" xref="S2.SS1.SSS0.Px5.p2.9.m9.1.1">→</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.SSS0.Px5.p2.9.m9.1c">\rightarrow</annotation></semantics></math> LIMIT. This is not consistent with the order in which it is written. With this processing, it can also generate chain-of-thought prompting data.</p>
</div>
</section>
</section>
<section id="S2.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.2 </span>Alignment on Scaling Law</h3>

<div id="S2.SS2.p1" class="ltx_para ltx_noindent">
<p id="S2.SS2.p1.1" class="ltx_p">Previous work <cite class="ltx_cite ltx_citemacro_citep">(Kaplan et al., <a href="#bib.bib18" title="" class="ltx_ref">2020</a>; Hoffmann et al., <a href="#bib.bib16" title="" class="ltx_ref">2022</a>)</cite> shows a positive correlation between the cross-entropy loss of LLMs and the amount of computing resources used for training, as described by the empirical scaling law.
To verify whether the scaling law holds for our <span id="S2.SS2.p1.1.1" class="ltx_text ltx_font_smallcaps">S3Eval</span>, we employ a set of checkpoints of Pythia-12B <cite class="ltx_cite ltx_citemacro_citep">(Biderman et al., <a href="#bib.bib7" title="" class="ltx_ref">2023</a>)</cite> that are open-sourced at different training steps, corresponding to different amounts of compute.
Using both General and Easy settings, we observe a consistent pattern as illustrated in Figure <a href="#S2.F3" title="Figure 3 ‣ 2.2 Alignment on Scaling Law ‣ 2 Synthetic Suite: Alignment with Realistic Benchmark ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>: the scores show a smooth progression of improvement that aligns with the scaling law with increasing the training steps.
The steady, incremental performance gains over time, lacking any spikes, demonstrate <span id="S2.SS2.p1.1.2" class="ltx_text ltx_font_smallcaps">S3Eval</span>’s reliability as a evaluation suite.
Overall, these experimental results confirm the scaling law’s accuracy in forecasting model gains during training across diverse evaluation settings.</p>
</div>
<figure id="S2.F3" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S2.F3.1" class="ltx_figure ltx_figure_panel ltx_minipage ltx_align_center ltx_align_bottom" style="width:178.9pt;"><img src="/html/2310.15147/assets/x3.png" id="S2.F3.1.g1" class="ltx_graphics ltx_img_square" width="380" height="365" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 2: </span>The performance of Pythia-12B on the Easy and General settings of <span id="S2.F3.1.2.1" class="ltx_text ltx_font_smallcaps">S3Eval</span> was evaluated across different training steps.</figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S2.F3.2" class="ltx_figure ltx_figure_panel ltx_minipage ltx_align_center ltx_align_bottom" style="width:178.9pt;"><img src="/html/2310.15147/assets/x4.png" id="S2.F3.2.g1" class="ltx_graphics ltx_img_square" width="422" height="386" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 3: </span>The performance of different LLMs on <span id="S2.F3.2.2.1" class="ltx_text ltx_font_smallcaps">S3Eval</span> and WikiTableQuestions.</figcaption>
</figure>
</div>
</div>
</figure>
</section>
<section id="S2.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.3 </span>Alignment on Benchmark Performance</h3>

<div id="S2.SS3.p1" class="ltx_para ltx_noindent">
<p id="S2.SS3.p1.1" class="ltx_p">In the above, we validated that the LLMs also exhibits the scaling law observed in NL on the <span id="S2.SS3.p1.1.1" class="ltx_text ltx_font_smallcaps">S3Eval</span> suite. A natural question that arises is whether its performance on <span id="S2.SS3.p1.1.2" class="ltx_text ltx_font_smallcaps">S3Eval</span> is correlated with the performance on real-world, NL benchmarks.
To examine the hypothesis, we first compare the performance of different LLMs on <span id="S2.SS3.p1.1.3" class="ltx_text ltx_font_smallcaps">S3Eval</span> and on WikiTableQuestions <cite class="ltx_cite ltx_citemacro_citep">(Pasupat &amp; Liang, <a href="#bib.bib30" title="" class="ltx_ref">2015</a>)</cite>, a table question answering dataset consisting of questions and answers.
It is worth noting that to align the difficulty, we use the SQL queries from WikiTableQuestions <cite class="ltx_cite ltx_citemacro_citep">(Shi et al., <a href="#bib.bib35" title="" class="ltx_ref">2020</a>)</cite> as our <span id="S2.SS3.p1.1.4" class="ltx_text ltx_font_smallcaps">S3Eval</span> evaluation set.</p>
</div>
<div id="S2.SS3.p2" class="ltx_para ltx_noindent">
<p id="S2.SS3.p2.2" class="ltx_p">To systematically compare the performance, following previous work <cite class="ltx_cite ltx_citemacro_citep">(Liu et al., <a href="#bib.bib23" title="" class="ltx_ref">2023a</a>)</cite>,
we consider two correlation measures: the Pearson correlation coefficient (<math id="S2.SS3.p2.1.m1.1" class="ltx_Math" alttext="r" display="inline"><semantics id="S2.SS3.p2.1.m1.1a"><mi id="S2.SS3.p2.1.m1.1.1" xref="S2.SS3.p2.1.m1.1.1.cmml">r</mi><annotation-xml encoding="MathML-Content" id="S2.SS3.p2.1.m1.1b"><ci id="S2.SS3.p2.1.m1.1.1.cmml" xref="S2.SS3.p2.1.m1.1.1">𝑟</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p2.1.m1.1c">r</annotation></semantics></math>), which evaluates the linear relationship between model scores on the two benchmarks, and the Kendall rank correlation coefficient (<math id="S2.SS3.p2.2.m2.1" class="ltx_Math" alttext="\tau" display="inline"><semantics id="S2.SS3.p2.2.m2.1a"><mi id="S2.SS3.p2.2.m2.1.1" xref="S2.SS3.p2.2.m2.1.1.cmml">τ</mi><annotation-xml encoding="MathML-Content" id="S2.SS3.p2.2.m2.1b"><ci id="S2.SS3.p2.2.m2.1.1.cmml" xref="S2.SS3.p2.2.m2.1.1">𝜏</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p2.2.m2.1c">\tau</annotation></semantics></math>), which assesses whether the relative ranking of models is consistent across the benchmarks.
The specific steps for computing the correlation coefficients are as follows:</p>
<table id="S2.E1" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S2.E1.m1.5" class="ltx_Math" alttext="r=\frac{{\sum_{i=1}^{n}(x_{i}-\bar{x})(y_{i}-\bar{y})}}{{\sqrt{{\sum_{i=1}^{n}(x_{i}-\bar{x})^{2}}\sum_{i=1}^{n}(y_{i}-\bar{y})^{2}}}}." display="block"><semantics id="S2.E1.m1.5a"><mrow id="S2.E1.m1.5.5.1" xref="S2.E1.m1.5.5.1.1.cmml"><mrow id="S2.E1.m1.5.5.1.1" xref="S2.E1.m1.5.5.1.1.cmml"><mi id="S2.E1.m1.5.5.1.1.2" xref="S2.E1.m1.5.5.1.1.2.cmml">r</mi><mo id="S2.E1.m1.5.5.1.1.1" xref="S2.E1.m1.5.5.1.1.1.cmml">=</mo><mfrac id="S2.E1.m1.4.4" xref="S2.E1.m1.4.4.cmml"><mrow id="S2.E1.m1.2.2.2" xref="S2.E1.m1.2.2.2.cmml"><msubsup id="S2.E1.m1.2.2.2.3" xref="S2.E1.m1.2.2.2.3.cmml"><mo id="S2.E1.m1.2.2.2.3.2.2" xref="S2.E1.m1.2.2.2.3.2.2.cmml">∑</mo><mrow id="S2.E1.m1.2.2.2.3.2.3" xref="S2.E1.m1.2.2.2.3.2.3.cmml"><mi id="S2.E1.m1.2.2.2.3.2.3.2" xref="S2.E1.m1.2.2.2.3.2.3.2.cmml">i</mi><mo id="S2.E1.m1.2.2.2.3.2.3.1" xref="S2.E1.m1.2.2.2.3.2.3.1.cmml">=</mo><mn id="S2.E1.m1.2.2.2.3.2.3.3" xref="S2.E1.m1.2.2.2.3.2.3.3.cmml">1</mn></mrow><mi id="S2.E1.m1.2.2.2.3.3" xref="S2.E1.m1.2.2.2.3.3.cmml">n</mi></msubsup><mrow id="S2.E1.m1.2.2.2.2" xref="S2.E1.m1.2.2.2.2.cmml"><mrow id="S2.E1.m1.1.1.1.1.1.1" xref="S2.E1.m1.1.1.1.1.1.1.1.cmml"><mo lspace="0em" stretchy="false" id="S2.E1.m1.1.1.1.1.1.1.2" xref="S2.E1.m1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S2.E1.m1.1.1.1.1.1.1.1" xref="S2.E1.m1.1.1.1.1.1.1.1.cmml"><msub id="S2.E1.m1.1.1.1.1.1.1.1.2" xref="S2.E1.m1.1.1.1.1.1.1.1.2.cmml"><mi id="S2.E1.m1.1.1.1.1.1.1.1.2.2" xref="S2.E1.m1.1.1.1.1.1.1.1.2.2.cmml">x</mi><mi id="S2.E1.m1.1.1.1.1.1.1.1.2.3" xref="S2.E1.m1.1.1.1.1.1.1.1.2.3.cmml">i</mi></msub><mo id="S2.E1.m1.1.1.1.1.1.1.1.1" xref="S2.E1.m1.1.1.1.1.1.1.1.1.cmml">−</mo><mover accent="true" id="S2.E1.m1.1.1.1.1.1.1.1.3" xref="S2.E1.m1.1.1.1.1.1.1.1.3.cmml"><mi id="S2.E1.m1.1.1.1.1.1.1.1.3.2" xref="S2.E1.m1.1.1.1.1.1.1.1.3.2.cmml">x</mi><mo id="S2.E1.m1.1.1.1.1.1.1.1.3.1" xref="S2.E1.m1.1.1.1.1.1.1.1.3.1.cmml">¯</mo></mover></mrow><mo stretchy="false" id="S2.E1.m1.1.1.1.1.1.1.3" xref="S2.E1.m1.1.1.1.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="S2.E1.m1.2.2.2.2.3" xref="S2.E1.m1.2.2.2.2.3.cmml">​</mo><mrow id="S2.E1.m1.2.2.2.2.2.1" xref="S2.E1.m1.2.2.2.2.2.1.1.cmml"><mo stretchy="false" id="S2.E1.m1.2.2.2.2.2.1.2" xref="S2.E1.m1.2.2.2.2.2.1.1.cmml">(</mo><mrow id="S2.E1.m1.2.2.2.2.2.1.1" xref="S2.E1.m1.2.2.2.2.2.1.1.cmml"><msub id="S2.E1.m1.2.2.2.2.2.1.1.2" xref="S2.E1.m1.2.2.2.2.2.1.1.2.cmml"><mi id="S2.E1.m1.2.2.2.2.2.1.1.2.2" xref="S2.E1.m1.2.2.2.2.2.1.1.2.2.cmml">y</mi><mi id="S2.E1.m1.2.2.2.2.2.1.1.2.3" xref="S2.E1.m1.2.2.2.2.2.1.1.2.3.cmml">i</mi></msub><mo id="S2.E1.m1.2.2.2.2.2.1.1.1" xref="S2.E1.m1.2.2.2.2.2.1.1.1.cmml">−</mo><mover accent="true" id="S2.E1.m1.2.2.2.2.2.1.1.3" xref="S2.E1.m1.2.2.2.2.2.1.1.3.cmml"><mi id="S2.E1.m1.2.2.2.2.2.1.1.3.2" xref="S2.E1.m1.2.2.2.2.2.1.1.3.2.cmml">y</mi><mo id="S2.E1.m1.2.2.2.2.2.1.1.3.1" xref="S2.E1.m1.2.2.2.2.2.1.1.3.1.cmml">¯</mo></mover></mrow><mo stretchy="false" id="S2.E1.m1.2.2.2.2.2.1.3" xref="S2.E1.m1.2.2.2.2.2.1.1.cmml">)</mo></mrow></mrow></mrow><msqrt id="S2.E1.m1.4.4.4" xref="S2.E1.m1.4.4.4.cmml"><mrow id="S2.E1.m1.4.4.4.2.2" xref="S2.E1.m1.4.4.4.2.2.cmml"><msubsup id="S2.E1.m1.4.4.4.2.2.3" xref="S2.E1.m1.4.4.4.2.2.3.cmml"><mo id="S2.E1.m1.4.4.4.2.2.3.2.2" xref="S2.E1.m1.4.4.4.2.2.3.2.2.cmml">∑</mo><mrow id="S2.E1.m1.4.4.4.2.2.3.2.3" xref="S2.E1.m1.4.4.4.2.2.3.2.3.cmml"><mi id="S2.E1.m1.4.4.4.2.2.3.2.3.2" xref="S2.E1.m1.4.4.4.2.2.3.2.3.2.cmml">i</mi><mo id="S2.E1.m1.4.4.4.2.2.3.2.3.1" xref="S2.E1.m1.4.4.4.2.2.3.2.3.1.cmml">=</mo><mn id="S2.E1.m1.4.4.4.2.2.3.2.3.3" xref="S2.E1.m1.4.4.4.2.2.3.2.3.3.cmml">1</mn></mrow><mi id="S2.E1.m1.4.4.4.2.2.3.3" xref="S2.E1.m1.4.4.4.2.2.3.3.cmml">n</mi></msubsup><mrow id="S2.E1.m1.4.4.4.2.2.2" xref="S2.E1.m1.4.4.4.2.2.2.cmml"><msup id="S2.E1.m1.3.3.3.1.1.1.1" xref="S2.E1.m1.3.3.3.1.1.1.1.cmml"><mrow id="S2.E1.m1.3.3.3.1.1.1.1.1.1" xref="S2.E1.m1.3.3.3.1.1.1.1.1.1.1.cmml"><mo lspace="0em" stretchy="false" id="S2.E1.m1.3.3.3.1.1.1.1.1.1.2" xref="S2.E1.m1.3.3.3.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S2.E1.m1.3.3.3.1.1.1.1.1.1.1" xref="S2.E1.m1.3.3.3.1.1.1.1.1.1.1.cmml"><msub id="S2.E1.m1.3.3.3.1.1.1.1.1.1.1.2" xref="S2.E1.m1.3.3.3.1.1.1.1.1.1.1.2.cmml"><mi id="S2.E1.m1.3.3.3.1.1.1.1.1.1.1.2.2" xref="S2.E1.m1.3.3.3.1.1.1.1.1.1.1.2.2.cmml">x</mi><mi id="S2.E1.m1.3.3.3.1.1.1.1.1.1.1.2.3" xref="S2.E1.m1.3.3.3.1.1.1.1.1.1.1.2.3.cmml">i</mi></msub><mo id="S2.E1.m1.3.3.3.1.1.1.1.1.1.1.1" xref="S2.E1.m1.3.3.3.1.1.1.1.1.1.1.1.cmml">−</mo><mover accent="true" id="S2.E1.m1.3.3.3.1.1.1.1.1.1.1.3" xref="S2.E1.m1.3.3.3.1.1.1.1.1.1.1.3.cmml"><mi id="S2.E1.m1.3.3.3.1.1.1.1.1.1.1.3.2" xref="S2.E1.m1.3.3.3.1.1.1.1.1.1.1.3.2.cmml">x</mi><mo id="S2.E1.m1.3.3.3.1.1.1.1.1.1.1.3.1" xref="S2.E1.m1.3.3.3.1.1.1.1.1.1.1.3.1.cmml">¯</mo></mover></mrow><mo stretchy="false" id="S2.E1.m1.3.3.3.1.1.1.1.1.1.3" xref="S2.E1.m1.3.3.3.1.1.1.1.1.1.1.cmml">)</mo></mrow><mn id="S2.E1.m1.3.3.3.1.1.1.1.3" xref="S2.E1.m1.3.3.3.1.1.1.1.3.cmml">2</mn></msup><mo lspace="0em" rspace="0em" id="S2.E1.m1.4.4.4.2.2.2.3" xref="S2.E1.m1.4.4.4.2.2.2.3.cmml">​</mo><mrow id="S2.E1.m1.4.4.4.2.2.2.2" xref="S2.E1.m1.4.4.4.2.2.2.2.cmml"><msubsup id="S2.E1.m1.4.4.4.2.2.2.2.2" xref="S2.E1.m1.4.4.4.2.2.2.2.2.cmml"><mo rspace="0em" id="S2.E1.m1.4.4.4.2.2.2.2.2.2.2" xref="S2.E1.m1.4.4.4.2.2.2.2.2.2.2.cmml">∑</mo><mrow id="S2.E1.m1.4.4.4.2.2.2.2.2.2.3" xref="S2.E1.m1.4.4.4.2.2.2.2.2.2.3.cmml"><mi id="S2.E1.m1.4.4.4.2.2.2.2.2.2.3.2" xref="S2.E1.m1.4.4.4.2.2.2.2.2.2.3.2.cmml">i</mi><mo id="S2.E1.m1.4.4.4.2.2.2.2.2.2.3.1" xref="S2.E1.m1.4.4.4.2.2.2.2.2.2.3.1.cmml">=</mo><mn id="S2.E1.m1.4.4.4.2.2.2.2.2.2.3.3" xref="S2.E1.m1.4.4.4.2.2.2.2.2.2.3.3.cmml">1</mn></mrow><mi id="S2.E1.m1.4.4.4.2.2.2.2.2.3" xref="S2.E1.m1.4.4.4.2.2.2.2.2.3.cmml">n</mi></msubsup><msup id="S2.E1.m1.4.4.4.2.2.2.2.1" xref="S2.E1.m1.4.4.4.2.2.2.2.1.cmml"><mrow id="S2.E1.m1.4.4.4.2.2.2.2.1.1.1" xref="S2.E1.m1.4.4.4.2.2.2.2.1.1.1.1.cmml"><mo stretchy="false" id="S2.E1.m1.4.4.4.2.2.2.2.1.1.1.2" xref="S2.E1.m1.4.4.4.2.2.2.2.1.1.1.1.cmml">(</mo><mrow id="S2.E1.m1.4.4.4.2.2.2.2.1.1.1.1" xref="S2.E1.m1.4.4.4.2.2.2.2.1.1.1.1.cmml"><msub id="S2.E1.m1.4.4.4.2.2.2.2.1.1.1.1.2" xref="S2.E1.m1.4.4.4.2.2.2.2.1.1.1.1.2.cmml"><mi id="S2.E1.m1.4.4.4.2.2.2.2.1.1.1.1.2.2" xref="S2.E1.m1.4.4.4.2.2.2.2.1.1.1.1.2.2.cmml">y</mi><mi id="S2.E1.m1.4.4.4.2.2.2.2.1.1.1.1.2.3" xref="S2.E1.m1.4.4.4.2.2.2.2.1.1.1.1.2.3.cmml">i</mi></msub><mo id="S2.E1.m1.4.4.4.2.2.2.2.1.1.1.1.1" xref="S2.E1.m1.4.4.4.2.2.2.2.1.1.1.1.1.cmml">−</mo><mover accent="true" id="S2.E1.m1.4.4.4.2.2.2.2.1.1.1.1.3" xref="S2.E1.m1.4.4.4.2.2.2.2.1.1.1.1.3.cmml"><mi id="S2.E1.m1.4.4.4.2.2.2.2.1.1.1.1.3.2" xref="S2.E1.m1.4.4.4.2.2.2.2.1.1.1.1.3.2.cmml">y</mi><mo id="S2.E1.m1.4.4.4.2.2.2.2.1.1.1.1.3.1" xref="S2.E1.m1.4.4.4.2.2.2.2.1.1.1.1.3.1.cmml">¯</mo></mover></mrow><mo stretchy="false" id="S2.E1.m1.4.4.4.2.2.2.2.1.1.1.3" xref="S2.E1.m1.4.4.4.2.2.2.2.1.1.1.1.cmml">)</mo></mrow><mn id="S2.E1.m1.4.4.4.2.2.2.2.1.3" xref="S2.E1.m1.4.4.4.2.2.2.2.1.3.cmml">2</mn></msup></mrow></mrow></mrow></msqrt></mfrac></mrow><mo lspace="0em" id="S2.E1.m1.5.5.1.2" xref="S2.E1.m1.5.5.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.E1.m1.5b"><apply id="S2.E1.m1.5.5.1.1.cmml" xref="S2.E1.m1.5.5.1"><eq id="S2.E1.m1.5.5.1.1.1.cmml" xref="S2.E1.m1.5.5.1.1.1"></eq><ci id="S2.E1.m1.5.5.1.1.2.cmml" xref="S2.E1.m1.5.5.1.1.2">𝑟</ci><apply id="S2.E1.m1.4.4.cmml" xref="S2.E1.m1.4.4"><divide id="S2.E1.m1.4.4.5.cmml" xref="S2.E1.m1.4.4"></divide><apply id="S2.E1.m1.2.2.2.cmml" xref="S2.E1.m1.2.2.2"><apply id="S2.E1.m1.2.2.2.3.cmml" xref="S2.E1.m1.2.2.2.3"><csymbol cd="ambiguous" id="S2.E1.m1.2.2.2.3.1.cmml" xref="S2.E1.m1.2.2.2.3">superscript</csymbol><apply id="S2.E1.m1.2.2.2.3.2.cmml" xref="S2.E1.m1.2.2.2.3"><csymbol cd="ambiguous" id="S2.E1.m1.2.2.2.3.2.1.cmml" xref="S2.E1.m1.2.2.2.3">subscript</csymbol><sum id="S2.E1.m1.2.2.2.3.2.2.cmml" xref="S2.E1.m1.2.2.2.3.2.2"></sum><apply id="S2.E1.m1.2.2.2.3.2.3.cmml" xref="S2.E1.m1.2.2.2.3.2.3"><eq id="S2.E1.m1.2.2.2.3.2.3.1.cmml" xref="S2.E1.m1.2.2.2.3.2.3.1"></eq><ci id="S2.E1.m1.2.2.2.3.2.3.2.cmml" xref="S2.E1.m1.2.2.2.3.2.3.2">𝑖</ci><cn type="integer" id="S2.E1.m1.2.2.2.3.2.3.3.cmml" xref="S2.E1.m1.2.2.2.3.2.3.3">1</cn></apply></apply><ci id="S2.E1.m1.2.2.2.3.3.cmml" xref="S2.E1.m1.2.2.2.3.3">𝑛</ci></apply><apply id="S2.E1.m1.2.2.2.2.cmml" xref="S2.E1.m1.2.2.2.2"><times id="S2.E1.m1.2.2.2.2.3.cmml" xref="S2.E1.m1.2.2.2.2.3"></times><apply id="S2.E1.m1.1.1.1.1.1.1.1.cmml" xref="S2.E1.m1.1.1.1.1.1.1"><minus id="S2.E1.m1.1.1.1.1.1.1.1.1.cmml" xref="S2.E1.m1.1.1.1.1.1.1.1.1"></minus><apply id="S2.E1.m1.1.1.1.1.1.1.1.2.cmml" xref="S2.E1.m1.1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S2.E1.m1.1.1.1.1.1.1.1.2.1.cmml" xref="S2.E1.m1.1.1.1.1.1.1.1.2">subscript</csymbol><ci id="S2.E1.m1.1.1.1.1.1.1.1.2.2.cmml" xref="S2.E1.m1.1.1.1.1.1.1.1.2.2">𝑥</ci><ci id="S2.E1.m1.1.1.1.1.1.1.1.2.3.cmml" xref="S2.E1.m1.1.1.1.1.1.1.1.2.3">𝑖</ci></apply><apply id="S2.E1.m1.1.1.1.1.1.1.1.3.cmml" xref="S2.E1.m1.1.1.1.1.1.1.1.3"><ci id="S2.E1.m1.1.1.1.1.1.1.1.3.1.cmml" xref="S2.E1.m1.1.1.1.1.1.1.1.3.1">¯</ci><ci id="S2.E1.m1.1.1.1.1.1.1.1.3.2.cmml" xref="S2.E1.m1.1.1.1.1.1.1.1.3.2">𝑥</ci></apply></apply><apply id="S2.E1.m1.2.2.2.2.2.1.1.cmml" xref="S2.E1.m1.2.2.2.2.2.1"><minus id="S2.E1.m1.2.2.2.2.2.1.1.1.cmml" xref="S2.E1.m1.2.2.2.2.2.1.1.1"></minus><apply id="S2.E1.m1.2.2.2.2.2.1.1.2.cmml" xref="S2.E1.m1.2.2.2.2.2.1.1.2"><csymbol cd="ambiguous" id="S2.E1.m1.2.2.2.2.2.1.1.2.1.cmml" xref="S2.E1.m1.2.2.2.2.2.1.1.2">subscript</csymbol><ci id="S2.E1.m1.2.2.2.2.2.1.1.2.2.cmml" xref="S2.E1.m1.2.2.2.2.2.1.1.2.2">𝑦</ci><ci id="S2.E1.m1.2.2.2.2.2.1.1.2.3.cmml" xref="S2.E1.m1.2.2.2.2.2.1.1.2.3">𝑖</ci></apply><apply id="S2.E1.m1.2.2.2.2.2.1.1.3.cmml" xref="S2.E1.m1.2.2.2.2.2.1.1.3"><ci id="S2.E1.m1.2.2.2.2.2.1.1.3.1.cmml" xref="S2.E1.m1.2.2.2.2.2.1.1.3.1">¯</ci><ci id="S2.E1.m1.2.2.2.2.2.1.1.3.2.cmml" xref="S2.E1.m1.2.2.2.2.2.1.1.3.2">𝑦</ci></apply></apply></apply></apply><apply id="S2.E1.m1.4.4.4.cmml" xref="S2.E1.m1.4.4.4"><root id="S2.E1.m1.4.4.4a.cmml" xref="S2.E1.m1.4.4.4"></root><apply id="S2.E1.m1.4.4.4.2.2.cmml" xref="S2.E1.m1.4.4.4.2.2"><apply id="S2.E1.m1.4.4.4.2.2.3.cmml" xref="S2.E1.m1.4.4.4.2.2.3"><csymbol cd="ambiguous" id="S2.E1.m1.4.4.4.2.2.3.1.cmml" xref="S2.E1.m1.4.4.4.2.2.3">superscript</csymbol><apply id="S2.E1.m1.4.4.4.2.2.3.2.cmml" xref="S2.E1.m1.4.4.4.2.2.3"><csymbol cd="ambiguous" id="S2.E1.m1.4.4.4.2.2.3.2.1.cmml" xref="S2.E1.m1.4.4.4.2.2.3">subscript</csymbol><sum id="S2.E1.m1.4.4.4.2.2.3.2.2.cmml" xref="S2.E1.m1.4.4.4.2.2.3.2.2"></sum><apply id="S2.E1.m1.4.4.4.2.2.3.2.3.cmml" xref="S2.E1.m1.4.4.4.2.2.3.2.3"><eq id="S2.E1.m1.4.4.4.2.2.3.2.3.1.cmml" xref="S2.E1.m1.4.4.4.2.2.3.2.3.1"></eq><ci id="S2.E1.m1.4.4.4.2.2.3.2.3.2.cmml" xref="S2.E1.m1.4.4.4.2.2.3.2.3.2">𝑖</ci><cn type="integer" id="S2.E1.m1.4.4.4.2.2.3.2.3.3.cmml" xref="S2.E1.m1.4.4.4.2.2.3.2.3.3">1</cn></apply></apply><ci id="S2.E1.m1.4.4.4.2.2.3.3.cmml" xref="S2.E1.m1.4.4.4.2.2.3.3">𝑛</ci></apply><apply id="S2.E1.m1.4.4.4.2.2.2.cmml" xref="S2.E1.m1.4.4.4.2.2.2"><times id="S2.E1.m1.4.4.4.2.2.2.3.cmml" xref="S2.E1.m1.4.4.4.2.2.2.3"></times><apply id="S2.E1.m1.3.3.3.1.1.1.1.cmml" xref="S2.E1.m1.3.3.3.1.1.1.1"><csymbol cd="ambiguous" id="S2.E1.m1.3.3.3.1.1.1.1.2.cmml" xref="S2.E1.m1.3.3.3.1.1.1.1">superscript</csymbol><apply id="S2.E1.m1.3.3.3.1.1.1.1.1.1.1.cmml" xref="S2.E1.m1.3.3.3.1.1.1.1.1.1"><minus id="S2.E1.m1.3.3.3.1.1.1.1.1.1.1.1.cmml" xref="S2.E1.m1.3.3.3.1.1.1.1.1.1.1.1"></minus><apply id="S2.E1.m1.3.3.3.1.1.1.1.1.1.1.2.cmml" xref="S2.E1.m1.3.3.3.1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S2.E1.m1.3.3.3.1.1.1.1.1.1.1.2.1.cmml" xref="S2.E1.m1.3.3.3.1.1.1.1.1.1.1.2">subscript</csymbol><ci id="S2.E1.m1.3.3.3.1.1.1.1.1.1.1.2.2.cmml" xref="S2.E1.m1.3.3.3.1.1.1.1.1.1.1.2.2">𝑥</ci><ci id="S2.E1.m1.3.3.3.1.1.1.1.1.1.1.2.3.cmml" xref="S2.E1.m1.3.3.3.1.1.1.1.1.1.1.2.3">𝑖</ci></apply><apply id="S2.E1.m1.3.3.3.1.1.1.1.1.1.1.3.cmml" xref="S2.E1.m1.3.3.3.1.1.1.1.1.1.1.3"><ci id="S2.E1.m1.3.3.3.1.1.1.1.1.1.1.3.1.cmml" xref="S2.E1.m1.3.3.3.1.1.1.1.1.1.1.3.1">¯</ci><ci id="S2.E1.m1.3.3.3.1.1.1.1.1.1.1.3.2.cmml" xref="S2.E1.m1.3.3.3.1.1.1.1.1.1.1.3.2">𝑥</ci></apply></apply><cn type="integer" id="S2.E1.m1.3.3.3.1.1.1.1.3.cmml" xref="S2.E1.m1.3.3.3.1.1.1.1.3">2</cn></apply><apply id="S2.E1.m1.4.4.4.2.2.2.2.cmml" xref="S2.E1.m1.4.4.4.2.2.2.2"><apply id="S2.E1.m1.4.4.4.2.2.2.2.2.cmml" xref="S2.E1.m1.4.4.4.2.2.2.2.2"><csymbol cd="ambiguous" id="S2.E1.m1.4.4.4.2.2.2.2.2.1.cmml" xref="S2.E1.m1.4.4.4.2.2.2.2.2">superscript</csymbol><apply id="S2.E1.m1.4.4.4.2.2.2.2.2.2.cmml" xref="S2.E1.m1.4.4.4.2.2.2.2.2"><csymbol cd="ambiguous" id="S2.E1.m1.4.4.4.2.2.2.2.2.2.1.cmml" xref="S2.E1.m1.4.4.4.2.2.2.2.2">subscript</csymbol><sum id="S2.E1.m1.4.4.4.2.2.2.2.2.2.2.cmml" xref="S2.E1.m1.4.4.4.2.2.2.2.2.2.2"></sum><apply id="S2.E1.m1.4.4.4.2.2.2.2.2.2.3.cmml" xref="S2.E1.m1.4.4.4.2.2.2.2.2.2.3"><eq id="S2.E1.m1.4.4.4.2.2.2.2.2.2.3.1.cmml" xref="S2.E1.m1.4.4.4.2.2.2.2.2.2.3.1"></eq><ci id="S2.E1.m1.4.4.4.2.2.2.2.2.2.3.2.cmml" xref="S2.E1.m1.4.4.4.2.2.2.2.2.2.3.2">𝑖</ci><cn type="integer" id="S2.E1.m1.4.4.4.2.2.2.2.2.2.3.3.cmml" xref="S2.E1.m1.4.4.4.2.2.2.2.2.2.3.3">1</cn></apply></apply><ci id="S2.E1.m1.4.4.4.2.2.2.2.2.3.cmml" xref="S2.E1.m1.4.4.4.2.2.2.2.2.3">𝑛</ci></apply><apply id="S2.E1.m1.4.4.4.2.2.2.2.1.cmml" xref="S2.E1.m1.4.4.4.2.2.2.2.1"><csymbol cd="ambiguous" id="S2.E1.m1.4.4.4.2.2.2.2.1.2.cmml" xref="S2.E1.m1.4.4.4.2.2.2.2.1">superscript</csymbol><apply id="S2.E1.m1.4.4.4.2.2.2.2.1.1.1.1.cmml" xref="S2.E1.m1.4.4.4.2.2.2.2.1.1.1"><minus id="S2.E1.m1.4.4.4.2.2.2.2.1.1.1.1.1.cmml" xref="S2.E1.m1.4.4.4.2.2.2.2.1.1.1.1.1"></minus><apply id="S2.E1.m1.4.4.4.2.2.2.2.1.1.1.1.2.cmml" xref="S2.E1.m1.4.4.4.2.2.2.2.1.1.1.1.2"><csymbol cd="ambiguous" id="S2.E1.m1.4.4.4.2.2.2.2.1.1.1.1.2.1.cmml" xref="S2.E1.m1.4.4.4.2.2.2.2.1.1.1.1.2">subscript</csymbol><ci id="S2.E1.m1.4.4.4.2.2.2.2.1.1.1.1.2.2.cmml" xref="S2.E1.m1.4.4.4.2.2.2.2.1.1.1.1.2.2">𝑦</ci><ci id="S2.E1.m1.4.4.4.2.2.2.2.1.1.1.1.2.3.cmml" xref="S2.E1.m1.4.4.4.2.2.2.2.1.1.1.1.2.3">𝑖</ci></apply><apply id="S2.E1.m1.4.4.4.2.2.2.2.1.1.1.1.3.cmml" xref="S2.E1.m1.4.4.4.2.2.2.2.1.1.1.1.3"><ci id="S2.E1.m1.4.4.4.2.2.2.2.1.1.1.1.3.1.cmml" xref="S2.E1.m1.4.4.4.2.2.2.2.1.1.1.1.3.1">¯</ci><ci id="S2.E1.m1.4.4.4.2.2.2.2.1.1.1.1.3.2.cmml" xref="S2.E1.m1.4.4.4.2.2.2.2.1.1.1.1.3.2">𝑦</ci></apply></apply><cn type="integer" id="S2.E1.m1.4.4.4.2.2.2.2.1.3.cmml" xref="S2.E1.m1.4.4.4.2.2.2.2.1.3">2</cn></apply></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E1.m1.5c">r=\frac{{\sum_{i=1}^{n}(x_{i}-\bar{x})(y_{i}-\bar{y})}}{{\sqrt{{\sum_{i=1}^{n}(x_{i}-\bar{x})^{2}}\sum_{i=1}^{n}(y_{i}-\bar{y})^{2}}}}.</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(1)</span></td>
</tr></tbody>
</table>
<table id="S2.E2" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S2.E2.m1.2" class="ltx_Math" alttext="\tau=\frac{{2\times\text{{(number of concordant pairs}}-\text{{number of discordant pairs)}}}}{{{n(n-1)}}}," display="block"><semantics id="S2.E2.m1.2a"><mrow id="S2.E2.m1.2.2.1" xref="S2.E2.m1.2.2.1.1.cmml"><mrow id="S2.E2.m1.2.2.1.1" xref="S2.E2.m1.2.2.1.1.cmml"><mi id="S2.E2.m1.2.2.1.1.2" xref="S2.E2.m1.2.2.1.1.2.cmml">τ</mi><mo id="S2.E2.m1.2.2.1.1.1" xref="S2.E2.m1.2.2.1.1.1.cmml">=</mo><mfrac id="S2.E2.m1.1.1" xref="S2.E2.m1.1.1.cmml"><mrow id="S2.E2.m1.1.1.3" xref="S2.E2.m1.1.1.3.cmml"><mrow id="S2.E2.m1.1.1.3.2" xref="S2.E2.m1.1.1.3.2.cmml"><mn id="S2.E2.m1.1.1.3.2.2" xref="S2.E2.m1.1.1.3.2.2.cmml">2</mn><mo lspace="0.222em" rspace="0.222em" id="S2.E2.m1.1.1.3.2.1" xref="S2.E2.m1.1.1.3.2.1.cmml">×</mo><mtext id="S2.E2.m1.1.1.3.2.3" xref="S2.E2.m1.1.1.3.2.3a.cmml">(number of concordant pairs</mtext></mrow><mo id="S2.E2.m1.1.1.3.1" xref="S2.E2.m1.1.1.3.1.cmml">−</mo><mtext id="S2.E2.m1.1.1.3.3" xref="S2.E2.m1.1.1.3.3a.cmml">number of discordant pairs)</mtext></mrow><mrow id="S2.E2.m1.1.1.1" xref="S2.E2.m1.1.1.1.cmml"><mi id="S2.E2.m1.1.1.1.3" xref="S2.E2.m1.1.1.1.3.cmml">n</mi><mo lspace="0em" rspace="0em" id="S2.E2.m1.1.1.1.2" xref="S2.E2.m1.1.1.1.2.cmml">​</mo><mrow id="S2.E2.m1.1.1.1.1.1" xref="S2.E2.m1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.E2.m1.1.1.1.1.1.2" xref="S2.E2.m1.1.1.1.1.1.1.cmml">(</mo><mrow id="S2.E2.m1.1.1.1.1.1.1" xref="S2.E2.m1.1.1.1.1.1.1.cmml"><mi id="S2.E2.m1.1.1.1.1.1.1.2" xref="S2.E2.m1.1.1.1.1.1.1.2.cmml">n</mi><mo id="S2.E2.m1.1.1.1.1.1.1.1" xref="S2.E2.m1.1.1.1.1.1.1.1.cmml">−</mo><mn id="S2.E2.m1.1.1.1.1.1.1.3" xref="S2.E2.m1.1.1.1.1.1.1.3.cmml">1</mn></mrow><mo stretchy="false" id="S2.E2.m1.1.1.1.1.1.3" xref="S2.E2.m1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mfrac></mrow><mo id="S2.E2.m1.2.2.1.2" xref="S2.E2.m1.2.2.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.E2.m1.2b"><apply id="S2.E2.m1.2.2.1.1.cmml" xref="S2.E2.m1.2.2.1"><eq id="S2.E2.m1.2.2.1.1.1.cmml" xref="S2.E2.m1.2.2.1.1.1"></eq><ci id="S2.E2.m1.2.2.1.1.2.cmml" xref="S2.E2.m1.2.2.1.1.2">𝜏</ci><apply id="S2.E2.m1.1.1.cmml" xref="S2.E2.m1.1.1"><divide id="S2.E2.m1.1.1.2.cmml" xref="S2.E2.m1.1.1"></divide><apply id="S2.E2.m1.1.1.3.cmml" xref="S2.E2.m1.1.1.3"><minus id="S2.E2.m1.1.1.3.1.cmml" xref="S2.E2.m1.1.1.3.1"></minus><apply id="S2.E2.m1.1.1.3.2.cmml" xref="S2.E2.m1.1.1.3.2"><times id="S2.E2.m1.1.1.3.2.1.cmml" xref="S2.E2.m1.1.1.3.2.1"></times><cn type="integer" id="S2.E2.m1.1.1.3.2.2.cmml" xref="S2.E2.m1.1.1.3.2.2">2</cn><ci id="S2.E2.m1.1.1.3.2.3a.cmml" xref="S2.E2.m1.1.1.3.2.3"><mtext id="S2.E2.m1.1.1.3.2.3.cmml" xref="S2.E2.m1.1.1.3.2.3">(number of concordant pairs</mtext></ci></apply><ci id="S2.E2.m1.1.1.3.3a.cmml" xref="S2.E2.m1.1.1.3.3"><mtext id="S2.E2.m1.1.1.3.3.cmml" xref="S2.E2.m1.1.1.3.3">number of discordant pairs)</mtext></ci></apply><apply id="S2.E2.m1.1.1.1.cmml" xref="S2.E2.m1.1.1.1"><times id="S2.E2.m1.1.1.1.2.cmml" xref="S2.E2.m1.1.1.1.2"></times><ci id="S2.E2.m1.1.1.1.3.cmml" xref="S2.E2.m1.1.1.1.3">𝑛</ci><apply id="S2.E2.m1.1.1.1.1.1.1.cmml" xref="S2.E2.m1.1.1.1.1.1"><minus id="S2.E2.m1.1.1.1.1.1.1.1.cmml" xref="S2.E2.m1.1.1.1.1.1.1.1"></minus><ci id="S2.E2.m1.1.1.1.1.1.1.2.cmml" xref="S2.E2.m1.1.1.1.1.1.1.2">𝑛</ci><cn type="integer" id="S2.E2.m1.1.1.1.1.1.1.3.cmml" xref="S2.E2.m1.1.1.1.1.1.1.3">1</cn></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E2.m1.2c">\tau=\frac{{2\times\text{{(number of concordant pairs}}-\text{{number of discordant pairs)}}}}{{{n(n-1)}}},</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(2)</span></td>
</tr></tbody>
</table>
<p id="S2.SS3.p2.6" class="ltx_p">where <math id="S2.SS3.p2.3.m1.1" class="ltx_Math" alttext="x_{i}" display="inline"><semantics id="S2.SS3.p2.3.m1.1a"><msub id="S2.SS3.p2.3.m1.1.1" xref="S2.SS3.p2.3.m1.1.1.cmml"><mi id="S2.SS3.p2.3.m1.1.1.2" xref="S2.SS3.p2.3.m1.1.1.2.cmml">x</mi><mi id="S2.SS3.p2.3.m1.1.1.3" xref="S2.SS3.p2.3.m1.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS3.p2.3.m1.1b"><apply id="S2.SS3.p2.3.m1.1.1.cmml" xref="S2.SS3.p2.3.m1.1.1"><csymbol cd="ambiguous" id="S2.SS3.p2.3.m1.1.1.1.cmml" xref="S2.SS3.p2.3.m1.1.1">subscript</csymbol><ci id="S2.SS3.p2.3.m1.1.1.2.cmml" xref="S2.SS3.p2.3.m1.1.1.2">𝑥</ci><ci id="S2.SS3.p2.3.m1.1.1.3.cmml" xref="S2.SS3.p2.3.m1.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p2.3.m1.1c">x_{i}</annotation></semantics></math> and <math id="S2.SS3.p2.4.m2.1" class="ltx_Math" alttext="y_{i}" display="inline"><semantics id="S2.SS3.p2.4.m2.1a"><msub id="S2.SS3.p2.4.m2.1.1" xref="S2.SS3.p2.4.m2.1.1.cmml"><mi id="S2.SS3.p2.4.m2.1.1.2" xref="S2.SS3.p2.4.m2.1.1.2.cmml">y</mi><mi id="S2.SS3.p2.4.m2.1.1.3" xref="S2.SS3.p2.4.m2.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS3.p2.4.m2.1b"><apply id="S2.SS3.p2.4.m2.1.1.cmml" xref="S2.SS3.p2.4.m2.1.1"><csymbol cd="ambiguous" id="S2.SS3.p2.4.m2.1.1.1.cmml" xref="S2.SS3.p2.4.m2.1.1">subscript</csymbol><ci id="S2.SS3.p2.4.m2.1.1.2.cmml" xref="S2.SS3.p2.4.m2.1.1.2">𝑦</ci><ci id="S2.SS3.p2.4.m2.1.1.3.cmml" xref="S2.SS3.p2.4.m2.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p2.4.m2.1c">y_{i}</annotation></semantics></math> are the evaluation results of two different methods for the same model, respectively.
The strong correlation between LLMs’ performance on the SQL execution task and the table question answering task, as evidenced by the high <math id="S2.SS3.p2.5.m3.1" class="ltx_Math" alttext="r" display="inline"><semantics id="S2.SS3.p2.5.m3.1a"><mi id="S2.SS3.p2.5.m3.1.1" xref="S2.SS3.p2.5.m3.1.1.cmml">r</mi><annotation-xml encoding="MathML-Content" id="S2.SS3.p2.5.m3.1b"><ci id="S2.SS3.p2.5.m3.1.1.cmml" xref="S2.SS3.p2.5.m3.1.1">𝑟</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p2.5.m3.1c">r</annotation></semantics></math> (e.g., 99.1) and high <math id="S2.SS3.p2.6.m4.1" class="ltx_Math" alttext="\tau" display="inline"><semantics id="S2.SS3.p2.6.m4.1a"><mi id="S2.SS3.p2.6.m4.1.1" xref="S2.SS3.p2.6.m4.1.1.cmml">τ</mi><annotation-xml encoding="MathML-Content" id="S2.SS3.p2.6.m4.1b"><ci id="S2.SS3.p2.6.m4.1.1.cmml" xref="S2.SS3.p2.6.m4.1.1">𝜏</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p2.6.m4.1c">\tau</annotation></semantics></math> (e.g., 93.6) in Figure <a href="#S2.F3" title="Figure 3 ‣ 2.2 Alignment on Scaling Law ‣ 2 Synthetic Suite: Alignment with Realistic Benchmark ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>.</p>
</div>
<figure id="S2.F4" class="ltx_figure">
<div id="S2.F4.7" class="ltx_block">
<figure id="S2.F4.sf1" class="ltx_figure ltx_align_center"><img src="/html/2310.15147/assets/x5.png" id="S2.F4.sf1.g1" class="ltx_graphics ltx_img_square" width="181" height="160" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(a) </span>Performance of large language models</figcaption>
</figure>
<figure id="S2.F4.sf2" class="ltx_figure ltx_align_center"><img src="/html/2310.15147/assets/x6.png" id="S2.F4.sf2.g1" class="ltx_graphics ltx_img_square" width="181" height="157" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(b) </span>Performance of code large language models</figcaption>
</figure>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 4: </span>Each point in the scatterplot represents the LLM performance on the benchmarks corresponding to the horizontal and vertical coordinates. The black straight line is the trend line. The larger the values of <math id="S2.F4.4.m1.1" class="ltx_Math" alttext="r" display="inline"><semantics id="S2.F4.4.m1.1b"><mi id="S2.F4.4.m1.1.1" xref="S2.F4.4.m1.1.1.cmml">r</mi><annotation-xml encoding="MathML-Content" id="S2.F4.4.m1.1c"><ci id="S2.F4.4.m1.1.1.cmml" xref="S2.F4.4.m1.1.1">𝑟</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.F4.4.m1.1d">r</annotation></semantics></math> and <math id="S2.F4.5.m2.1" class="ltx_Math" alttext="\tau" display="inline"><semantics id="S2.F4.5.m2.1b"><mi id="S2.F4.5.m2.1.1" xref="S2.F4.5.m2.1.1.cmml">τ</mi><annotation-xml encoding="MathML-Content" id="S2.F4.5.m2.1c"><ci id="S2.F4.5.m2.1.1.cmml" xref="S2.F4.5.m2.1.1">𝜏</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.F4.5.m2.1d">\tau</annotation></semantics></math>, the higher the correlation between the two benchmarks. We consider <math id="S2.F4.6.m3.1" class="ltx_Math" alttext="\tau" display="inline"><semantics id="S2.F4.6.m3.1b"><mi id="S2.F4.6.m3.1.1" xref="S2.F4.6.m3.1.1.cmml">τ</mi><annotation-xml encoding="MathML-Content" id="S2.F4.6.m3.1c"><ci id="S2.F4.6.m3.1.1.cmml" xref="S2.F4.6.m3.1.1">𝜏</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.F4.6.m3.1d">\tau</annotation></semantics></math> &gt; 0.8 to be high concurrence.</figcaption>
</figure>
<div id="S2.SS3.p3" class="ltx_para ltx_noindent">
<p id="S2.SS3.p3.1" class="ltx_p">Although <span id="S2.SS3.p3.1.1" class="ltx_text ltx_font_smallcaps">S3Eval</span> has shown significant correlation with WikiTableQuestions, the fact that they are both tasks on tables may cause one to question whether <span id="S2.SS3.p3.1.2" class="ltx_text ltx_font_smallcaps">S3Eval</span> can serve as a proxy task to evaluate LLMs’ capabilities on generic reasoning tasks.
Therefore, we also compare the performance on <span id="S2.SS3.p3.1.3" class="ltx_text ltx_font_smallcaps">S3Eval</span> with the results of generic popular benchmarks like BBH <cite class="ltx_cite ltx_citemacro_citep">(Suzgun et al., <a href="#bib.bib37" title="" class="ltx_ref">2023</a>)</cite> and HumanEval <cite class="ltx_cite ltx_citemacro_citep">(Chen et al., <a href="#bib.bib11" title="" class="ltx_ref">2021a</a>)</cite>.
The results depicted in Figure <a href="#S2.F4.sf1" title="In Figure 4 ‣ 2.3 Alignment on Benchmark Performance ‣ 2 Synthetic Suite: Alignment with Realistic Benchmark ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4(a)</span></a> demonstrate a strong correlation between LLM performance on <span id="S2.SS3.p3.1.4" class="ltx_text ltx_font_smallcaps">S3Eval</span> and the BBH benchmark, with BBH performance obtained from the OpenCompass platform using few-shot chain-of-thought prompting <cite class="ltx_cite ltx_citemacro_citep">(OpenCompass, <a href="#bib.bib29" title="" class="ltx_ref">2023</a>)</cite>.
Similarly, Figure <a href="#S2.F4.sf2" title="In Figure 4 ‣ 2.3 Alignment on Benchmark Performance ‣ 2 Synthetic Suite: Alignment with Realistic Benchmark ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4(b)</span></a> illustrates the alignment between <span id="S2.SS3.p3.1.5" class="ltx_text ltx_font_smallcaps">S3Eval</span> performance and pass@1 scores on HumanEval <cite class="ltx_cite ltx_citemacro_citep">(Chen et al., <a href="#bib.bib12" title="" class="ltx_ref">2021b</a>)</cite> for code LLMs.
The results demonstrate that <span id="S2.SS3.p3.1.6" class="ltx_text ltx_font_smallcaps">S3Eval</span> serves as a robust proxy task for assessing the reasoning capabilities of LLMs on realistic benchmarks.
Concrete experimental results are provided in Table <a href="#A1.T3" title="Table 3 ‣ A.1 Overall Performance ‣ Appendix A Evaluation Experiments Results ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>.</p>
</div>
<figure id="S2.F6" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S2.F6.1" class="ltx_figure ltx_figure_panel ltx_minipage ltx_align_center ltx_align_bottom" style="width:190.8pt;"><img src="/html/2310.15147/assets/x7.png" id="S2.F6.1.g1" class="ltx_graphics ltx_img_square" width="422" height="382" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 5: </span>SQL execution training experiments on <span id="S2.F6.1.2.1" class="ltx_text ltx_font_smallcaps">S3Eval</span>.</figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S2.F6.2" class="ltx_figure ltx_figure_panel ltx_minipage ltx_align_center ltx_align_bottom" style="width:196.8pt;"><img src="/html/2310.15147/assets/x8.png" id="S2.F6.2.g1" class="ltx_graphics ltx_img_square" width="422" height="371" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 6: </span>Experiment results of different LLMs on different context lengths.</figcaption>
</figure>
</div>
</div>
</figure>
</section>
</section>
<section id="S3" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">3 </span>Scalable Suite: Unlimited Evaluation Resources</h2>

<div id="S3.p1" class="ltx_para ltx_noindent">
<p id="S3.p1.1" class="ltx_p"><span id="S3.p1.1.1" class="ltx_text ltx_font_smallcaps">S3Eval</span> provides a unique capability to generate infinite number of examples (Section <a href="#S3.SS1" title="3.1 Scalable Number of Evaluation Examples ‣ 3 Scalable Suite: Unlimited Evaluation Resources ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.1</span></a>) with infinite length (Section <a href="#S3.SS2" title="3.2 Scalable Length of Evaluation Examples ‣ 3 Scalable Suite: Unlimited Evaluation Resources ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.2</span></a>).</p>
</div>
<section id="S3.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.1 </span>Scalable Number of Evaluation Examples</h3>

<div id="S3.SS1.p1" class="ltx_para ltx_noindent">
<p id="S3.SS1.p1.1" class="ltx_p">The strength of <span id="S3.SS1.p1.1.1" class="ltx_text ltx_font_smallcaps">S3Eval</span> is its ability to generate unlimited number of examples for evaluation. This stems from two key design choices in <span id="S3.SS1.p1.1.2" class="ltx_text ltx_font_smallcaps">S3Eval</span>: (1) the synthetic table size can be scaled to different number of rows and columns, and (2) the table cells are synthesized from randomly generated strings. Combined with the provided large library of SQL query templates, these features enable the creation of a near-infinite set of unique evaluation examples.
This kind of capacity enables the continuous creation of novel examples unseen during training, which helps safeguard test data integrity by preventing leakage of the evaluation set into the training corpus.</p>
</div>
<div id="S3.SS1.p2" class="ltx_para ltx_noindent">
<p id="S3.SS1.p2.3" class="ltx_p">However, the absence of data leakage does not necessarily mean that <span id="S3.SS1.p2.3.1" class="ltx_text ltx_font_smallcaps">S3Eval</span>’s performance always represents the model’s out-of-distribution generalization ability. It is because the model may perform well on <span id="S3.SS1.p2.3.2" class="ltx_text ltx_font_smallcaps">S3Eval</span> via domain-specific training on the SQL execution task, rather than acquiring more general abilities.
To investigate whether LLMs can “hack” <span id="S3.SS1.p2.3.3" class="ltx_text ltx_font_smallcaps">S3Eval</span> via domain-specific training, we fine-tuned StarCoder-1B <cite class="ltx_cite ltx_citemacro_citep">(Li et al., <a href="#bib.bib20" title="" class="ltx_ref">2023a</a>)</cite>, which is not able to solve SQL execution tasks, on a randomly generated dataset of one million examples.
The performance of the fine-tuned StarCoder-1B is illustrated in Figure <a href="#S2.F6" title="Figure 6 ‣ 2.3 Alignment on Benchmark Performance ‣ 2 Synthetic Suite: Alignment with Realistic Benchmark ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>, where it is evaluated on three types of test datasets: <span id="S3.SS1.p2.3.4" class="ltx_text ltx_font_typewriter">Seen Table</span> (same tables as training), <span id="S3.SS1.p2.3.5" class="ltx_text ltx_font_typewriter">Unseen Table</span> (new tables in same format as training tables), and <span id="S3.SS1.p2.3.6" class="ltx_text ltx_font_typewriter">Unseen Templates</span> (new SQL query templates).
For the unseen table setting, we explore different table shapes, where <math id="S3.SS1.p2.1.m1.1" class="ltx_Math" alttext="(x\times y)" display="inline"><semantics id="S3.SS1.p2.1.m1.1a"><mrow id="S3.SS1.p2.1.m1.1.1.1" xref="S3.SS1.p2.1.m1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS1.p2.1.m1.1.1.1.2" xref="S3.SS1.p2.1.m1.1.1.1.1.cmml">(</mo><mrow id="S3.SS1.p2.1.m1.1.1.1.1" xref="S3.SS1.p2.1.m1.1.1.1.1.cmml"><mi id="S3.SS1.p2.1.m1.1.1.1.1.2" xref="S3.SS1.p2.1.m1.1.1.1.1.2.cmml">x</mi><mo lspace="0.222em" rspace="0.222em" id="S3.SS1.p2.1.m1.1.1.1.1.1" xref="S3.SS1.p2.1.m1.1.1.1.1.1.cmml">×</mo><mi id="S3.SS1.p2.1.m1.1.1.1.1.3" xref="S3.SS1.p2.1.m1.1.1.1.1.3.cmml">y</mi></mrow><mo stretchy="false" id="S3.SS1.p2.1.m1.1.1.1.3" xref="S3.SS1.p2.1.m1.1.1.1.1.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.1.m1.1b"><apply id="S3.SS1.p2.1.m1.1.1.1.1.cmml" xref="S3.SS1.p2.1.m1.1.1.1"><times id="S3.SS1.p2.1.m1.1.1.1.1.1.cmml" xref="S3.SS1.p2.1.m1.1.1.1.1.1"></times><ci id="S3.SS1.p2.1.m1.1.1.1.1.2.cmml" xref="S3.SS1.p2.1.m1.1.1.1.1.2">𝑥</ci><ci id="S3.SS1.p2.1.m1.1.1.1.1.3.cmml" xref="S3.SS1.p2.1.m1.1.1.1.1.3">𝑦</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.1.m1.1c">(x\times y)</annotation></semantics></math> means the table consists of <math id="S3.SS1.p2.2.m2.1" class="ltx_Math" alttext="x" display="inline"><semantics id="S3.SS1.p2.2.m2.1a"><mi id="S3.SS1.p2.2.m2.1.1" xref="S3.SS1.p2.2.m2.1.1.cmml">x</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.2.m2.1b"><ci id="S3.SS1.p2.2.m2.1.1.cmml" xref="S3.SS1.p2.2.m2.1.1">𝑥</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.2.m2.1c">x</annotation></semantics></math> rows and <math id="S3.SS1.p2.3.m3.1" class="ltx_Math" alttext="y" display="inline"><semantics id="S3.SS1.p2.3.m3.1a"><mi id="S3.SS1.p2.3.m3.1.1" xref="S3.SS1.p2.3.m3.1.1.cmml">y</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.3.m3.1b"><ci id="S3.SS1.p2.3.m3.1.1.cmml" xref="S3.SS1.p2.3.m3.1.1">𝑦</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.3.m3.1c">y</annotation></semantics></math> columns.</p>
</div>
<div id="S3.SS1.p3" class="ltx_para ltx_noindent">
<p id="S3.SS1.p3.1" class="ltx_p">The experimental results demonstrate that for Unseen Tables with different shapes, regardless of their size, the performance of the fine-tuned StarCoder experiences a substantial decline compared to Seen Tables. Likewise, when faced with Unseen Templates, the performance of the fine-tuned StarCoder exhibits a significant drop.
The results indicate that even if LLMs have been heavily trained on SQL execution tasks, their out-of-distribution performance can still be accurately evaluated by using novel SQL templates. These new SQL templates can be easily generated thanks to the vast grammar of SQL queries. Additionally, evaluating LLMs on larger tables that they were not trained on can also reveal part of their out-of-distribution capabilities.</p>
</div>
</section>
<section id="S3.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.2 </span>Scalable Length of Evaluation Examples</h3>

<div id="S3.SS2.p1" class="ltx_para ltx_noindent">
<p id="S3.SS2.p1.1" class="ltx_p">One advantage of <span id="S3.SS2.p1.1.1" class="ltx_text ltx_font_smallcaps">S3Eval</span> is its scalability and adjustable context length per example.
The flexibility allows <span id="S3.SS2.p1.1.2" class="ltx_text ltx_font_smallcaps">S3Eval</span> to rigorously evaluate LLMs that claim capability with long contexts.
To clearly expose limitations of current LLMs, we intentionally chose the <span id="S3.SS2.p1.1.3" class="ltx_text ltx_font_bold">Easy</span> setting in <span id="S3.SS2.p1.1.4" class="ltx_text ltx_font_smallcaps">S3Eval</span> to evaluate their performance. Specifically, we establish table configurations with approximately 2K, 4K, 8K, and 16K tokens, by using different numbers of rows and fixing the number of columns.
We generate a dataset consisting of 500 samples for each evaluation setting.
The experimental results on up to 16K context length are plotted in Figure <a href="#S2.F6" title="Figure 6 ‣ 2.3 Alignment on Benchmark Performance ‣ 2 Synthetic Suite: Alignment with Realistic Benchmark ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>.
As observed, the performance of almost all LLMs, significantly decreases as the context length increases.
Of all the models, Claude-1.3-100K is the only one that maintains a relatively strong performance trend.
Detailed results can be found in Appendix <a href="#A1.SS4" title="A.4 Long-Context Experiments ‣ Appendix A Evaluation Experiments Results ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A.4</span></a>.</p>
</div>
<div id="S3.SS2.p2" class="ltx_para ltx_noindent">
<p id="S3.SS2.p2.1" class="ltx_p">Given the limitation of existing LLMs on long-context tasks, we are curious about the bottleneck of them.
By using <span id="S3.SS2.p2.1.1" class="ltx_text ltx_font_smallcaps">S3Eval</span>, we can systematically investigate the long-context modeling capabilities of LLMs by controlling the distribution of answers in the evaluation suite.
Specifically, we use the <span id="S3.SS2.p2.1.2" class="ltx_text ltx_font_bold">Easy</span> setting and fix the number of answers to four cells (i.e., the result of the SQL execution is always spanning four cells).
As illustrated in Figure <a href="#S3.F7" title="Figure 7 ‣ 3.2 Scalable Length of Evaluation Examples ‣ 3 Scalable Suite: Unlimited Evaluation Resources ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a>, we introduce two distribution patterns, <span id="S3.SS2.p2.1.3" class="ltx_text ltx_font_italic">Dense</span> and <span id="S3.SS2.p2.1.4" class="ltx_text ltx_font_italic">Sparse</span> <span id="footnote1" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">1</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">1</sup><span class="ltx_tag ltx_tag_note">1</span>Examples of these two patterns can be found in Appendix <a href="#A3.SS1" title="C.1 Dense and Sparse Examples ‣ Appendix C Data Demostration ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">C.1</span></a>.</span></span></span> to probe the limitations of current LLMs.
The dense mode only requires the model to understand the local context, whereas the sparse mode requires the model to have a broader, global understanding of the context across multiple blocks.
The sparse mode intuitively poses more challenges and demands more complex reasoning across a broader scope of the provided context.
We conduct experiments on ChatGPT and Yarn-llama2-13B <cite class="ltx_cite ltx_citemacro_citep">(Peng et al., <a href="#bib.bib32" title="" class="ltx_ref">2023</a>)</cite>.
The experimental results indicate that both models perform significantly better in dense mode compared to sparse mode, as shown in Figure <a href="#S3.F7" title="Figure 7 ‣ 3.2 Scalable Length of Evaluation Examples ‣ 3 Scalable Suite: Unlimited Evaluation Resources ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a>.
This indicates that LLMs struggle to retrieve information over long sequences, even though their pre-training included lengthy contexts.
This may be caused by the fact that the training data does not contain sufficient examples of long-distance dependencies for the model to learn effectively.
Furthermore, the steep drop in performance from 4K to 8K tokens for both ChatGPT and Yarn-Llama2 in dense mode indicates that current length extension techniques may not be as effective as hoped.
In summary, we believe that <span id="S3.SS2.p2.1.5" class="ltx_text ltx_font_smallcaps">S3Eval</span> provides a valuable framework for evaluating long-context language models, as it allows testing models on dialogues of arbitrary length. This establishes a solid foundation for advancing research on large language models that can leverage long-term context.</p>
</div>
<figure id="S3.F7" class="ltx_figure"><img src="/html/2310.15147/assets/x9.png" id="S3.F7.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="360" height="175" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 7: </span>Experiment results of ChatGPT and Yarn-Llama 2 on <span id="S3.F7.9.1" class="ltx_text ltx_font_italic">Dense</span> and <span id="S3.F7.10.2" class="ltx_text ltx_font_italic">Sparse</span> Settings. <span id="S3.F7.11.3" class="ltx_text ltx_font_italic">Dense</span> means that the answer cells (i.e., <span id="S3.F7.12.4" class="ltx_text ltx_font_typewriter">Liu</span>, <span id="S3.F7.13.5" class="ltx_text ltx_font_typewriter">Chen</span>, <span id="S3.F7.14.6" class="ltx_text ltx_font_typewriter">Lei</span>, <span id="S3.F7.15.7" class="ltx_text ltx_font_typewriter">Wang</span>) lie in adjacent rows, and <span id="S3.F7.16.8" class="ltx_text ltx_font_italic">Sparse</span> means that the answer cells are separated. The model performs better on local queries which only involves adjacent cells.</figcaption>
</figure>
</section>
</section>
<section id="S4" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">4 </span>Systematic Suite: Controllable Analysis</h2>

<div id="S4.p1" class="ltx_para ltx_noindent">
<p id="S4.p1.1" class="ltx_p"><span id="S4.p1.1.1" class="ltx_text ltx_font_smallcaps">S3Eval</span> provides a comprehensive framework that empowers developers to synthesize diverse evaluation examples for systematically assessing LLMs from multiple perspectives.
In this section, inspired by the work of lost in the middle <cite class="ltx_cite ltx_citemacro_citep">(Liu et al., <a href="#bib.bib24" title="" class="ltx_ref">2023b</a>)</cite>, we first analyze the impact of answer position on performance (Section <a href="#S4.SS1" title="4.1 Answer Position Analysis ‣ 4 Systematic Suite: Controllable Analysis ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.1</span></a>).
Then we evaluate LLMs from different viewpoints, and we have conducted some initial explorations on the reasoning types analysis (Section <a href="#S4.SS2" title="4.2 Reasoning Type Analysis ‣ 4 Systematic Suite: Controllable Analysis ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.2</span></a>).
Last, we provide some insights by analyzing LLMs on three selected SQL templates (Section <a href="#S4.SS3" title="4.3 Template Controlled Analysis ‣ 4 Systematic Suite: Controllable Analysis ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.3</span></a>).
These experiments reveal counter-intuitive performance trends and new discoveries that may inspire further research and extension of the work.</p>
</div>
<section id="S4.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.1 </span>Answer Position Analysis</h3>

<div id="S4.SS1.p1" class="ltx_para ltx_noindent">
<p id="S4.SS1.p1.1" class="ltx_p">We investigate the influence of the answer’s position on the performance of LLMs, which is generally considered important.
Unlike standard NLP benchmarks where it is difficult to control the position of the answer, <span id="S4.SS1.p1.1.1" class="ltx_text ltx_font_smallcaps">S3Eval</span> allows for fine-grained control of answer position at the token level.
To mitigate the influence of long contexts, we only analyzed answers that fell within a limited context window (i.e., less than 4K tokens).</p>
</div>
<div id="S4.SS1.p2" class="ltx_para ltx_noindent">
<p id="S4.SS1.p2.1" class="ltx_p">Echoing the findings of <cite class="ltx_cite ltx_citemacro_citet">Liu et al. (<a href="#bib.bib24" title="" class="ltx_ref">2023b</a>)</cite>, “lost in the middle”, our results in Figure <a href="#S4.F8" title="Figure 8 ‣ 4.1 Answer Position Analysis ‣ 4 Systematic Suite: Controllable Analysis ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">8</span></a> demonstrate that both ChatGPT and CodeLlama achieve higher performance when the answer is located at the beginning or end of the context, compared to when it appears in the middle.
In addition, we found a periodic fluctuation trend in the performance of both models as the position of the answer shifts within the context.
For example, the performance of ChatGPT increases from 0 to around 200, then starts to decrease from around 200 to 500.
This wave-like pattern in performance appears to correlate with the position embedding approach used by LLMs.</p>
</div>
<div id="S4.SS1.p3" class="ltx_para ltx_noindent">
<p id="S4.SS1.p3.1" class="ltx_p">In contrast to previous studies that used long-context question answering tasks <cite class="ltx_cite ltx_citemacro_citep">(Liu et al., <a href="#bib.bib24" title="" class="ltx_ref">2023b</a>; Bai et al., <a href="#bib.bib5" title="" class="ltx_ref">2023b</a>)</cite> for analysis and are thus limited to controlling answer positions at the paragraph level, <span id="S4.SS1.p3.1.1" class="ltx_text ltx_font_smallcaps">S3Eval</span> provides a more precise approach by focusing on token level. This key difference enables <span id="S4.SS1.p3.1.2" class="ltx_text ltx_font_smallcaps">S3Eval</span> to offer fine-grained control and promote the exploration of relevant phenomena.</p>
</div>
<figure id="S4.F8" class="ltx_figure">
<div id="S4.F8.1" class="ltx_block">
<figure id="S4.F8.sf1" class="ltx_figure"><img src="/html/2310.15147/assets/x10.png" id="S4.F8.sf1.g1" class="ltx_graphics ltx_img_landscape" width="190" height="143" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(a) </span>ChatGPT</figcaption>
</figure>
<figure id="S4.F8.sf2" class="ltx_figure"><img src="/html/2310.15147/assets/x11.png" id="S4.F8.sf2.g1" class="ltx_graphics ltx_img_landscape" width="190" height="142" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(b) </span>CodeLlama</figcaption>
</figure>
</div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 8: </span>The relationship between LLMs performance and the position of the answer token. The purple data points in figure represent the performance under the ‘sliding window=50’, while the orange curve represents the trend line. ‘sliding window’ means that each data point represents the performance of the answer position within its window range. This is advantageous for smoothing.
</figcaption>
</figure>
<figure id="S4.T2" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 2: </span>Reasoning types experiments examples of different LLMs.</figcaption>
<table id="S4.T2.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="S4.T2.1.1" class="ltx_tr">
<td id="S4.T2.1.1.1" class="ltx_td ltx_align_center ltx_border_tt"><span id="S4.T2.1.1.1.1" class="ltx_text ltx_font_bold">Operator</span></td>
<td id="S4.T2.1.1.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_tt">
<span id="S4.T2.1.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T2.1.1.2.1.1" class="ltx_p" style="width:108.1pt;"><span id="S4.T2.1.1.2.1.1.1" class="ltx_text ltx_font_bold">Example SQL</span></span>
</span>
</td>
<td id="S4.T2.1.1.3" class="ltx_td ltx_align_center ltx_border_tt"><span id="S4.T2.1.1.3.1" class="ltx_text ltx_font_bold">ChatGPT</span></td>
<td id="S4.T2.1.1.4" class="ltx_td ltx_align_center ltx_border_tt"><span id="S4.T2.1.1.4.1" class="ltx_text ltx_font_bold">Claude</span></td>
<td id="S4.T2.1.1.5" class="ltx_td ltx_align_center ltx_border_tt"><span id="S4.T2.1.1.5.1" class="ltx_text ltx_font_bold">Mistral</span></td>
<td id="S4.T2.1.1.6" class="ltx_td ltx_align_center ltx_border_tt"><span id="S4.T2.1.1.6.1" class="ltx_text ltx_font_bold">CodeLlama</span></td>
</tr>
<tr id="S4.T2.1.2" class="ltx_tr">
<td id="S4.T2.1.2.1" class="ltx_td ltx_align_center ltx_border_t"><span id="S4.T2.1.2.1.1" class="ltx_text ltx_font_bold" style="color:#FFA500;">Filter</span></td>
<td id="S4.T2.1.2.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="S4.T2.1.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T2.1.2.2.1.1" class="ltx_p" style="width:108.1pt;">SELECT lyonnais FROM table WHERE farmer = ’mijl’ AND lashing &gt;288</span>
</span>
</td>
<td id="S4.T2.1.2.3" class="ltx_td ltx_align_center ltx_border_t">79.6</td>
<td id="S4.T2.1.2.4" class="ltx_td ltx_align_center ltx_border_t">79.2</td>
<td id="S4.T2.1.2.5" class="ltx_td ltx_align_center ltx_border_t">64.8</td>
<td id="S4.T2.1.2.6" class="ltx_td ltx_align_center ltx_border_t">72.8</td>
</tr>
<tr id="S4.T2.1.3" class="ltx_tr">
<td id="S4.T2.1.3.1" class="ltx_td ltx_align_center ltx_border_t"><span id="S4.T2.1.3.1.1" class="ltx_text ltx_font_bold" style="color:#008000;">Arithmetic</span></td>
<td id="S4.T2.1.3.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="S4.T2.1.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T2.1.3.2.1.1" class="ltx_p" style="width:108.1pt;">SELECT synset + refuge FROM table WHERE blender = ’owxdbzjg’</span>
</span>
</td>
<td id="S4.T2.1.3.3" class="ltx_td ltx_align_center ltx_border_t">67.2</td>
<td id="S4.T2.1.3.4" class="ltx_td ltx_align_center ltx_border_t">59.4</td>
<td id="S4.T2.1.3.5" class="ltx_td ltx_align_center ltx_border_t">5.4</td>
<td id="S4.T2.1.3.6" class="ltx_td ltx_align_center ltx_border_t">10.6</td>
</tr>
<tr id="S4.T2.1.4" class="ltx_tr">
<td id="S4.T2.1.4.1" class="ltx_td ltx_align_center ltx_border_t"><span id="S4.T2.1.4.1.1" class="ltx_text ltx_font_bold" style="color:#DDA0DD;">Comparative</span></td>
<td id="S4.T2.1.4.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="S4.T2.1.4.2.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T2.1.4.2.1.1" class="ltx_p" style="width:108.1pt;">SELECT upsetter &lt; jollity FROM table WHERE kelp = 150</span>
</span>
</td>
<td id="S4.T2.1.4.3" class="ltx_td ltx_align_center ltx_border_t">45.2</td>
<td id="S4.T2.1.4.4" class="ltx_td ltx_align_center ltx_border_t">46.4</td>
<td id="S4.T2.1.4.5" class="ltx_td ltx_align_center ltx_border_t">44.8</td>
<td id="S4.T2.1.4.6" class="ltx_td ltx_align_center ltx_border_t">46.6</td>
</tr>
<tr id="S4.T2.1.5" class="ltx_tr">
<td id="S4.T2.1.5.1" class="ltx_td ltx_align_center ltx_border_t"><span id="S4.T2.1.5.1.1" class="ltx_text ltx_font_bold" style="color:#FF6666;">Aggregate</span></td>
<td id="S4.T2.1.5.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="S4.T2.1.5.2.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T2.1.5.2.1.1" class="ltx_p" style="width:108.1pt;">SELECT MIN(skeptic) FROM table</span>
</span>
</td>
<td id="S4.T2.1.5.3" class="ltx_td ltx_align_center ltx_border_t">38.4</td>
<td id="S4.T2.1.5.4" class="ltx_td ltx_align_center ltx_border_t">39.4</td>
<td id="S4.T2.1.5.5" class="ltx_td ltx_align_center ltx_border_t">28.4</td>
<td id="S4.T2.1.5.6" class="ltx_td ltx_align_center ltx_border_t">33.8</td>
</tr>
<tr id="S4.T2.1.6" class="ltx_tr">
<td id="S4.T2.1.6.1" class="ltx_td ltx_align_center ltx_border_t"><span id="S4.T2.1.6.1.1" class="ltx_text ltx_font_bold" style="color:#8A2BE2;">Group</span></td>
<td id="S4.T2.1.6.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="S4.T2.1.6.2.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T2.1.6.2.1.1" class="ltx_p" style="width:108.1pt;">SELECT lats FROM table GROUP BY shastan HAVING sum ( logbook ) = 56</span>
</span>
</td>
<td id="S4.T2.1.6.3" class="ltx_td ltx_align_center ltx_border_t">38.1</td>
<td id="S4.T2.1.6.4" class="ltx_td ltx_align_center ltx_border_t">28.2</td>
<td id="S4.T2.1.6.5" class="ltx_td ltx_align_center ltx_border_t">31.0</td>
<td id="S4.T2.1.6.6" class="ltx_td ltx_align_center ltx_border_t">37.8</td>
</tr>
<tr id="S4.T2.1.7" class="ltx_tr">
<td id="S4.T2.1.7.1" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t"><span id="S4.T2.1.7.1.1" class="ltx_text ltx_font_bold" style="color:#00FFFF;">Superlative</span></td>
<td id="S4.T2.1.7.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_bb ltx_border_t">
<span id="S4.T2.1.7.2.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T2.1.7.2.1.1" class="ltx_p" style="width:108.1pt;">SELECT severity FROM table ORDER BY bierce DESC Limit 1</span>
</span>
</td>
<td id="S4.T2.1.7.3" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">24.8</td>
<td id="S4.T2.1.7.4" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">41.4</td>
<td id="S4.T2.1.7.5" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">19.2</td>
<td id="S4.T2.1.7.6" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">28.3</td>
</tr>
</table>
</figure>
</section>
<section id="S4.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.2 </span>Reasoning Type Analysis</h3>

<div id="S4.SS2.p1" class="ltx_para ltx_noindent">
<p id="S4.SS2.p1.1" class="ltx_p"><span id="S4.SS2.p1.1.1" class="ltx_text ltx_font_smallcaps">S3Eval</span> enables the creation of multiple templates to generate different SQL statements, with each statement representing a distinct reasoning type.
We selected six common reasoning types to investigate the reasoning capabilities of LLMs and examined four different LLMs: ChatGPT, Claude, Mistral-7B, and CodeLlama-34B.
Following <cite class="ltx_cite ltx_citemacro_citet">Liu et al. (<a href="#bib.bib25" title="" class="ltx_ref">2022</a>)</cite>, the six reasoning types <span id="footnote2" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">2</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">2</sup><span class="ltx_tag ltx_tag_note">2</span>Detailed templates for each type can be found in Appendix <a href="#A3.SS2" title="C.2 SQL Template ‣ Appendix C Data Demostration ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">C.2</span></a>.</span></span></span> we considered are <span id="S4.SS2.p1.1.2" class="ltx_text ltx_font_typewriter">Filter</span>, <span id="S4.SS2.p1.1.3" class="ltx_text ltx_font_typewriter">Aggregate</span>, <span id="S4.SS2.p1.1.4" class="ltx_text ltx_font_typewriter">Arithmetic</span>, <span id="S4.SS2.p1.1.5" class="ltx_text ltx_font_typewriter">Superlative</span>, <span id="S4.SS2.p1.1.6" class="ltx_text ltx_font_typewriter">Comparative</span>, and <span id="S4.SS2.p1.1.7" class="ltx_text ltx_font_typewriter">Group</span>. The example SQL and the experimental results of different LLMs are presented in Table <a href="#S4.T2" title="Table 2 ‣ 4.1 Answer Position Analysis ‣ 4 Systematic Suite: Controllable Analysis ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>.
The expressive power of SQL queries enables <span id="S4.SS2.p1.1.8" class="ltx_text ltx_font_smallcaps">S3Eval</span> to be used for evaluating diverse scenarios such as numerical reasoning, multi-hop reasoning, complex code understanding, and multi-turn interaction with intermediate execution results.</p>
</div>
</section>
<section id="S4.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.3 </span>Template Controlled Analysis</h3>

<div id="S4.SS3.p1" class="ltx_para ltx_noindent">
<p id="S4.SS3.p1.1" class="ltx_p">Each data template in <span id="S4.SS3.p1.1.1" class="ltx_text ltx_font_smallcaps">S3Eval</span> includes corresponding reasoning types, and thus it provides fine-grained control over the evaluation examples.
To stimulate new insights and uncover counter-intuitive performance phenomena of LLMs, we present several controlled analysis examples using simple templates as a starting point.</p>
</div>
<div id="S4.SS3.p2" class="ltx_para ltx_noindent">
<p id="S4.SS3.p2.1" class="ltx_p"><span id="S4.SS3.p2.1.1" class="ltx_text ltx_font_bold">Template1: SELECT <span id="S4.SS3.p2.1.1.1" class="ltx_text">[</span>text_col1<span id="S4.SS3.p2.1.1.2" class="ltx_text">]</span> FROM table WHERE (<span id="S4.SS3.p2.1.1.3" class="ltx_text">[</span>text_col2<span id="S4.SS3.p2.1.1.4" class="ltx_text">]</span> = <span id="S4.SS3.p2.1.1.5" class="ltx_text">[</span>text2<span id="S4.SS3.p2.1.1.6" class="ltx_text">]</span>)</span></p>
</div>
<figure id="S4.F9" class="ltx_figure"><img src="/html/2310.15147/assets/x12.png" id="S4.F9.g1" class="ltx_graphics ltx_img_square" width="461" height="419" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 9: </span>ChatGPT performance with different locations of [text_col1] and [text_col2]. The performance improves when the example has the location of [text_col1] before [text_col2].</figcaption>
</figure>
<div id="S4.SS3.p3" class="ltx_para ltx_noindent">
<p id="S4.SS3.p3.2" class="ltx_p">We first explore the relationship between the model performance and the locations of [text_col1] and [text_col2].
To begin with, we generated a set of <math id="S4.SS3.p3.1.m1.1" class="ltx_Math" alttext="10\times 15" display="inline"><semantics id="S4.SS3.p3.1.m1.1a"><mrow id="S4.SS3.p3.1.m1.1.1" xref="S4.SS3.p3.1.m1.1.1.cmml"><mn id="S4.SS3.p3.1.m1.1.1.2" xref="S4.SS3.p3.1.m1.1.1.2.cmml">10</mn><mo lspace="0.222em" rspace="0.222em" id="S4.SS3.p3.1.m1.1.1.1" xref="S4.SS3.p3.1.m1.1.1.1.cmml">×</mo><mn id="S4.SS3.p3.1.m1.1.1.3" xref="S4.SS3.p3.1.m1.1.1.3.cmml">15</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.SS3.p3.1.m1.1b"><apply id="S4.SS3.p3.1.m1.1.1.cmml" xref="S4.SS3.p3.1.m1.1.1"><times id="S4.SS3.p3.1.m1.1.1.1.cmml" xref="S4.SS3.p3.1.m1.1.1.1"></times><cn type="integer" id="S4.SS3.p3.1.m1.1.1.2.cmml" xref="S4.SS3.p3.1.m1.1.1.2">10</cn><cn type="integer" id="S4.SS3.p3.1.m1.1.1.3.cmml" xref="S4.SS3.p3.1.m1.1.1.3">15</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p3.1.m1.1c">10\times 15</annotation></semantics></math> tables, each comprising <math id="S4.SS3.p3.2.m2.1" class="ltx_Math" alttext="15" display="inline"><semantics id="S4.SS3.p3.2.m2.1a"><mn id="S4.SS3.p3.2.m2.1.1" xref="S4.SS3.p3.2.m2.1.1.cmml">15</mn><annotation-xml encoding="MathML-Content" id="S4.SS3.p3.2.m2.1b"><cn type="integer" id="S4.SS3.p3.2.m2.1.1.cmml" xref="S4.SS3.p3.2.m2.1.1">15</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p3.2.m2.1c">15</annotation></semantics></math> distinct columns.
We created 400 unique combinations by pairing each value in text_col1 with each value in text_col2. For each of the 400 pairs, we generated 40 evaluation examples, resulting in a total of 16,000 evaluation examples.
After SQL execution experiments, we calculated the scores of each pair and constructed a heatmap, which is illustrated in Figure <a href="#S4.F9" title="Figure 9 ‣ 4.3 Template Controlled Analysis ‣ 4 Systematic Suite: Controllable Analysis ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">9</span></a>. The heatmap indicates that the performance is overall better when [text_col1] is the previous column. And the model performance is also better when the [text_col1] column is before [text_col2] column.
It indicates that the model tends to focus on the beginning of a specific paragraph. Moreover, in multi-hop reasoning, LLMs excel at hopping to the context preceding a intermediate hop, but struggles when it comes to searching backward.</p>
</div>
<div id="S4.SS3.p4" class="ltx_para ltx_noindent">
<p id="S4.SS3.p4.1" class="ltx_p"><span id="S4.SS3.p4.1.1" class="ltx_text ltx_font_bold">Template2: SELECT <span id="S4.SS3.p4.1.1.1" class="ltx_text">[</span>text_col1<span id="S4.SS3.p4.1.1.2" class="ltx_text">]</span> FROM table WHERE (<span id="S4.SS3.p4.1.1.3" class="ltx_text">[</span>text_col2<span id="S4.SS3.p4.1.1.4" class="ltx_text">]</span> = <span id="S4.SS3.p4.1.1.5" class="ltx_text">[</span>text2<span id="S4.SS3.p4.1.1.6" class="ltx_text">]</span>) <math id="S4.SS3.p4.1.1.m1.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S4.SS3.p4.1.1.m1.1a"><mo id="S4.SS3.p4.1.1.m1.1.1" xref="S4.SS3.p4.1.1.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S4.SS3.p4.1.1.m1.1b"><times id="S4.SS3.p4.1.1.m1.1.1.cmml" xref="S4.SS3.p4.1.1.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p4.1.1.m1.1c">\times</annotation></semantics></math> N</span></p>
</div>
<div id="S4.SS3.p5" class="ltx_para ltx_noindent">
<p id="S4.SS3.p5.1" class="ltx_p">We then investigate the impact of the number of WHERE conditions on LLM performance. Intuitively, more conditions should make it harder for LLM to execute SQL since the instruction becomes more complex.
However, the experimental results contradict this intuition, as shown in blue in Figure <a href="#S4.F11" title="Figure 11 ‣ 4.3 Template Controlled Analysis ‣ 4 Systematic Suite: Controllable Analysis ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">11</span></a>. We speculate that this counter-intuitive result stems from how LLMs actually reason: by looking up string co-occurrences rather than logically considering all conditions.</p>
</div>
<figure id="S4.F11" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S4.F11.1" class="ltx_figure ltx_figure_panel ltx_minipage ltx_align_center ltx_align_middle" style="width:194.8pt;"><img src="/html/2310.15147/assets/x13.png" id="S4.F11.1.g1" class="ltx_graphics ltx_img_square" width="422" height="380" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 10: </span>Trend of ChatGPT performance with where condition number using Template2.</figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S4.F11.2" class="ltx_figure ltx_figure_panel ltx_minipage ltx_align_center ltx_align_middle" style="width:194.8pt;"><img src="/html/2310.15147/assets/x14.png" id="S4.F11.2.g1" class="ltx_graphics ltx_img_square" width="422" height="365" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 11: </span>Trend of ChatGPT performance with the COUNT value in Template3. Only when the COUNT value is the largest or smallest, the model have good performance.</figcaption>
</figure>
</div>
</div>
</figure>
<div id="S4.SS3.p6" class="ltx_para ltx_noindent">
<p id="S4.SS3.p6.1" class="ltx_p"><span id="S4.SS3.p6.1.1" class="ltx_text ltx_font_bold">Template3: SELECT COUNT(<span id="S4.SS3.p6.1.1.1" class="ltx_text">[</span>text_col<span id="S4.SS3.p6.1.1.2" class="ltx_text">]</span>) FROM table WHERE <span id="S4.SS3.p6.1.1.3" class="ltx_text">[</span>text_col<span id="S4.SS3.p6.1.1.4" class="ltx_text">]</span> = <span id="S4.SS3.p6.1.1.5" class="ltx_text">[</span>text<span id="S4.SS3.p6.1.1.6" class="ltx_text">]</span> </span>.</p>
</div>
<div id="S4.SS3.p7" class="ltx_para ltx_noindent">
<p id="S4.SS3.p7.1" class="ltx_p">We analyze the counting ability of LLMs, which is an important numerical reasoning capability.
To avoid potential symbolic effects of SQLs, we also use the instruction style (Section <a href="#S2.SS1" title="2.1 Suite Construction ‣ 2 Synthetic Suite: Alignment with Realistic Benchmark ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2.1</span></a>) to prompt the model (e.g. Please count the number of “<span id="S4.SS3.p7.1.1" class="ltx_text">[</span>text_col<span id="S4.SS3.p7.1.2" class="ltx_text">]</span> is <span id="S4.SS3.p7.1.3" class="ltx_text">[</span>text<span id="S4.SS3.p7.1.4" class="ltx_text">]</span>”). As shown in Figure <a href="#S4.F11" title="Figure 11 ‣ 4.3 Template Controlled Analysis ‣ 4 Systematic Suite: Controllable Analysis ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">11</span></a>, whether it is zero-shot or few-shot, SQL style or instruction style, the performance of LLMs is best when the COUNT value is the smallest or the largest. When the COUNT value is in the middle, the performance of the model is almost zero.</p>
</div>
<div id="S4.SS3.p8" class="ltx_para ltx_noindent">
<p id="S4.SS3.p8.1" class="ltx_p">In the future, developers can employ the <span id="S4.SS3.p8.1.1" class="ltx_text ltx_font_smallcaps">S3Eval</span> suite to analyze the performance of LLMs with various complex SQL queries and discover new insights. They can also investigate more on the multi-step instruction prompting (Section <a href="#A3.SS5" title="C.5 Multi-step Instruction (Few-shot) ‣ Appendix C Data Demostration ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">C.5</span></a>) and chain-of-thought prompting (Section <a href="#A3.SS6" title="C.6 Chain-of-Thought SQL Execution Prompting Examples ‣ Appendix C Data Demostration ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">C.6</span></a>) to better understand LLMs.</p>
</div>
</section>
</section>
<section id="S5" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">5 </span>Related Work</h2>

<div id="S5.p1" class="ltx_para ltx_noindent">
<p id="S5.p1.1" class="ltx_p"><span id="S5.p1.1.1" class="ltx_text ltx_font_bold">Language Model Evaluation</span>
Evaluating large language models (LLMs) has garnered significant interest in the NLP community <cite class="ltx_cite ltx_citemacro_citep">(Chang et al., <a href="#bib.bib10" title="" class="ltx_ref">2023</a>)</cite>. This allows us to gain a deeper understanding of the specific capabilities and limitations of LLMs while guiding further research.
Researchers proposed MMLU <cite class="ltx_cite ltx_citemacro_citep">(Hendrycks et al., <a href="#bib.bib14" title="" class="ltx_ref">2021a</a>)</cite> to measure the knowledge acquired by a language model during pre-training. In recent years, with the development of LLMs, a series of general evaluation benchmarks have emerged. For instance, BBH <cite class="ltx_cite ltx_citemacro_citep">(Suzgun et al., <a href="#bib.bib37" title="" class="ltx_ref">2023</a>)</cite> and AGIEval <cite class="ltx_cite ltx_citemacro_citep">(Zhong et al., <a href="#bib.bib44" title="" class="ltx_ref">2023</a>)</cite> assess the reasoning ablitities. GSM8K <cite class="ltx_cite ltx_citemacro_citep">(Cobbe et al., <a href="#bib.bib13" title="" class="ltx_ref">2021</a>)</cite> evalutes the math reasoning, HumanEval <cite class="ltx_cite ltx_citemacro_citep">(Chen et al., <a href="#bib.bib11" title="" class="ltx_ref">2021a</a>)</cite>, MBPP <cite class="ltx_cite ltx_citemacro_citep">(Austin et al., <a href="#bib.bib3" title="" class="ltx_ref">2021</a>)</cite> and DS-1000 <cite class="ltx_cite ltx_citemacro_citep">(Lai et al., <a href="#bib.bib19" title="" class="ltx_ref">2023</a>)</cite> measure code capalities. Additionally, MINT <cite class="ltx_cite ltx_citemacro_citep">(Wang et al., <a href="#bib.bib41" title="" class="ltx_ref">2023</a>)</cite> and InterCode <cite class="ltx_cite ltx_citemacro_citep">(Yang et al., <a href="#bib.bib42" title="" class="ltx_ref">2023</a>)</cite> have been proposed to evaluate both code and multi-turn capability of LLMs. Our work aims to provide an evaluation suite for measuring reasoning ability.
Many previous works on long-text modeling rely on the perplexity  <cite class="ltx_cite ltx_citemacro_citep">(Sun et al., <a href="#bib.bib36" title="" class="ltx_ref">2021</a>; Peng et al., <a href="#bib.bib32" title="" class="ltx_ref">2023</a>)</cite> or performance on simple artificial tasks <cite class="ltx_cite ltx_citemacro_citep">(Li &amp; Roth, <a href="#bib.bib21" title="" class="ltx_ref">2002</a>; Berant et al., <a href="#bib.bib6" title="" class="ltx_ref">2013</a>; Mohtashami &amp; Jaggi, <a href="#bib.bib26" title="" class="ltx_ref">2023</a>)</cite>. Concurrently, ZeroSCROLLS <cite class="ltx_cite ltx_citemacro_citep">(Shaham et al., <a href="#bib.bib34" title="" class="ltx_ref">2023</a>)</cite>, L-Eval <cite class="ltx_cite ltx_citemacro_citep">(An et al., <a href="#bib.bib1" title="" class="ltx_ref">2023</a>)</cite> and LongBench <cite class="ltx_cite ltx_citemacro_citep">(Bai et al., <a href="#bib.bib5" title="" class="ltx_ref">2023b</a>)</cite> are proposed as evaluation benchmarks for long-text modeling. However, these benchmarks are built from existing public datasets and have fixed evaluation types. In contrast, <span id="S5.p1.1.2" class="ltx_text ltx_font_smallcaps">S3Eval</span> can effectively assess comprehension of infinitely long-context. Furthermore, <span id="S5.p1.1.3" class="ltx_text ltx_font_smallcaps">S3Eval</span> allows customization of settings to generate evaluation data that meets specific needs, enabling effective evaluation of model deficiencies and discovery of new insights into LLMs.</p>
</div>
<div id="S5.p2" class="ltx_para ltx_noindent">
<p id="S5.p2.1" class="ltx_p"><span id="S5.p2.1.1" class="ltx_text ltx_font_bold">Synthetic Data</span>
Investigating the use of synthetic data to explore NLP problems is worthwhile. Previous research has explored the effectiveness of synthetic tasks in various ways, such as table pretraining <cite class="ltx_cite ltx_citemacro_citep">(Liu et al., <a href="#bib.bib25" title="" class="ltx_ref">2022</a>; Jiang et al., <a href="#bib.bib17" title="" class="ltx_ref">2022</a>)</cite>, applying synchronous context-free grammars in text-to-SQL parsing <cite class="ltx_cite ltx_citemacro_citep">(Zhong et al., <a href="#bib.bib43" title="" class="ltx_ref">2020</a>)</cite>, and utilizing synthetic data for numerical reasoning. In our work, we utilize the controllability of SQL synthesis to allow users to customize the performance analysis of LLM. This approach represents the first scalable and infinite LLM benchmark. DyVal <cite class="ltx_cite ltx_citemacro_citep">(Zhu et al., <a href="#bib.bib45" title="" class="ltx_ref">2023</a>)</cite> is relevant to our work as it harnesses the structural advantage of directed acyclic graphs to dynamically generate evaluation samples with controllable complexities.</p>
</div>
</section>
<section id="S6" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">6 </span>Conclusion</h2>

<div id="S6.p1" class="ltx_para ltx_noindent">
<p id="S6.p1.1" class="ltx_p">In this paper, we have introduced <span id="S6.p1.1.1" class="ltx_text ltx_font_smallcaps">S3Eval</span>, a novel synthetic evaluation suite for LLMs using SQL execution. <span id="S6.p1.1.2" class="ltx_text ltx_font_smallcaps">S3Eval</span> represents a scalable and systematic approach to evaluate LLMs on a dynamic task.
Our experiments demonstrate strong alignment between <span id="S6.p1.1.3" class="ltx_text ltx_font_smallcaps">S3Eval</span> and traditional evaluation benchmarks.
The key innovations of <span id="S6.p1.1.4" class="ltx_text ltx_font_smallcaps">S3Eval</span> are its flexibility, allowing unlimited context length and unlimited evaluation examples, and its fine-grained, systematic nature which enables detailed analysis of model capabilities and flaws.
We believe <span id="S6.p1.1.5" class="ltx_text ltx_font_smallcaps">S3Eval</span> can serve as a valuable benchmark for LLM development and shed light on the dynamic synthetic benchmark construction.</p>
</div>
</section>
<section id="bib" class="ltx_bibliography">
<h2 class="ltx_title ltx_title_bibliography">References</h2>

<ul class="ltx_biblist">
<li id="bib.bib1" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">An et al. (2023)</span>
<span class="ltx_bibblock">
Chenxin An, Shansan Gong, Ming Zhong, Mukai Li, Jun Zhang, Lingpeng Kong, and
Xipeng Qiu.

</span>
<span class="ltx_bibblock">L-eval: Instituting standardized evaluation for long context language
models.

</span>
<span class="ltx_bibblock"><em id="bib.bib1.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2307.11088, 2023.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.48550/arXiv.2307.11088</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.48550/arXiv.2307.11088" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.48550/arXiv.2307.11088</a>.

</span>
</li>
<li id="bib.bib2" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Andonian et al. (2021)</span>
<span class="ltx_bibblock">
Alex Andonian, Quentin Anthony, Stella Biderman, Sid Black, Preetham Gali, Leo
Gao, Eric Hallahan, Josh Levy-Kramer, Connor Leahy, Lucas Nestler, Kip
Parker, Michael Pieler, Shivanshu Purohit, Tri Songz, Wang Phil, and Samuel
Weinbach.

</span>
<span class="ltx_bibblock">GPT-NeoX: Large Scale Autoregressive Language Modeling in PyTorch,
8 2021.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://www.github.com/eleutherai/gpt-neox" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://www.github.com/eleutherai/gpt-neox</a>.

</span>
</li>
<li id="bib.bib3" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Austin et al. (2021)</span>
<span class="ltx_bibblock">
Jacob Austin, Augustus Odena, Maxwell I. Nye, Maarten Bosma, Henryk
Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V.
Le, and Charles Sutton.

</span>
<span class="ltx_bibblock">Program synthesis with large language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib3.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2108.07732, 2021.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/2108.07732" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/2108.07732</a>.

</span>
</li>
<li id="bib.bib4" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bai et al. (2023a)</span>
<span class="ltx_bibblock">
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan,
Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji
Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men,
Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang,
Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang,
Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan,
Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou,
Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu.

</span>
<span class="ltx_bibblock">Qwen technical report.

</span>
<span class="ltx_bibblock"><em id="bib.bib4.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2309.16609, 2023a.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.48550/arXiv.2309.16609</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.48550/arXiv.2309.16609" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.48550/arXiv.2309.16609</a>.

</span>
</li>
<li id="bib.bib5" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bai et al. (2023b)</span>
<span class="ltx_bibblock">
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang,
Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and
Juanzi Li.

</span>
<span class="ltx_bibblock">Longbench: A bilingual, multitask benchmark for long context
understanding.

</span>
<span class="ltx_bibblock"><em id="bib.bib5.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2308.14508, 2023b.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.48550/arXiv.2308.14508</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.48550/arXiv.2308.14508" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.48550/arXiv.2308.14508</a>.

</span>
</li>
<li id="bib.bib6" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Berant et al. (2013)</span>
<span class="ltx_bibblock">
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang.

</span>
<span class="ltx_bibblock">Semantic parsing on freebase from question-answer pairs.

</span>
<span class="ltx_bibblock">In <em id="bib.bib6.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2013 Conference on Empirical Methods in
Natural Language Processing, EMNLP 2013, 18-21 October 2013, Grand Hyatt
Seattle, Seattle, Washington, USA, A meeting of SIGDAT, a Special Interest
Group of the ACL</em>, pp.  1533–1544. ACL, 2013.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://aclanthology.org/D13-1160/" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://aclanthology.org/D13-1160/</a>.

</span>
</li>
<li id="bib.bib7" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Biderman et al. (2023)</span>
<span class="ltx_bibblock">
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley,
Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit,
USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar
van der Wal.

</span>
<span class="ltx_bibblock">Pythia: A suite for analyzing large language models across training
and scaling.

</span>
<span class="ltx_bibblock">In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt,
Sivan Sabato, and Jonathan Scarlett (eds.), <em id="bib.bib7.1.1" class="ltx_emph ltx_font_italic">International Conference on
Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA</em>,
volume 202 of <em id="bib.bib7.2.2" class="ltx_emph ltx_font_italic">Proceedings of Machine Learning Research</em>, pp. 2397–2430. PMLR, 2023.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://proceedings.mlr.press/v202/biderman23a.html" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://proceedings.mlr.press/v202/biderman23a.html</a>.

</span>
</li>
<li id="bib.bib8" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bird (2006)</span>
<span class="ltx_bibblock">
Steven Bird.

</span>
<span class="ltx_bibblock">NLTK: the natural language toolkit.

</span>
<span class="ltx_bibblock">In Nicoletta Calzolari, Claire Cardie, and Pierre Isabelle (eds.),
<em id="bib.bib8.1.1" class="ltx_emph ltx_font_italic">ACL 2006, 21st International Conference on Computational Linguistics
and 44th Annual Meeting of the Association for Computational Linguistics,
Proceedings of the Conference, Sydney, Australia, 17-21 July 2006</em>. The
Association for Computer Linguistics, 2006.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.3115/1225403.1225421</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://aclanthology.org/P06-4018/" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://aclanthology.org/P06-4018/</a>.

</span>
</li>
<li id="bib.bib9" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Brown et al. (2020)</span>
<span class="ltx_bibblock">
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan,
Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda
Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom
Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens
Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott
Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec
Radford, Ilya Sutskever, and Dario Amodei.

</span>
<span class="ltx_bibblock">Language models are few-shot learners.

</span>
<span class="ltx_bibblock">In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell,
Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), <em id="bib.bib9.1.1" class="ltx_emph ltx_font_italic">Advances in Neural
Information Processing Systems 33: Annual Conference on Neural Information
Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual</em>, 2020.

</span>
<span class="ltx_bibblock">URL
<a target="_blank" href="https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html</a>.

</span>
</li>
<li id="bib.bib10" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chang et al. (2023)</span>
<span class="ltx_bibblock">
Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Kaijie Zhu, Hao Chen, Linyi Yang,
Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang,
Philip S. Yu, Qiang Yang, and Xing Xie.

</span>
<span class="ltx_bibblock">A survey on evaluation of large language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib10.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2307.03109, 2023.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.48550/arXiv.2307.03109</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.48550/arXiv.2307.03109" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.48550/arXiv.2307.03109</a>.

</span>
</li>
<li id="bib.bib11" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chen et al. (2021a)</span>
<span class="ltx_bibblock">
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé
de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas
Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov,
Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick
Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian,
Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings,
Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss,
William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor
Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher
Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa,
Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter
Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and
Wojciech Zaremba.

</span>
<span class="ltx_bibblock">Evaluating large language models trained on code.

</span>
<span class="ltx_bibblock"><em id="bib.bib11.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2107.03374, 2021a.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/2107.03374" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/2107.03374</a>.

</span>
</li>
<li id="bib.bib12" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chen et al. (2021b)</span>
<span class="ltx_bibblock">
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé
de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas
Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov,
Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick
Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian,
Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings,
Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss,
William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor
Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher
Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa,
Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter
Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and
Wojciech Zaremba.

</span>
<span class="ltx_bibblock">Evaluating large language models trained on code.

</span>
<span class="ltx_bibblock"><em id="bib.bib12.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2107.03374, 2021b.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/2107.03374" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/2107.03374</a>.

</span>
</li>
<li id="bib.bib13" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Cobbe et al. (2021)</span>
<span class="ltx_bibblock">
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz
Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano,
Christopher Hesse, and John Schulman.

</span>
<span class="ltx_bibblock">Training verifiers to solve math word problems.

</span>
<span class="ltx_bibblock"><em id="bib.bib13.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2110.14168, 2021.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/2110.14168" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/2110.14168</a>.

</span>
</li>
<li id="bib.bib14" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hendrycks et al. (2021a)</span>
<span class="ltx_bibblock">
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn
Song, and Jacob Steinhardt.

</span>
<span class="ltx_bibblock">Measuring massive multitask language understanding.

</span>
<span class="ltx_bibblock">In <em id="bib.bib14.1.1" class="ltx_emph ltx_font_italic">9th International Conference on Learning Representations,
ICLR 2021, Virtual Event, Austria, May 3-7, 2021</em>. OpenReview.net,
2021a.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://openreview.net/forum?id=d7KBjmI3GmQ" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://openreview.net/forum?id=d7KBjmI3GmQ</a>.

</span>
</li>
<li id="bib.bib15" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hendrycks et al. (2021b)</span>
<span class="ltx_bibblock">
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric
Tang, Dawn Song, and Jacob Steinhardt.

</span>
<span class="ltx_bibblock">Measuring mathematical problem solving with the MATH dataset.

</span>
<span class="ltx_bibblock">In Joaquin Vanschoren and Sai-Kit Yeung (eds.), <em id="bib.bib15.1.1" class="ltx_emph ltx_font_italic">Proceedings
of the Neural Information Processing Systems Track on Datasets and Benchmarks
1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual</em>,
2021b.

</span>
<span class="ltx_bibblock">URL
<a target="_blank" href="https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/be83ab3ecd0db773eb2dc1b0a17836a1-Abstract-round2.html" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/be83ab3ecd0db773eb2dc1b0a17836a1-Abstract-round2.html</a>.

</span>
</li>
<li id="bib.bib16" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hoffmann et al. (2022)</span>
<span class="ltx_bibblock">
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor
Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes
Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den
Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich
Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre.

</span>
<span class="ltx_bibblock">Training compute-optimal large language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib16.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2203.15556, 2022.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.48550/arXiv.2203.15556</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.48550/arXiv.2203.15556" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.48550/arXiv.2203.15556</a>.

</span>
</li>
<li id="bib.bib17" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Jiang et al. (2022)</span>
<span class="ltx_bibblock">
Zhengbao Jiang, Yi Mao, Pengcheng He, Graham Neubig, and Weizhu Chen.

</span>
<span class="ltx_bibblock">Omnitab: Pretraining with natural and synthetic data for few-shot
table-based question answering.

</span>
<span class="ltx_bibblock">In Marine Carpuat, Marie-Catherine de Marneffe, and Iván
Vladimir Meza Ruíz (eds.), <em id="bib.bib17.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2022 Conference of
the North American Chapter of the Association for Computational Linguistics:
Human Language Technologies, NAACL 2022, Seattle, WA, United States, July
10-15, 2022</em>, pp.  932–942. Association for Computational Linguistics,
2022.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.18653/v1/2022.naacl-main.68</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.18653/v1/2022.naacl-main.68" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.18653/v1/2022.naacl-main.68</a>.

</span>
</li>
<li id="bib.bib18" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kaplan et al. (2020)</span>
<span class="ltx_bibblock">
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon
Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei.

</span>
<span class="ltx_bibblock">Scaling laws for neural language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib18.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2001.08361, 2020.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/2001.08361" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/2001.08361</a>.

</span>
</li>
<li id="bib.bib19" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lai et al. (2023)</span>
<span class="ltx_bibblock">
Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke
Zettlemoyer, Wen-Tau Yih, Daniel Fried, Sida I. Wang, and Tao Yu.

</span>
<span class="ltx_bibblock">DS-1000: A natural and reliable benchmark for data science code
generation.

</span>
<span class="ltx_bibblock">In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt,
Sivan Sabato, and Jonathan Scarlett (eds.), <em id="bib.bib19.1.1" class="ltx_emph ltx_font_italic">International Conference on
Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA</em>,
volume 202 of <em id="bib.bib19.2.2" class="ltx_emph ltx_font_italic">Proceedings of Machine Learning Research</em>, pp. 18319–18345. PMLR, 2023.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://proceedings.mlr.press/v202/lai23b.html" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://proceedings.mlr.press/v202/lai23b.html</a>.

</span>
</li>
<li id="bib.bib20" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Li et al. (2023a)</span>
<span class="ltx_bibblock">
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov,
Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu,
Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig
Davaadorj, Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas
Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi,
Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy V,
Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan
Dey, Zhihan Zhang, Nour Moustafa-Fahmy, Urvashi Bhattacharyya, Wenhao Yu,
Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov,
Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger,
Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer
Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor,
Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite,
Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro
von Werra, and Harm de Vries.

</span>
<span class="ltx_bibblock">Starcoder: may the source be with you!

</span>
<span class="ltx_bibblock"><em id="bib.bib20.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2305.06161, 2023a.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.48550/arXiv.2305.06161</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.48550/arXiv.2305.06161" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.48550/arXiv.2305.06161</a>.

</span>
</li>
<li id="bib.bib21" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Li &amp; Roth (2002)</span>
<span class="ltx_bibblock">
Xin Li and Dan Roth.

</span>
<span class="ltx_bibblock">Learning question classifiers.

</span>
<span class="ltx_bibblock">In <em id="bib.bib21.1.1" class="ltx_emph ltx_font_italic">19th International Conference on Computational Linguistics,
COLING 2002, Howard International House and Academia Sinica, Taipei,
Taiwan, August 24 - September 1, 2002</em>, 2002.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://aclanthology.org/C02-1150/" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://aclanthology.org/C02-1150/</a>.

</span>
</li>
<li id="bib.bib22" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Li et al. (2023b)</span>
<span class="ltx_bibblock">
Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya
Gunasekar, and Yin Tat Lee.

</span>
<span class="ltx_bibblock">Textbooks are all you need II: phi-1.5 technical report.

</span>
<span class="ltx_bibblock"><em id="bib.bib22.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2309.05463, 2023b.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.48550/arXiv.2309.05463</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.48550/arXiv.2309.05463" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.48550/arXiv.2309.05463</a>.

</span>
</li>
<li id="bib.bib23" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Liu et al. (2023a)</span>
<span class="ltx_bibblock">
Nelson F. Liu, Tony Lee, Robin Jia, and Percy Liang.

</span>
<span class="ltx_bibblock">Do question answering modeling improvements hold across benchmarks?

</span>
<span class="ltx_bibblock">In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.),
<em id="bib.bib23.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 61st Annual Meeting of the Association for
Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto,
Canada, July 9-14, 2023</em>, pp.  13186–13218. Association for Computational
Linguistics, 2023a.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.18653/v1/2023.acl-long.736</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.18653/v1/2023.acl-long.736" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.18653/v1/2023.acl-long.736</a>.

</span>
</li>
<li id="bib.bib24" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Liu et al. (2023b)</span>
<span class="ltx_bibblock">
Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua,
Fabio Petroni, and Percy Liang.

</span>
<span class="ltx_bibblock">Lost in the middle: How language models use long contexts.

</span>
<span class="ltx_bibblock"><em id="bib.bib24.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2307.03172, 2023b.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.48550/arXiv.2307.03172</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.48550/arXiv.2307.03172" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.48550/arXiv.2307.03172</a>.

</span>
</li>
<li id="bib.bib25" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Liu et al. (2022)</span>
<span class="ltx_bibblock">
Qian Liu, Bei Chen, Jiaqi Guo, Morteza Ziyadi, Zeqi Lin, Weizhu Chen, and
Jian-Guang Lou.

</span>
<span class="ltx_bibblock">TAPEX: table pre-training via learning a neural SQL executor.

</span>
<span class="ltx_bibblock">In <em id="bib.bib25.1.1" class="ltx_emph ltx_font_italic">The Tenth International Conference on Learning
Representations, ICLR 2022, Virtual Event, April 25-29, 2022</em>.
OpenReview.net, 2022.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://openreview.net/forum?id=O50443AsCP" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://openreview.net/forum?id=O50443AsCP</a>.

</span>
</li>
<li id="bib.bib26" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Mohtashami &amp; Jaggi (2023)</span>
<span class="ltx_bibblock">
Amirkeivan Mohtashami and Martin Jaggi.

</span>
<span class="ltx_bibblock">Landmark attention: Random-access infinite context length for
transformers.

</span>
<span class="ltx_bibblock"><em id="bib.bib26.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2305.16300, 2023.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.48550/arXiv.2305.16300</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.48550/arXiv.2305.16300" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.48550/arXiv.2305.16300</a>.

</span>
</li>
<li id="bib.bib27" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Nijkamp et al. (2022)</span>
<span class="ltx_bibblock">
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio
Savarese, and Caiming Xiong.

</span>
<span class="ltx_bibblock">A conversational paradigm for program synthesis.

</span>
<span class="ltx_bibblock"><em id="bib.bib27.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2203.13474, 2022.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.48550/arXiv.2203.13474</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.48550/arXiv.2203.13474" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.48550/arXiv.2203.13474</a>.

</span>
</li>
<li id="bib.bib28" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Nijkamp et al. (2023)</span>
<span class="ltx_bibblock">
Erik Nijkamp, Tian Xie, Hiroaki Hayashi, Bo Pang, Congying Xia, Chen Xing,
Jesse Vig, Semih Yavuz, Philippe Laban, Ben Krause, Senthil Purushwalkam,
Tong Niu, Wojciech Kryscinski, Lidiya Murakhovs’ka, Prafulla Kumar Choubey,
Alexander R. Fabbri, Ye Liu, Rui Meng, Lifu Tu, Meghana Bhat, Chien-Sheng
Wu, Silvio Savarese, Yingbo Zhou, Shafiq Joty, and Caiming Xiong.

</span>
<span class="ltx_bibblock">Xgen-7b technical report.

</span>
<span class="ltx_bibblock"><em id="bib.bib28.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2309.03450, 2023.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.48550/arXiv.2309.03450</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.48550/arXiv.2309.03450" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.48550/arXiv.2309.03450</a>.

</span>
</li>
<li id="bib.bib29" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">OpenCompass (2023)</span>
<span class="ltx_bibblock">
OpenCompass.

</span>
<span class="ltx_bibblock">Opencompass: A universal evaluation platform for foundation models.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://github.com/open-compass/opencompass" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/open-compass/opencompass</a>, 2023.

</span>
</li>
<li id="bib.bib30" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Pasupat &amp; Liang (2015)</span>
<span class="ltx_bibblock">
Panupong Pasupat and Percy Liang.

</span>
<span class="ltx_bibblock">Compositional semantic parsing on semi-structured tables.

</span>
<span class="ltx_bibblock">In <em id="bib.bib30.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 53rd Annual Meeting of the Association
for Computational Linguistics and the 7th International Joint Conference on
Natural Language Processing of the Asian Federation of Natural Language
Processing, ACL 2015, July 26-31, 2015, Beijing, China, Volume 1: Long
Papers</em>, pp.  1470–1480. The Association for Computer Linguistics, 2015.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.3115/v1/p15-1142</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.3115/v1/p15-1142" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.3115/v1/p15-1142</a>.

</span>
</li>
<li id="bib.bib31" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Penedo et al. (2023)</span>
<span class="ltx_bibblock">
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru,
Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei,
and Julien Launay.

</span>
<span class="ltx_bibblock">The refinedweb dataset for falcon LLM: outperforming curated
corpora with web data, and web data only.

</span>
<span class="ltx_bibblock"><em id="bib.bib31.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2306.01116, 2023.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.48550/arXiv.2306.01116</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.48550/arXiv.2306.01116" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.48550/arXiv.2306.01116</a>.

</span>
</li>
<li id="bib.bib32" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Peng et al. (2023)</span>
<span class="ltx_bibblock">
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole.

</span>
<span class="ltx_bibblock">Yarn: Efficient context window extension of large language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib32.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2309.00071, 2023.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.48550/arXiv.2309.00071</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.48550/arXiv.2309.00071" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.48550/arXiv.2309.00071</a>.

</span>
</li>
<li id="bib.bib33" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Rozière et al. (2023)</span>
<span class="ltx_bibblock">
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat,
Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy
Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt,
Cristian Canton-Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre
Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas
Usunier, Thomas Scialom, and Gabriel Synnaeve.

</span>
<span class="ltx_bibblock">Code llama: Open foundation models for code.

</span>
<span class="ltx_bibblock"><em id="bib.bib33.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2308.12950, 2023.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.48550/arXiv.2308.12950</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.48550/arXiv.2308.12950" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.48550/arXiv.2308.12950</a>.

</span>
</li>
<li id="bib.bib34" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Shaham et al. (2023)</span>
<span class="ltx_bibblock">
Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, and Omer Levy.

</span>
<span class="ltx_bibblock">Zeroscrolls: A zero-shot benchmark for long text understanding.

</span>
<span class="ltx_bibblock"><em id="bib.bib34.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2305.14196, 2023.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.48550/arXiv.2305.14196</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.48550/arXiv.2305.14196" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.48550/arXiv.2305.14196</a>.

</span>
</li>
<li id="bib.bib35" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Shi et al. (2020)</span>
<span class="ltx_bibblock">
Tianze Shi, Chen Zhao, Jordan L. Boyd-Graber, Hal Daumé III, and
Lillian Lee.

</span>
<span class="ltx_bibblock">On the potential of lexico-logical alignments for semantic parsing to
SQL queries.

</span>
<span class="ltx_bibblock">In Trevor Cohn, Yulan He, and Yang Liu (eds.), <em id="bib.bib35.1.1" class="ltx_emph ltx_font_italic">Findings of the
Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20
November 2020</em>, volume EMNLP 2020 of <em id="bib.bib35.2.2" class="ltx_emph ltx_font_italic">Findings of ACL</em>, pp. 1849–1864. Association for Computational Linguistics, 2020.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.18653/v1/2020.findings-emnlp.167</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.18653/v1/2020.findings-emnlp.167" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.18653/v1/2020.findings-emnlp.167</a>.

</span>
</li>
<li id="bib.bib36" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Sun et al. (2021)</span>
<span class="ltx_bibblock">
Simeng Sun, Kalpesh Krishna, Andrew Mattarella-Micke, and Mohit Iyyer.

</span>
<span class="ltx_bibblock">Do long-range language models actually use long-range context?

</span>
<span class="ltx_bibblock">In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and
Scott Wen-tau Yih (eds.), <em id="bib.bib36.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2021 Conference on
Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event
/ Punta Cana, Dominican Republic, 7-11 November, 2021</em>, pp.  807–822.
Association for Computational Linguistics, 2021.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.18653/v1/2021.emnlp-main.62</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.18653/v1/2021.emnlp-main.62" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.18653/v1/2021.emnlp-main.62</a>.

</span>
</li>
<li id="bib.bib37" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Suzgun et al. (2023)</span>
<span class="ltx_bibblock">
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann,
Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed Chi, Denny Zhou,
and Jason Wei.

</span>
<span class="ltx_bibblock">Challenging big-bench tasks and whether chain-of-thought can solve
them.

</span>
<span class="ltx_bibblock">In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.),
<em id="bib.bib37.1.1" class="ltx_emph ltx_font_italic">Findings of the Association for Computational Linguistics: ACL 2023,
Toronto, Canada, July 9-14, 2023</em>, pp.  13003–13051. Association for
Computational Linguistics, 2023.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.18653/v1/2023.findings-acl.824</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.18653/v1/2023.findings-acl.824" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.18653/v1/2023.findings-acl.824</a>.

</span>
</li>
<li id="bib.bib38" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Team (2023)</span>
<span class="ltx_bibblock">
InternLM Team.

</span>
<span class="ltx_bibblock">Internlm: A multilingual language model with progressively enhanced
capabilities.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://github.com/InternLM/InternLM" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/InternLM/InternLM</a>, 2023.

</span>
</li>
<li id="bib.bib39" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Touvron et al. (2023a)</span>
<span class="ltx_bibblock">
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne
Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric
Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave,
and Guillaume Lample.

</span>
<span class="ltx_bibblock">Llama: Open and efficient foundation language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib39.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2302.13971, 2023a.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.48550/arXiv.2302.13971</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.48550/arXiv.2302.13971" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.48550/arXiv.2302.13971</a>.

</span>
</li>
<li id="bib.bib40" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Touvron et al. (2023b)</span>
<span class="ltx_bibblock">
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine
Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale,
Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem
Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,
Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar
Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa,
Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux,
Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier
Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew
Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan
Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang,
Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan
Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang,
Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom.

</span>
<span class="ltx_bibblock">Llama 2: Open foundation and fine-tuned chat models.

</span>
<span class="ltx_bibblock"><em id="bib.bib40.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2307.09288, 2023b.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.48550/arXiv.2307.09288</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.48550/arXiv.2307.09288" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.48550/arXiv.2307.09288</a>.

</span>
</li>
<li id="bib.bib41" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2023)</span>
<span class="ltx_bibblock">
Xingyao Wang, Zihan Wang, Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng, and
Heng Ji.

</span>
<span class="ltx_bibblock">MINT: evaluating llms in multi-turn interaction with tools and
language feedback.

</span>
<span class="ltx_bibblock"><em id="bib.bib41.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2309.10691, 2023.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.48550/arXiv.2309.10691</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.48550/arXiv.2309.10691" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.48550/arXiv.2309.10691</a>.

</span>
</li>
<li id="bib.bib42" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yang et al. (2023)</span>
<span class="ltx_bibblock">
John Yang, Akshara Prabhakar, Karthik Narasimhan, and Shunyu Yao.

</span>
<span class="ltx_bibblock">Intercode: Standardizing and benchmarking interactive coding with
execution feedback.

</span>
<span class="ltx_bibblock"><em id="bib.bib42.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2306.14898, 2023.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.48550/arXiv.2306.14898</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.48550/arXiv.2306.14898" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.48550/arXiv.2306.14898</a>.

</span>
</li>
<li id="bib.bib43" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhong et al. (2020)</span>
<span class="ltx_bibblock">
Victor Zhong, Mike Lewis, Sida I Wang, and Luke Zettlemoyer.

</span>
<span class="ltx_bibblock">Grounded adaptation for zero-shot executable semantic parsing.

</span>
<span class="ltx_bibblock">In <em id="bib.bib43.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2020 Conference on Empirical Methods in
Natural Language Processing (EMNLP)</em>, pp.  6869–6882, 2020.

</span>
</li>
<li id="bib.bib44" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhong et al. (2023)</span>
<span class="ltx_bibblock">
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin
Saied, Weizhu Chen, and Nan Duan.

</span>
<span class="ltx_bibblock">Agieval: A human-centric benchmark for evaluating foundation
models.

</span>
<span class="ltx_bibblock"><em id="bib.bib44.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2304.06364, 2023.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.48550/arXiv.2304.06364</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.48550/arXiv.2304.06364" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.48550/arXiv.2304.06364</a>.

</span>
</li>
<li id="bib.bib45" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhu et al. (2023)</span>
<span class="ltx_bibblock">
Kaijie Zhu, Jiaao Chen, Jindong Wang, Neil Zhenqiang Gong, Diyi Yang, and Xing
Xie.

</span>
<span class="ltx_bibblock">Dyval: Graph-informed dynamic evaluation of large language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib45.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2309.17167, 2023.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.48550/arXiv.2309.17167</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.48550/arXiv.2309.17167" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.48550/arXiv.2309.17167</a>.

</span>
</li>
</ul>
</section>
<div class="ltx_pagination ltx_role_newpage"></div>
<div id="p2" class="ltx_para">
<span id="p2.1" class="ltx_ERROR undefined">\etocdepthtag</span>
<p id="p2.2" class="ltx_p">.tocappendix
<span id="p2.2.1" class="ltx_ERROR undefined">\etocsettagdepth</span>chapternone
<span id="p2.2.2" class="ltx_ERROR undefined">\etocsettagdepth</span>appendixsubsection</p>
</div>
<nav class="ltx_TOC ltx_list_toc ltx_toc_toc"><h6 class="ltx_title ltx_title_contents">Contents</h6>
<ol class="ltx_toclist">
<li class="ltx_tocentry ltx_tocentry_section"><a href="#S1" title="In S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">1 </span>Introduction</span></a></li>
<li class="ltx_tocentry ltx_tocentry_section">
<a href="#S2" title="In S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">2 </span>Synthetic Suite: Alignment with Realistic Benchmark</span></a>
<ol class="ltx_toclist ltx_toclist_section">
<li class="ltx_tocentry ltx_tocentry_subsection"><a href="#S2.SS1" title="In 2 Synthetic Suite: Alignment with Realistic Benchmark ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">2.1 </span>Suite Construction</span></a></li>
<li class="ltx_tocentry ltx_tocentry_subsection"><a href="#S2.SS2" title="In 2 Synthetic Suite: Alignment with Realistic Benchmark ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">2.2 </span>Alignment on Scaling Law</span></a></li>
<li class="ltx_tocentry ltx_tocentry_subsection"><a href="#S2.SS3" title="In 2 Synthetic Suite: Alignment with Realistic Benchmark ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">2.3 </span>Alignment on Benchmark Performance</span></a></li>
</ol>
</li>
<li class="ltx_tocentry ltx_tocentry_section">
<a href="#S3" title="In S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">3 </span>Scalable Suite: Unlimited Evaluation Resources</span></a>
<ol class="ltx_toclist ltx_toclist_section">
<li class="ltx_tocentry ltx_tocentry_subsection"><a href="#S3.SS1" title="In 3 Scalable Suite: Unlimited Evaluation Resources ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">3.1 </span>Scalable Number of Evaluation Examples</span></a></li>
<li class="ltx_tocentry ltx_tocentry_subsection"><a href="#S3.SS2" title="In 3 Scalable Suite: Unlimited Evaluation Resources ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">3.2 </span>Scalable Length of Evaluation Examples</span></a></li>
</ol>
</li>
<li class="ltx_tocentry ltx_tocentry_section">
<a href="#S4" title="In S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">4 </span>Systematic Suite: Controllable Analysis</span></a>
<ol class="ltx_toclist ltx_toclist_section">
<li class="ltx_tocentry ltx_tocentry_subsection"><a href="#S4.SS1" title="In 4 Systematic Suite: Controllable Analysis ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">4.1 </span>Answer Position Analysis</span></a></li>
<li class="ltx_tocentry ltx_tocentry_subsection"><a href="#S4.SS2" title="In 4 Systematic Suite: Controllable Analysis ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">4.2 </span>Reasoning Type Analysis</span></a></li>
<li class="ltx_tocentry ltx_tocentry_subsection"><a href="#S4.SS3" title="In 4 Systematic Suite: Controllable Analysis ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">4.3 </span>Template Controlled Analysis</span></a></li>
</ol>
</li>
<li class="ltx_tocentry ltx_tocentry_section"><a href="#S5" title="In S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">5 </span>Related Work</span></a></li>
<li class="ltx_tocentry ltx_tocentry_section"><a href="#S6" title="In S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">6 </span>Conclusion</span></a></li>
<li class="ltx_tocentry ltx_tocentry_appendix">
<a href="#A1" title="In S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">A </span>Evaluation Experiments Results</span></a>
<ol class="ltx_toclist ltx_toclist_appendix">
<li class="ltx_tocentry ltx_tocentry_subsection"><a href="#A1.SS1" title="In Appendix A Evaluation Experiments Results ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">A.1 </span>Overall Performance</span></a></li>
<li class="ltx_tocentry ltx_tocentry_subsection"><a href="#A1.SS2" title="In Appendix A Evaluation Experiments Results ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">A.2 </span>Reliability Experiments</span></a></li>
<li class="ltx_tocentry ltx_tocentry_subsection"><a href="#A1.SS3" title="In Appendix A Evaluation Experiments Results ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">A.3 </span>Other SQL Prompting Styles</span></a></li>
<li class="ltx_tocentry ltx_tocentry_subsection"><a href="#A1.SS4" title="In Appendix A Evaluation Experiments Results ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">A.4 </span>Long-Context Experiments</span></a></li>
</ol>
</li>
<li class="ltx_tocentry ltx_tocentry_appendix">
<a href="#A2" title="In S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">B </span>Controllable Analysis Results</span></a>
<ol class="ltx_toclist ltx_toclist_appendix">
<li class="ltx_tocentry ltx_tocentry_subsection"><a href="#A2.SS1" title="In Appendix B Controllable Analysis Results ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">B.1 </span>Answer Position Analysis</span></a></li>
<li class="ltx_tocentry ltx_tocentry_subsection"><a href="#A2.SS2" title="In Appendix B Controllable Analysis Results ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">B.2 </span>Input Format Analysis</span></a></li>
<li class="ltx_tocentry ltx_tocentry_subsection"><a href="#A2.SS3" title="In Appendix B Controllable Analysis Results ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">B.3 </span>SQL Keywords Analysis</span></a></li>
<li class="ltx_tocentry ltx_tocentry_subsection"><a href="#A2.SS4" title="In Appendix B Controllable Analysis Results ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">B.4 </span>SQL Attribute Analysis</span></a></li>
</ol>
</li>
<li class="ltx_tocentry ltx_tocentry_appendix">
<a href="#A3" title="In S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">C </span>Data Demostration</span></a>
<ol class="ltx_toclist ltx_toclist_appendix">
<li class="ltx_tocentry ltx_tocentry_subsection"><a href="#A3.SS1" title="In Appendix C Data Demostration ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">C.1 </span>Dense and Sparse Examples</span></a></li>
<li class="ltx_tocentry ltx_tocentry_subsection"><a href="#A3.SS2" title="In Appendix C Data Demostration ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">C.2 </span>SQL Template</span></a></li>
<li class="ltx_tocentry ltx_tocentry_subsection"><a href="#A3.SS3" title="In Appendix C Data Demostration ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">C.3 </span>Table Input Format</span></a></li>
<li class="ltx_tocentry ltx_tocentry_subsection"><a href="#A3.SS4" title="In Appendix C Data Demostration ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">C.4 </span>SQL Execution Examples (Few-shot)</span></a></li>
<li class="ltx_tocentry ltx_tocentry_subsection"><a href="#A3.SS5" title="In Appendix C Data Demostration ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">C.5 </span>Multi-step Instruction (Few-shot)</span></a></li>
<li class="ltx_tocentry ltx_tocentry_subsection"><a href="#A3.SS6" title="In Appendix C Data Demostration ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">C.6 </span>Chain-of-Thought SQL Execution Prompting Examples</span></a></li>
<li class="ltx_tocentry ltx_tocentry_subsection"><a href="#A3.SS7" title="In Appendix C Data Demostration ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">C.7 </span>Real Table SQL Execution (Few-shot)</span></a></li>
<li class="ltx_tocentry ltx_tocentry_subsection"><a href="#A3.SS8" title="In Appendix C Data Demostration ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">C.8 </span>Real Table Question Answering (Few-shot)</span></a></li>
</ol>
</li>
<li class="ltx_tocentry ltx_tocentry_appendix">
<a href="#A4" title="In S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">D </span>Experiments Settings Details</span></a>
<ol class="ltx_toclist ltx_toclist_appendix">
<li class="ltx_tocentry ltx_tocentry_subsection"><a href="#A4.SS1" title="In Appendix D Experiments Settings Details ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">D.1 </span>Setting Description</span></a></li>
<li class="ltx_tocentry ltx_tocentry_subsection"><a href="#A4.SS2" title="In Appendix D Experiments Settings Details ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">D.2 </span>General Setting</span></a></li>
<li class="ltx_tocentry ltx_tocentry_subsection"><a href="#A4.SS3" title="In Appendix D Experiments Settings Details ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">D.3 </span>LLMs Used In This Paper</span></a></li>
<li class="ltx_tocentry ltx_tocentry_subsection"><a href="#A4.SS4" title="In Appendix D Experiments Settings Details ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_title"><span class="ltx_tag ltx_tag_ref">D.4 </span>Markdown vs. Flatten Setting Experiments</span></a></li>
</ol>
</li>
</ol></nav>
<section id="A1" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix A </span>Evaluation Experiments Results</h2>

<section id="A1.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.1 </span>Overall Performance</h3>

<div id="A1.SS1.p1" class="ltx_para ltx_noindent">
<p id="A1.SS1.p1.1" class="ltx_p">The detail performance are shown in Table <a href="#A1.T3" title="Table 3 ‣ A.1 Overall Performance ‣ Appendix A Evaluation Experiments Results ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>.</p>
</div>
<figure id="A1.T3" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 3: </span>SQL Execution Task Performance on different LLMs.</figcaption>
<table id="A1.T3.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="A1.T3.1.1" class="ltx_tr">
<td id="A1.T3.1.1.1" class="ltx_td ltx_border_tt"></td>
<td id="A1.T3.1.1.2" class="ltx_td ltx_border_tt"></td>
<td id="A1.T3.1.1.3" class="ltx_td ltx_align_center ltx_border_tt" colspan="2"><span id="A1.T3.1.1.3.1" class="ltx_text ltx_font_bold">Synthetic Task</span></td>
<td id="A1.T3.1.1.4" class="ltx_td ltx_align_center ltx_border_tt" colspan="2"><span id="A1.T3.1.1.4.1" class="ltx_text ltx_font_bold">Realistic Benchmark</span>
</td>
<td id="A1.T3.1.1.5" class="ltx_td ltx_border_tt"></td>
</tr>
<tr id="A1.T3.1.2" class="ltx_tr">
<td id="A1.T3.1.2.1" class="ltx_td"></td>
<td id="A1.T3.1.2.2" class="ltx_td"></td>
<td id="A1.T3.1.2.3" class="ltx_td ltx_align_center ltx_border_t">
<span id="A1.T3.1.2.3.1" class="ltx_text ltx_font_smallcaps">S3Eval</span>-Easy</td>
<td id="A1.T3.1.2.4" class="ltx_td ltx_align_center ltx_border_t">
<span id="A1.T3.1.2.4.1" class="ltx_text ltx_font_smallcaps">S3Eval</span>-General</td>
<td id="A1.T3.1.2.5" class="ltx_td ltx_align_center ltx_border_t">WTQ</td>
<td id="A1.T3.1.2.6" class="ltx_td ltx_align_center ltx_border_t">Reasoning Task</td>
<td id="A1.T3.1.2.7" class="ltx_td"></td>
</tr>
<tr id="A1.T3.1.3" class="ltx_tr">
<td id="A1.T3.1.3.1" class="ltx_td ltx_align_center ltx_border_t" rowspan="19"><span id="A1.T3.1.3.1.1" class="ltx_text">LLM</span></td>
<td id="A1.T3.1.3.2" class="ltx_td ltx_align_left ltx_border_t">GPT-4</td>
<td id="A1.T3.1.3.3" class="ltx_td ltx_align_center ltx_border_t"><span id="A1.T3.1.3.3.1" class="ltx_text ltx_font_bold">99.4</span></td>
<td id="A1.T3.1.3.4" class="ltx_td ltx_align_center ltx_border_t"><span id="A1.T3.1.3.4.1" class="ltx_text ltx_font_bold">63.1</span></td>
<td id="A1.T3.1.3.5" class="ltx_td ltx_align_center ltx_border_t">70.8</td>
<td id="A1.T3.1.3.6" class="ltx_td ltx_align_center ltx_border_t">86.7</td>
<td id="A1.T3.1.3.7" class="ltx_td ltx_border_t"></td>
</tr>
<tr id="A1.T3.1.4" class="ltx_tr">
<td id="A1.T3.1.4.1" class="ltx_td ltx_align_left">ChatGPT</td>
<td id="A1.T3.1.4.2" class="ltx_td ltx_align_center">97.0</td>
<td id="A1.T3.1.4.3" class="ltx_td ltx_align_center">47.2</td>
<td id="A1.T3.1.4.4" class="ltx_td ltx_align_center">62.0</td>
<td id="A1.T3.1.4.5" class="ltx_td ltx_align_center">70.1</td>
<td id="A1.T3.1.4.6" class="ltx_td"></td>
</tr>
<tr id="A1.T3.1.5" class="ltx_tr">
<td id="A1.T3.1.5.1" class="ltx_td ltx_align_left">Claude-1</td>
<td id="A1.T3.1.5.2" class="ltx_td ltx_align_center">98.2</td>
<td id="A1.T3.1.5.3" class="ltx_td ltx_align_center">44.3</td>
<td id="A1.T3.1.5.4" class="ltx_td ltx_align_center">63.4</td>
<td id="A1.T3.1.5.5" class="ltx_td ltx_align_center">67.3</td>
<td id="A1.T3.1.5.6" class="ltx_td"></td>
</tr>
<tr id="A1.T3.1.6" class="ltx_tr">
<td id="A1.T3.1.6.1" class="ltx_td ltx_align_left">Llama-2-70B</td>
<td id="A1.T3.1.6.2" class="ltx_td ltx_align_center">94.2</td>
<td id="A1.T3.1.6.3" class="ltx_td ltx_align_center">41.3</td>
<td id="A1.T3.1.6.4" class="ltx_td ltx_align_center">55.9</td>
<td id="A1.T3.1.6.5" class="ltx_td ltx_align_center">64.9</td>
<td id="A1.T3.1.6.6" class="ltx_td"></td>
</tr>
<tr id="A1.T3.1.7" class="ltx_tr">
<td id="A1.T3.1.7.1" class="ltx_td ltx_align_left">Mistral-7B</td>
<td id="A1.T3.1.7.2" class="ltx_td ltx_align_center">87.4</td>
<td id="A1.T3.1.7.3" class="ltx_td ltx_align_center">34.3</td>
<td id="A1.T3.1.7.4" class="ltx_td ltx_align_center">55.7</td>
<td id="A1.T3.1.7.5" class="ltx_td ltx_align_center">53.7</td>
<td id="A1.T3.1.7.6" class="ltx_td"></td>
</tr>
<tr id="A1.T3.1.8" class="ltx_tr">
<td id="A1.T3.1.8.1" class="ltx_td ltx_align_left">Llama2-13B</td>
<td id="A1.T3.1.8.2" class="ltx_td ltx_align_center">75.0</td>
<td id="A1.T3.1.8.3" class="ltx_td ltx_align_center">30.9</td>
<td id="A1.T3.1.8.4" class="ltx_td ltx_align_center">49.2</td>
<td id="A1.T3.1.8.5" class="ltx_td ltx_align_center">45.6</td>
<td id="A1.T3.1.8.6" class="ltx_td"></td>
</tr>
<tr id="A1.T3.1.9" class="ltx_tr">
<td id="A1.T3.1.9.1" class="ltx_td ltx_align_left">InternLM-20B</td>
<td id="A1.T3.1.9.2" class="ltx_td ltx_align_center">78.0</td>
<td id="A1.T3.1.9.3" class="ltx_td ltx_align_center">32.3</td>
<td id="A1.T3.1.9.4" class="ltx_td ltx_align_center">49.4</td>
<td id="A1.T3.1.9.5" class="ltx_td ltx_align_center">52.5</td>
<td id="A1.T3.1.9.6" class="ltx_td"></td>
</tr>
<tr id="A1.T3.1.10" class="ltx_tr">
<td id="A1.T3.1.10.1" class="ltx_td ltx_align_left">Qwen-14B</td>
<td id="A1.T3.1.10.2" class="ltx_td ltx_align_center">71.8</td>
<td id="A1.T3.1.10.3" class="ltx_td ltx_align_center">25.8</td>
<td id="A1.T3.1.10.4" class="ltx_td ltx_align_center">46.7</td>
<td id="A1.T3.1.10.5" class="ltx_td ltx_align_center">53.7</td>
<td id="A1.T3.1.10.6" class="ltx_td"></td>
</tr>
<tr id="A1.T3.1.11" class="ltx_tr">
<td id="A1.T3.1.11.1" class="ltx_td ltx_align_left">Llama-2-7B</td>
<td id="A1.T3.1.11.2" class="ltx_td ltx_align_center">54.2</td>
<td id="A1.T3.1.11.3" class="ltx_td ltx_align_center">23.8</td>
<td id="A1.T3.1.11.4" class="ltx_td ltx_align_center">40.6</td>
<td id="A1.T3.1.11.5" class="ltx_td ltx_align_center">38.2</td>
<td id="A1.T3.1.11.6" class="ltx_td"></td>
</tr>
<tr id="A1.T3.1.12" class="ltx_tr">
<td id="A1.T3.1.12.1" class="ltx_td ltx_align_left">Qwen-7B</td>
<td id="A1.T3.1.12.2" class="ltx_td ltx_align_center">56.4</td>
<td id="A1.T3.1.12.3" class="ltx_td ltx_align_center">19.4</td>
<td id="A1.T3.1.12.4" class="ltx_td ltx_align_center">41.2</td>
<td id="A1.T3.1.12.5" class="ltx_td ltx_align_center">45.2</td>
<td id="A1.T3.1.12.6" class="ltx_td"></td>
</tr>
<tr id="A1.T3.1.13" class="ltx_tr">
<td id="A1.T3.1.13.1" class="ltx_td ltx_align_left">Xgen-7B</td>
<td id="A1.T3.1.13.2" class="ltx_td ltx_align_center">55.2</td>
<td id="A1.T3.1.13.3" class="ltx_td ltx_align_center">24.6</td>
<td id="A1.T3.1.13.4" class="ltx_td ltx_align_center">36.3</td>
<td id="A1.T3.1.13.5" class="ltx_td ltx_align_center">34.5</td>
<td id="A1.T3.1.13.6" class="ltx_td"></td>
</tr>
<tr id="A1.T3.1.14" class="ltx_tr">
<td id="A1.T3.1.14.1" class="ltx_td ltx_align_left">Internlm-7B</td>
<td id="A1.T3.1.14.2" class="ltx_td ltx_align_center">41.6</td>
<td id="A1.T3.1.14.3" class="ltx_td ltx_align_center">18.5</td>
<td id="A1.T3.1.14.4" class="ltx_td ltx_align_center">27.5</td>
<td id="A1.T3.1.14.5" class="ltx_td ltx_align_center">37.0</td>
<td id="A1.T3.1.14.6" class="ltx_td"></td>
</tr>
<tr id="A1.T3.1.15" class="ltx_tr">
<td id="A1.T3.1.15.1" class="ltx_td ltx_align_left">Phi-1_5</td>
<td id="A1.T3.1.15.2" class="ltx_td ltx_align_center">27.6</td>
<td id="A1.T3.1.15.3" class="ltx_td ltx_align_center">16.1</td>
<td id="A1.T3.1.15.4" class="ltx_td ltx_align_center">22.1</td>
<td id="A1.T3.1.15.5" class="ltx_td ltx_align_center">30.0</td>
<td id="A1.T3.1.15.6" class="ltx_td"></td>
</tr>
<tr id="A1.T3.1.16" class="ltx_tr">
<td id="A1.T3.1.16.1" class="ltx_td ltx_align_left">Stablelm-7B</td>
<td id="A1.T3.1.16.2" class="ltx_td ltx_align_center">6.0</td>
<td id="A1.T3.1.16.3" class="ltx_td ltx_align_center">4.2</td>
<td id="A1.T3.1.16.4" class="ltx_td ltx_align_center">14.7</td>
<td id="A1.T3.1.16.5" class="ltx_td ltx_align_center">24.3</td>
<td id="A1.T3.1.16.6" class="ltx_td"></td>
</tr>
<tr id="A1.T3.1.17" class="ltx_tr">
<td id="A1.T3.1.17.1" class="ltx_td ltx_align_left">Stablelm-3B</td>
<td id="A1.T3.1.17.2" class="ltx_td ltx_align_center">4.2</td>
<td id="A1.T3.1.17.3" class="ltx_td ltx_align_center">2.9</td>
<td id="A1.T3.1.17.4" class="ltx_td ltx_align_center">11.2</td>
<td id="A1.T3.1.17.5" class="ltx_td ltx_align_center">21.0</td>
<td id="A1.T3.1.17.6" class="ltx_td"></td>
</tr>
<tr id="A1.T3.1.18" class="ltx_tr">
<td id="A1.T3.1.18.1" class="ltx_td ltx_align_left">Pythia-12B</td>
<td id="A1.T3.1.18.2" class="ltx_td ltx_align_center">31.4</td>
<td id="A1.T3.1.18.3" class="ltx_td ltx_align_center">17.3</td>
<td id="A1.T3.1.18.4" class="ltx_td ltx_align_center">24.5</td>
<td id="A1.T3.1.18.5" class="ltx_td ltx_align_center">29.3</td>
<td id="A1.T3.1.18.6" class="ltx_td"></td>
</tr>
<tr id="A1.T3.1.19" class="ltx_tr">
<td id="A1.T3.1.19.1" class="ltx_td ltx_align_left">Pythia-6.9B</td>
<td id="A1.T3.1.19.2" class="ltx_td ltx_align_center">25.2</td>
<td id="A1.T3.1.19.3" class="ltx_td ltx_align_center">16.0</td>
<td id="A1.T3.1.19.4" class="ltx_td ltx_align_center">22.6</td>
<td id="A1.T3.1.19.5" class="ltx_td ltx_align_center">28.6</td>
<td id="A1.T3.1.19.6" class="ltx_td"></td>
</tr>
<tr id="A1.T3.1.20" class="ltx_tr">
<td id="A1.T3.1.20.1" class="ltx_td ltx_align_left">Pythia-2.8B</td>
<td id="A1.T3.1.20.2" class="ltx_td ltx_align_center">26.4</td>
<td id="A1.T3.1.20.3" class="ltx_td ltx_align_center">14.6</td>
<td id="A1.T3.1.20.4" class="ltx_td ltx_align_center">21.7</td>
<td id="A1.T3.1.20.5" class="ltx_td ltx_align_center">28.8</td>
<td id="A1.T3.1.20.6" class="ltx_td"></td>
</tr>
<tr id="A1.T3.1.21" class="ltx_tr">
<td id="A1.T3.1.21.1" class="ltx_td ltx_align_left">Pythia-1B</td>
<td id="A1.T3.1.21.2" class="ltx_td ltx_align_center">8.4</td>
<td id="A1.T3.1.21.3" class="ltx_td ltx_align_center">7.1</td>
<td id="A1.T3.1.21.4" class="ltx_td ltx_align_center">16.2</td>
<td id="A1.T3.1.21.5" class="ltx_td ltx_align_center">25.6</td>
<td id="A1.T3.1.21.6" class="ltx_td"></td>
</tr>
<tr id="A1.T3.1.22" class="ltx_tr">
<td id="A1.T3.1.22.1" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t" rowspan="10"><span id="A1.T3.1.22.1.1" class="ltx_text">Code LLM</span></td>
<td id="A1.T3.1.22.2" class="ltx_td ltx_align_left ltx_border_t">CodeLlama-34B</td>
<td id="A1.T3.1.22.3" class="ltx_td ltx_align_center ltx_border_t">91.4</td>
<td id="A1.T3.1.22.4" class="ltx_td ltx_align_center ltx_border_t">41.0</td>
<td id="A1.T3.1.22.5" class="ltx_td ltx_align_center ltx_border_t">53.9</td>
<td id="A1.T3.1.22.6" class="ltx_td ltx_align_center ltx_border_t">36.4</td>
<td id="A1.T3.1.22.7" class="ltx_td ltx_border_t"></td>
</tr>
<tr id="A1.T3.1.23" class="ltx_tr">
<td id="A1.T3.1.23.1" class="ltx_td ltx_align_left">CodeLlama-13B</td>
<td id="A1.T3.1.23.2" class="ltx_td ltx_align_center">90.0</td>
<td id="A1.T3.1.23.3" class="ltx_td ltx_align_center">35.7</td>
<td id="A1.T3.1.23.4" class="ltx_td ltx_align_center">49.9</td>
<td id="A1.T3.1.23.5" class="ltx_td ltx_align_center">30.6</td>
<td id="A1.T3.1.23.6" class="ltx_td"></td>
</tr>
<tr id="A1.T3.1.24" class="ltx_tr">
<td id="A1.T3.1.24.1" class="ltx_td ltx_align_left">CodeLlama-7B</td>
<td id="A1.T3.1.24.2" class="ltx_td ltx_align_center">75.2</td>
<td id="A1.T3.1.24.3" class="ltx_td ltx_align_center">34.2</td>
<td id="A1.T3.1.24.4" class="ltx_td ltx_align_center">44.9</td>
<td id="A1.T3.1.24.5" class="ltx_td ltx_align_center">26.3</td>
<td id="A1.T3.1.24.6" class="ltx_td"></td>
</tr>
<tr id="A1.T3.1.25" class="ltx_tr">
<td id="A1.T3.1.25.1" class="ltx_td ltx_align_left">StarCoder-15B</td>
<td id="A1.T3.1.25.2" class="ltx_td ltx_align_center">87.2</td>
<td id="A1.T3.1.25.3" class="ltx_td ltx_align_center">34.4</td>
<td id="A1.T3.1.25.4" class="ltx_td ltx_align_center">39.2</td>
<td id="A1.T3.1.25.5" class="ltx_td ltx_align_center">30.4</td>
<td id="A1.T3.1.25.6" class="ltx_td"></td>
</tr>
<tr id="A1.T3.1.26" class="ltx_tr">
<td id="A1.T3.1.26.1" class="ltx_td ltx_align_left">StarCoder-7B</td>
<td id="A1.T3.1.26.2" class="ltx_td ltx_align_center">88.4</td>
<td id="A1.T3.1.26.3" class="ltx_td ltx_align_center">32.4</td>
<td id="A1.T3.1.26.4" class="ltx_td ltx_align_center">33.3</td>
<td id="A1.T3.1.26.5" class="ltx_td ltx_align_center">28.3</td>
<td id="A1.T3.1.26.6" class="ltx_td"></td>
</tr>
<tr id="A1.T3.1.27" class="ltx_tr">
<td id="A1.T3.1.27.1" class="ltx_td ltx_align_left">StarCoder-3B</td>
<td id="A1.T3.1.27.2" class="ltx_td ltx_align_center">79.0</td>
<td id="A1.T3.1.27.3" class="ltx_td ltx_align_center">28.0</td>
<td id="A1.T3.1.27.4" class="ltx_td ltx_align_center">27.5</td>
<td id="A1.T3.1.27.5" class="ltx_td ltx_align_center">21.5</td>
<td id="A1.T3.1.27.6" class="ltx_td"></td>
</tr>
<tr id="A1.T3.1.28" class="ltx_tr">
<td id="A1.T3.1.28.1" class="ltx_td ltx_align_left">StarCoder-1B</td>
<td id="A1.T3.1.28.2" class="ltx_td ltx_align_center">37.4</td>
<td id="A1.T3.1.28.3" class="ltx_td ltx_align_center">15.4</td>
<td id="A1.T3.1.28.4" class="ltx_td ltx_align_center">21.1</td>
<td id="A1.T3.1.28.5" class="ltx_td ltx_align_center">15.2</td>
<td id="A1.T3.1.28.6" class="ltx_td"></td>
</tr>
<tr id="A1.T3.1.29" class="ltx_tr">
<td id="A1.T3.1.29.1" class="ltx_td ltx_align_left">CodeGen-15B</td>
<td id="A1.T3.1.29.2" class="ltx_td ltx_align_center">36.8</td>
<td id="A1.T3.1.29.3" class="ltx_td ltx_align_center">18.2</td>
<td id="A1.T3.1.29.4" class="ltx_td ltx_align_center">25.0</td>
<td id="A1.T3.1.29.5" class="ltx_td ltx_align_center">18.3</td>
<td id="A1.T3.1.29.6" class="ltx_td"></td>
</tr>
<tr id="A1.T3.1.30" class="ltx_tr">
<td id="A1.T3.1.30.1" class="ltx_td ltx_align_left">CodeGen-6B</td>
<td id="A1.T3.1.30.2" class="ltx_td ltx_align_center">25.0</td>
<td id="A1.T3.1.30.3" class="ltx_td ltx_align_center">16.9</td>
<td id="A1.T3.1.30.4" class="ltx_td ltx_align_center">17.8</td>
<td id="A1.T3.1.30.5" class="ltx_td ltx_align_center">18.2</td>
<td id="A1.T3.1.30.6" class="ltx_td"></td>
</tr>
<tr id="A1.T3.1.31" class="ltx_tr">
<td id="A1.T3.1.31.1" class="ltx_td ltx_align_left ltx_border_bb">CodeGen-2B</td>
<td id="A1.T3.1.31.2" class="ltx_td ltx_align_center ltx_border_bb">31.4</td>
<td id="A1.T3.1.31.3" class="ltx_td ltx_align_center ltx_border_bb">16.6</td>
<td id="A1.T3.1.31.4" class="ltx_td ltx_align_center ltx_border_bb">20.8</td>
<td id="A1.T3.1.31.5" class="ltx_td ltx_align_center ltx_border_bb">14.5</td>
<td id="A1.T3.1.31.6" class="ltx_td ltx_border_bb"></td>
</tr>
</table>
</figure>
</section>
<section id="A1.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.2 </span>Reliability Experiments</h3>

<section id="A1.SS2.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Symbolic Tasks vs. Natural Language Tasks.</h4>

<div id="A1.SS2.SSS0.Px1.p1" class="ltx_para ltx_noindent">
<p id="A1.SS2.SSS0.Px1.p1.1" class="ltx_p">Another point to prove is that symbolic tasks are consistent with their natural language counterparts. SQL execution is a suitable task because SQL can be intertranslated with an natural question. As can be seen from the “WTQ” column of the Table <a href="#A1.T3" title="Table 3 ‣ A.1 Overall Performance ‣ Appendix A Evaluation Experiments Results ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> and Figure <a href="#A1.F12.sf1" title="In Figure 12 ‣ SQL multi-step instruction experiments. ‣ A.3 Other SQL Prompting Styles ‣ Appendix A Evaluation Experiments Results ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">12(a)</span></a>, LLM’s ability to execute SQL is consistent with its table question answering ability.</p>
</div>
</section>
<section id="A1.SS2.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Synthetic data vs. Real data.</h4>

<div id="A1.SS2.SSS0.Px2.p1" class="ltx_para ltx_noindent">
<p id="A1.SS2.SSS0.Px2.p1.1" class="ltx_p">We want to verify if the synthesized SQL is simpler. The tables “SQL-general” and “WTQ-SQL” show the difference in performance between the model on synthetic and real data. We keep the average length of the tables similar, and the experimental results show that the synthetic SQL is more complex than the real SQL. And Figure <a href="#A1.F12.sf3" title="In Figure 12 ‣ SQL multi-step instruction experiments. ‣ A.3 Other SQL Prompting Styles ‣ Appendix A Evaluation Experiments Results ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">12(c)</span></a> shows that, the performance of LLMs on real tables and synthetic tables is very relevant.</p>
</div>
</section>
<section id="A1.SS2.SSS0.Px3" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Different <span id="A1.SS2.SSS0.Px3.1.1" class="ltx_text ltx_font_smallcaps">S3Eval</span> Settings.</h4>

<div id="A1.SS2.SSS0.Px3.p1" class="ltx_para ltx_noindent">
<p id="A1.SS2.SSS0.Px3.p1.1" class="ltx_p">As shown in Figure <a href="#A1.F12.sf2" title="In Figure 12 ‣ SQL multi-step instruction experiments. ‣ A.3 Other SQL Prompting Styles ‣ Appendix A Evaluation Experiments Results ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">12(b)</span></a>, even if the data settings are very different, LLMs are guaranteed a consistent performance ranking on <span id="A1.SS2.SSS0.Px3.p1.1.1" class="ltx_text ltx_font_smallcaps">S3Eval</span>.</p>
</div>
</section>
</section>
<section id="A1.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.3 </span>Other SQL Prompting Styles</h3>

<section id="A1.SS3.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">SQL execution task with Chain-of-Thought prompting.</h4>

<div id="A1.SS3.SSS0.Px1.p1" class="ltx_para ltx_noindent">
<p id="A1.SS3.SSS0.Px1.p1.1" class="ltx_p">SQL is a complex multi-step reasoning task. To verify whether it is a reliable reasoning task, <span id="A1.SS3.SSS0.Px1.p1.1.1" class="ltx_text ltx_font_smallcaps">S3Eval</span> generates multi-step execution instructions for SQL. ChatGPT’s performance (markdown) improves from <span id="A1.SS3.SSS0.Px1.p1.1.2" class="ltx_text ltx_font_bold">38.0</span> to <span id="A1.SS3.SSS0.Px1.p1.1.3" class="ltx_text ltx_font_bold">48.5</span> when using chain-of-thougnt prompts. The chain-of-thought examples are shown in below. The examples of chain-of-thought prompting are shown in Appendix <a href="#A3.SS6" title="C.6 Chain-of-Thought SQL Execution Prompting Examples ‣ Appendix C Data Demostration ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">C.6</span></a>.</p>
</div>
</section>
<section id="A1.SS3.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">SQL multi-step instruction experiments.</h4>

<div id="A1.SS3.SSS0.Px2.p1" class="ltx_para ltx_noindent">
<p id="A1.SS3.SSS0.Px2.p1.1" class="ltx_p">SQL multi-step instruction is an auxiliary task. We generate two new datasets using different settings than <span id="A1.SS3.SSS0.Px2.p1.1.1" class="ltx_text ltx_font_bold">Easy</span> and <span id="A1.SS3.SSS0.Px2.p1.1.2" class="ltx_text ltx_font_bold">General</span>, named <span id="A1.SS3.SSS0.Px2.p1.1.3" class="ltx_text ltx_font_bold">Data1</span> and <span id="A1.SS3.SSS0.Px2.p1.1.4" class="ltx_text ltx_font_bold">Data2</span>. Experiments results are shown in Table <a href="#A1.T5" title="Table 5 ‣ A.4 Long-Context Experiments ‣ Appendix A Evaluation Experiments Results ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>.</p>
</div>
<figure id="A1.F12" class="ltx_figure">
<div id="A1.F12.1" class="ltx_block">
<figure id="A1.F12.sf1" class="ltx_figure ltx_align_center"><img src="/html/2310.15147/assets/x15.png" id="A1.F12.sf1.g1" class="ltx_graphics ltx_img_square" width="190" height="172" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(a) </span>Alignment between QA task and SQL execution Task on WikiTableQuestions. </figcaption>
</figure>
<figure id="A1.F12.sf2" class="ltx_figure ltx_align_center"><img src="/html/2310.15147/assets/x16.png" id="A1.F12.sf2.g1" class="ltx_graphics ltx_img_square" width="203" height="171" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(b) </span>Alignment between General and Easy Settings.</figcaption>
</figure>
<figure id="A1.F12.sf3" class="ltx_figure ltx_align_center"><img src="/html/2310.15147/assets/x17.png" id="A1.F12.sf3.g1" class="ltx_graphics ltx_img_square" width="203" height="172" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(c) </span>Alignment between Synthetic and Real Table SQL execution task</figcaption>
</figure>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 12: </span>Experimental results of the alignment experiments.</figcaption>
</figure>
</section>
</section>
<section id="A1.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.4 </span>Long-Context Experiments</h3>

<div id="A1.SS4.p1" class="ltx_para ltx_noindent">
<p id="A1.SS4.p1.1" class="ltx_p">Context windows limit the long-context capabilities of LLMs. Previous researchers have proposed many ways to extend the length of context windows, often to 64K, 128K and so on. Existing benchmarks <cite class="ltx_cite ltx_citemacro_citep">(Bai et al., <a href="#bib.bib5" title="" class="ltx_ref">2023b</a>; An et al., <a href="#bib.bib1" title="" class="ltx_ref">2023</a>)</cite> collect data from existing NLP communities (which causes data leakage), and more importantly because collecting large amounts of data is difficult. <span id="A1.SS4.p1.1.1" class="ltx_text ltx_font_smallcaps">S3Eval</span>, on the other hand, is easy to collect data with variety and complexity. Existing benchmarks also can’t effectively evaluate very long texts, but <span id="A1.SS4.p1.1.2" class="ltx_text ltx_font_smallcaps">S3Eval</span> can evaluate arbitrary lengths.</p>
</div>
<div id="A1.SS4.p2" class="ltx_para">
<p id="A1.SS4.p2.1" class="ltx_p">YaRN <cite class="ltx_cite ltx_citemacro_citep">(Peng et al., <a href="#bib.bib32" title="" class="ltx_ref">2023</a>)</cite> extend LLaMA2 context windows to 128K, however, they only evaluated the model’s perplexity, which we believe is not a true reflection of its long-context understanding capability. So we use <span id="A1.SS4.p2.1.1" class="ltx_text ltx_font_smallcaps">S3Eval</span> to generate table data of different lengths and keep all parameters same to evaluate the performance of yarn-LLaMA2, and the experimental results are shown in Table <a href="#A1.T4" title="Table 4 ‣ A.4 Long-Context Experiments ‣ Appendix A Evaluation Experiments Results ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>. It shows that, yarn-llama2 has a noticeable dip in performance on 20K-80K, which is good for a small number of tasks as well. But compared to ChatGPT (which we can only test 16K length tables), there’s a noticeable gap.</p>
</div>
<figure id="A1.T4" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 4: </span>Long-Context experiments on <span id="A1.T4.2.1" class="ltx_text ltx_font_smallcaps">S3Eval</span>.</figcaption>
<table id="A1.T4.3" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="A1.T4.3.1" class="ltx_tr">
<td id="A1.T4.3.1.1" class="ltx_td ltx_align_center ltx_border_tt" rowspan="2"><span id="A1.T4.3.1.1.1" class="ltx_text">Model</span></td>
<td id="A1.T4.3.1.2" class="ltx_td ltx_align_center ltx_border_tt" rowspan="2"><span id="A1.T4.3.1.2.1" class="ltx_text">Max-Ctx</span></td>
<td id="A1.T4.3.1.3" class="ltx_td ltx_align_center ltx_border_tt" colspan="8">SQL Execution</td>
</tr>
<tr id="A1.T4.3.2" class="ltx_tr">
<td id="A1.T4.3.2.1" class="ltx_td ltx_align_center">2K</td>
<td id="A1.T4.3.2.2" class="ltx_td ltx_align_center">4K</td>
<td id="A1.T4.3.2.3" class="ltx_td ltx_align_center">8K</td>
<td id="A1.T4.3.2.4" class="ltx_td ltx_align_center">16K</td>
<td id="A1.T4.3.2.5" class="ltx_td ltx_align_center">20K</td>
<td id="A1.T4.3.2.6" class="ltx_td ltx_align_center">40K</td>
<td id="A1.T4.3.2.7" class="ltx_td ltx_align_center">60K</td>
<td id="A1.T4.3.2.8" class="ltx_td ltx_align_center">80K</td>
</tr>
<tr id="A1.T4.3.3" class="ltx_tr">
<td id="A1.T4.3.3.1" class="ltx_td ltx_align_left ltx_border_t">ChatGPT</td>
<td id="A1.T4.3.3.2" class="ltx_td ltx_align_center ltx_border_t">16k</td>
<td id="A1.T4.3.3.3" class="ltx_td ltx_align_center ltx_border_t">96.8</td>
<td id="A1.T4.3.3.4" class="ltx_td ltx_align_center ltx_border_t">95.2</td>
<td id="A1.T4.3.3.5" class="ltx_td ltx_align_center ltx_border_t">80.3</td>
<td id="A1.T4.3.3.6" class="ltx_td ltx_align_center ltx_border_t">68.7</td>
<td id="A1.T4.3.3.7" class="ltx_td ltx_align_center ltx_border_t">-</td>
<td id="A1.T4.3.3.8" class="ltx_td ltx_align_center ltx_border_t">-</td>
<td id="A1.T4.3.3.9" class="ltx_td ltx_align_center ltx_border_t">-</td>
<td id="A1.T4.3.3.10" class="ltx_td ltx_align_center ltx_border_t">-</td>
</tr>
<tr id="A1.T4.3.4" class="ltx_tr">
<td id="A1.T4.3.4.1" class="ltx_td ltx_align_left">Claude-1.3-100K</td>
<td id="A1.T4.3.4.2" class="ltx_td ltx_align_center">128k</td>
<td id="A1.T4.3.4.3" class="ltx_td ltx_align_center">97.2</td>
<td id="A1.T4.3.4.4" class="ltx_td ltx_align_center">96.8</td>
<td id="A1.T4.3.4.5" class="ltx_td ltx_align_center">91.8</td>
<td id="A1.T4.3.4.6" class="ltx_td ltx_align_center">85.2</td>
<td id="A1.T4.3.4.7" class="ltx_td ltx_align_center">-</td>
<td id="A1.T4.3.4.8" class="ltx_td ltx_align_center">-</td>
<td id="A1.T4.3.4.9" class="ltx_td ltx_align_center">-</td>
<td id="A1.T4.3.4.10" class="ltx_td ltx_align_center">-</td>
</tr>
<tr id="A1.T4.3.5" class="ltx_tr">
<td id="A1.T4.3.5.1" class="ltx_td ltx_align_left">Yarn-LLaMA2-13B</td>
<td id="A1.T4.3.5.2" class="ltx_td ltx_align_center">128k</td>
<td id="A1.T4.3.5.3" class="ltx_td ltx_align_center">76.3</td>
<td id="A1.T4.3.5.4" class="ltx_td ltx_align_center">57.0</td>
<td id="A1.T4.3.5.5" class="ltx_td ltx_align_center">40.6</td>
<td id="A1.T4.3.5.6" class="ltx_td ltx_align_center">25.1</td>
<td id="A1.T4.3.5.7" class="ltx_td ltx_align_center">20.6</td>
<td id="A1.T4.3.5.8" class="ltx_td ltx_align_center">17.6</td>
<td id="A1.T4.3.5.9" class="ltx_td ltx_align_center">17.0</td>
<td id="A1.T4.3.5.10" class="ltx_td ltx_align_center">12.0</td>
</tr>
<tr id="A1.T4.3.6" class="ltx_tr">
<td id="A1.T4.3.6.1" class="ltx_td ltx_align_left">XGen-7B</td>
<td id="A1.T4.3.6.2" class="ltx_td ltx_align_center">8k</td>
<td id="A1.T4.3.6.3" class="ltx_td ltx_align_center">51.6</td>
<td id="A1.T4.3.6.4" class="ltx_td ltx_align_center">41.8</td>
<td id="A1.T4.3.6.5" class="ltx_td ltx_align_center">25.4</td>
<td id="A1.T4.3.6.6" class="ltx_td ltx_align_center">-</td>
<td id="A1.T4.3.6.7" class="ltx_td ltx_align_center">-</td>
<td id="A1.T4.3.6.8" class="ltx_td ltx_align_center">-</td>
<td id="A1.T4.3.6.9" class="ltx_td ltx_align_center">-</td>
<td id="A1.T4.3.6.10" class="ltx_td ltx_align_center">-</td>
</tr>
<tr id="A1.T4.3.7" class="ltx_tr">
<td id="A1.T4.3.7.1" class="ltx_td ltx_align_left">LongChat-13B</td>
<td id="A1.T4.3.7.2" class="ltx_td ltx_align_center">16k</td>
<td id="A1.T4.3.7.3" class="ltx_td ltx_align_center">48.6</td>
<td id="A1.T4.3.7.4" class="ltx_td ltx_align_center">39.0</td>
<td id="A1.T4.3.7.5" class="ltx_td ltx_align_center">26.3</td>
<td id="A1.T4.3.7.6" class="ltx_td ltx_align_center">19.5</td>
<td id="A1.T4.3.7.7" class="ltx_td ltx_align_center">-</td>
<td id="A1.T4.3.7.8" class="ltx_td ltx_align_center">-</td>
<td id="A1.T4.3.7.9" class="ltx_td ltx_align_center">-</td>
<td id="A1.T4.3.7.10" class="ltx_td ltx_align_center">-</td>
</tr>
<tr id="A1.T4.3.8" class="ltx_tr">
<td id="A1.T4.3.8.1" class="ltx_td ltx_align_left">LongLlaMA-7B</td>
<td id="A1.T4.3.8.2" class="ltx_td ltx_align_center">256k</td>
<td id="A1.T4.3.8.3" class="ltx_td ltx_align_center">82.4</td>
<td id="A1.T4.3.8.4" class="ltx_td ltx_align_center">62.8</td>
<td id="A1.T4.3.8.5" class="ltx_td ltx_align_center">24.4</td>
<td id="A1.T4.3.8.6" class="ltx_td ltx_align_center">-</td>
<td id="A1.T4.3.8.7" class="ltx_td ltx_align_center">-</td>
<td id="A1.T4.3.8.8" class="ltx_td ltx_align_center">-</td>
<td id="A1.T4.3.8.9" class="ltx_td ltx_align_center">-</td>
<td id="A1.T4.3.8.10" class="ltx_td ltx_align_center">-</td>
</tr>
<tr id="A1.T4.3.9" class="ltx_tr">
<td id="A1.T4.3.9.1" class="ltx_td ltx_align_left ltx_border_bb">RWKV-Raven-14B</td>
<td id="A1.T4.3.9.2" class="ltx_td ltx_align_center ltx_border_bb">128k</td>
<td id="A1.T4.3.9.3" class="ltx_td ltx_align_center ltx_border_bb">10.5</td>
<td id="A1.T4.3.9.4" class="ltx_td ltx_align_center ltx_border_bb">7.4</td>
<td id="A1.T4.3.9.5" class="ltx_td ltx_align_center ltx_border_bb">6.2</td>
<td id="A1.T4.3.9.6" class="ltx_td ltx_align_center ltx_border_bb">-</td>
<td id="A1.T4.3.9.7" class="ltx_td ltx_align_center ltx_border_bb">-</td>
<td id="A1.T4.3.9.8" class="ltx_td ltx_align_center ltx_border_bb">-</td>
<td id="A1.T4.3.9.9" class="ltx_td ltx_align_center ltx_border_bb">-</td>
<td id="A1.T4.3.9.10" class="ltx_td ltx_align_center ltx_border_bb">-</td>
</tr>
</table>
</figure>
<figure id="A1.T5" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 5: </span>SQL Multi-Step Task performance on different LLMs.</figcaption>
<table id="A1.T5.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="A1.T5.1.1" class="ltx_tr">
<td id="A1.T5.1.1.1" class="ltx_td ltx_align_left ltx_border_tt" rowspan="3"><span id="A1.T5.1.1.1.1" class="ltx_text ltx_font_bold">Model</span></td>
<td id="A1.T5.1.1.2" class="ltx_td ltx_align_center ltx_border_tt" colspan="4"><span id="A1.T5.1.1.2.1" class="ltx_text ltx_font_bold">SQL Execution</span></td>
<td id="A1.T5.1.1.3" class="ltx_td ltx_align_center ltx_border_tt" colspan="4"><span id="A1.T5.1.1.3.1" class="ltx_text ltx_font_bold">SQL Multi-Step Instruction</span></td>
</tr>
<tr id="A1.T5.1.2" class="ltx_tr">
<td id="A1.T5.1.2.1" class="ltx_td ltx_align_center" colspan="2"><span id="A1.T5.1.2.1.1" class="ltx_text ltx_font_italic">Zero-Shot</span></td>
<td id="A1.T5.1.2.2" class="ltx_td ltx_align_center" colspan="2"><span id="A1.T5.1.2.2.1" class="ltx_text ltx_font_italic">Few-Shot</span></td>
<td id="A1.T5.1.2.3" class="ltx_td ltx_align_center" colspan="2"><span id="A1.T5.1.2.3.1" class="ltx_text ltx_font_italic">Zero-Shot</span></td>
<td id="A1.T5.1.2.4" class="ltx_td ltx_align_center" colspan="2"><span id="A1.T5.1.2.4.1" class="ltx_text ltx_font_italic">Few-Shot</span></td>
</tr>
<tr id="A1.T5.1.3" class="ltx_tr">
<td id="A1.T5.1.3.1" class="ltx_td ltx_align_center"><span id="A1.T5.1.3.1.1" class="ltx_text ltx_font_bold">Data1</span></td>
<td id="A1.T5.1.3.2" class="ltx_td ltx_align_center"><span id="A1.T5.1.3.2.1" class="ltx_text ltx_font_bold">Data2</span></td>
<td id="A1.T5.1.3.3" class="ltx_td ltx_align_center"><span id="A1.T5.1.3.3.1" class="ltx_text ltx_font_bold">Data1</span></td>
<td id="A1.T5.1.3.4" class="ltx_td ltx_align_center"><span id="A1.T5.1.3.4.1" class="ltx_text ltx_font_bold">Data2</span></td>
<td id="A1.T5.1.3.5" class="ltx_td ltx_align_center"><span id="A1.T5.1.3.5.1" class="ltx_text ltx_font_bold">Data1</span></td>
<td id="A1.T5.1.3.6" class="ltx_td ltx_align_center"><span id="A1.T5.1.3.6.1" class="ltx_text ltx_font_bold">Data2</span></td>
<td id="A1.T5.1.3.7" class="ltx_td ltx_align_center"><span id="A1.T5.1.3.7.1" class="ltx_text ltx_font_bold">Data1</span></td>
<td id="A1.T5.1.3.8" class="ltx_td ltx_nopad_r ltx_align_center"><span id="A1.T5.1.3.8.1" class="ltx_text ltx_font_bold">Data2</span></td>
</tr>
<tr id="A1.T5.1.4" class="ltx_tr">
<td id="A1.T5.1.4.1" class="ltx_td ltx_align_left ltx_border_t">ChatGPT</td>
<td id="A1.T5.1.4.2" class="ltx_td ltx_align_center ltx_border_t">96.4</td>
<td id="A1.T5.1.4.3" class="ltx_td ltx_align_center ltx_border_t">47.0</td>
<td id="A1.T5.1.4.4" class="ltx_td ltx_align_center ltx_border_t">97.0</td>
<td id="A1.T5.1.4.5" class="ltx_td ltx_align_center ltx_border_t">49.0</td>
<td id="A1.T5.1.4.6" class="ltx_td ltx_align_center ltx_border_t">97.9</td>
<td id="A1.T5.1.4.7" class="ltx_td ltx_align_center ltx_border_t">30.0</td>
<td id="A1.T5.1.4.8" class="ltx_td ltx_align_center ltx_border_t">98.8</td>
<td id="A1.T5.1.4.9" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t">34.8</td>
</tr>
<tr id="A1.T5.1.5" class="ltx_tr">
<td id="A1.T5.1.5.1" class="ltx_td ltx_align_left">Codellama-13B</td>
<td id="A1.T5.1.5.2" class="ltx_td ltx_align_center">71.2</td>
<td id="A1.T5.1.5.3" class="ltx_td ltx_align_center">34.3</td>
<td id="A1.T5.1.5.4" class="ltx_td ltx_align_center">90.0</td>
<td id="A1.T5.1.5.5" class="ltx_td ltx_align_center">39.8</td>
<td id="A1.T5.1.5.6" class="ltx_td ltx_align_center">63.9</td>
<td id="A1.T5.1.5.7" class="ltx_td ltx_align_center">12.1</td>
<td id="A1.T5.1.5.8" class="ltx_td ltx_align_center">88.0</td>
<td id="A1.T5.1.5.9" class="ltx_td ltx_nopad_r ltx_align_center">22.8</td>
</tr>
<tr id="A1.T5.1.6" class="ltx_tr">
<td id="A1.T5.1.6.1" class="ltx_td ltx_align_left">StarCoder-15B</td>
<td id="A1.T5.1.6.2" class="ltx_td ltx_align_center">52.3</td>
<td id="A1.T5.1.6.3" class="ltx_td ltx_align_center">24.7</td>
<td id="A1.T5.1.6.4" class="ltx_td ltx_align_center">85.8</td>
<td id="A1.T5.1.6.5" class="ltx_td ltx_align_center">37.6</td>
<td id="A1.T5.1.6.6" class="ltx_td ltx_align_center">44.4</td>
<td id="A1.T5.1.6.7" class="ltx_td ltx_align_center">14.4</td>
<td id="A1.T5.1.6.8" class="ltx_td ltx_align_center">84.2</td>
<td id="A1.T5.1.6.9" class="ltx_td ltx_nopad_r ltx_align_center">19.2</td>
</tr>
<tr id="A1.T5.1.7" class="ltx_tr">
<td id="A1.T5.1.7.1" class="ltx_td ltx_align_left">InternLM-20B</td>
<td id="A1.T5.1.7.2" class="ltx_td ltx_align_center">60.4</td>
<td id="A1.T5.1.7.3" class="ltx_td ltx_align_center">22.7</td>
<td id="A1.T5.1.7.4" class="ltx_td ltx_align_center">78.0</td>
<td id="A1.T5.1.7.5" class="ltx_td ltx_align_center">35.0</td>
<td id="A1.T5.1.7.6" class="ltx_td ltx_align_center">58.8</td>
<td id="A1.T5.1.7.7" class="ltx_td ltx_align_center">14.9</td>
<td id="A1.T5.1.7.8" class="ltx_td ltx_align_center">76.6</td>
<td id="A1.T5.1.7.9" class="ltx_td ltx_nopad_r ltx_align_center">28.1</td>
</tr>
<tr id="A1.T5.1.8" class="ltx_tr">
<td id="A1.T5.1.8.1" class="ltx_td ltx_align_left">InternLM-20B-Chat</td>
<td id="A1.T5.1.8.2" class="ltx_td ltx_align_center">71.2</td>
<td id="A1.T5.1.8.3" class="ltx_td ltx_align_center">31.3</td>
<td id="A1.T5.1.8.4" class="ltx_td ltx_align_center">78.0</td>
<td id="A1.T5.1.8.5" class="ltx_td ltx_align_center">34.2</td>
<td id="A1.T5.1.8.6" class="ltx_td ltx_align_center">67.6</td>
<td id="A1.T5.1.8.7" class="ltx_td ltx_align_center">21.9</td>
<td id="A1.T5.1.8.8" class="ltx_td ltx_align_center">74.4</td>
<td id="A1.T5.1.8.9" class="ltx_td ltx_nopad_r ltx_align_center">25.4</td>
</tr>
<tr id="A1.T5.1.9" class="ltx_tr">
<td id="A1.T5.1.9.1" class="ltx_td ltx_align_left">LLaMA2-13B</td>
<td id="A1.T5.1.9.2" class="ltx_td ltx_align_center">68.1</td>
<td id="A1.T5.1.9.3" class="ltx_td ltx_align_center">23.2</td>
<td id="A1.T5.1.9.4" class="ltx_td ltx_align_center">75.0</td>
<td id="A1.T5.1.9.5" class="ltx_td ltx_align_center">32.3</td>
<td id="A1.T5.1.9.6" class="ltx_td ltx_align_center">50.5</td>
<td id="A1.T5.1.9.7" class="ltx_td ltx_align_center">5.4</td>
<td id="A1.T5.1.9.8" class="ltx_td ltx_align_center">74.6</td>
<td id="A1.T5.1.9.9" class="ltx_td ltx_nopad_r ltx_align_center">18.2</td>
</tr>
<tr id="A1.T5.1.10" class="ltx_tr">
<td id="A1.T5.1.10.1" class="ltx_td ltx_align_left">LLaMA2-13B-Chat</td>
<td id="A1.T5.1.10.2" class="ltx_td ltx_align_center">51.6</td>
<td id="A1.T5.1.10.3" class="ltx_td ltx_align_center">16.4</td>
<td id="A1.T5.1.10.4" class="ltx_td ltx_align_center">71.5</td>
<td id="A1.T5.1.10.5" class="ltx_td ltx_align_center">28.3</td>
<td id="A1.T5.1.10.6" class="ltx_td ltx_align_center">9.4</td>
<td id="A1.T5.1.10.7" class="ltx_td ltx_align_center">1.0</td>
<td id="A1.T5.1.10.8" class="ltx_td ltx_align_center">64.2</td>
<td id="A1.T5.1.10.9" class="ltx_td ltx_nopad_r ltx_align_center">21.1</td>
</tr>
<tr id="A1.T5.1.11" class="ltx_tr">
<td id="A1.T5.1.11.1" class="ltx_td ltx_align_left ltx_border_bb">Vicuna-13B</td>
<td id="A1.T5.1.11.2" class="ltx_td ltx_align_center ltx_border_bb">57.6</td>
<td id="A1.T5.1.11.3" class="ltx_td ltx_align_center ltx_border_bb">26.8</td>
<td id="A1.T5.1.11.4" class="ltx_td ltx_align_center ltx_border_bb">81.6</td>
<td id="A1.T5.1.11.5" class="ltx_td ltx_align_center ltx_border_bb">35.4</td>
<td id="A1.T5.1.11.6" class="ltx_td ltx_align_center ltx_border_bb">48.9</td>
<td id="A1.T5.1.11.7" class="ltx_td ltx_align_center ltx_border_bb">11.5</td>
<td id="A1.T5.1.11.8" class="ltx_td ltx_align_center ltx_border_bb">78.8</td>
<td id="A1.T5.1.11.9" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_bb">24.2</td>
</tr>
</table>
</figure>
</section>
</section>
<section id="A2" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix B </span>Controllable Analysis Results</h2>

<section id="A2.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.1 </span>Answer Position Analysis</h3>

<div id="A2.SS1.p1" class="ltx_para ltx_noindent">
<p id="A2.SS1.p1.1" class="ltx_p">In addition to the figures in the main text, we also conduct experiments with row level. We use two methods to visualize the results. (1) Sliding windows (Figure <a href="#A2.F13.sf1" title="In Figure 13 ‣ B.1 Answer Position Analysis ‣ Appendix B Controllable Analysis Results ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">13(a)</span></a>,<a href="#A2.F13.sf2" title="In Figure 13 ‣ B.1 Answer Position Analysis ‣ Appendix B Controllable Analysis Results ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">13(b)</span></a>). We choose windows=5 and smooth the data to make a dot plot and a trend line. (2) Grouping calculations (Figure <a href="#A2.F13.sf3" title="In Figure 13 ‣ B.1 Answer Position Analysis ‣ Appendix B Controllable Analysis Results ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">13(c)</span></a>,<a href="#A2.F13.sf4" title="In Figure 13 ‣ B.1 Answer Position Analysis ‣ Appendix B Controllable Analysis Results ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">13(d)</span></a>). Group neighboring rows together with the granularity of 5, 10, and 20. For example, if granularity is 20, then we group the rows with answers located in 1-20, 20-40, 40-60, 60-80, and 80-100, for a total of five groups, and calculate the average scores.</p>
</div>
<figure id="A2.F13" class="ltx_figure">
<div id="A2.F13.1" class="ltx_block">
<figure id="A2.F13.sf1" class="ltx_figure ltx_align_center"><img src="/html/2310.15147/assets/x18.png" id="A2.F13.sf1.g1" class="ltx_graphics ltx_img_landscape" width="190" height="151" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(a) </span></figcaption>
</figure>
<figure id="A2.F13.sf2" class="ltx_figure ltx_align_center"><img src="/html/2310.15147/assets/x19.png" id="A2.F13.sf2.g1" class="ltx_graphics ltx_img_landscape" width="190" height="152" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(b) </span></figcaption>
</figure>
<figure id="A2.F13.sf3" class="ltx_figure ltx_align_center"><img src="/html/2310.15147/assets/x20.png" id="A2.F13.sf3.g1" class="ltx_graphics ltx_img_square" width="190" height="155" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(c) </span></figcaption>
</figure>
<figure id="A2.F13.sf4" class="ltx_figure ltx_align_center"><img src="/html/2310.15147/assets/x21.png" id="A2.F13.sf4.g1" class="ltx_graphics ltx_img_square" width="190" height="154" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(d) </span></figcaption>
</figure>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 13: </span>Effect of answer position on model performance. We use two methods to visualize the results. (1) Sliding windows (Figure <a href="#A2.F13.sf1" title="In Figure 13 ‣ B.1 Answer Position Analysis ‣ Appendix B Controllable Analysis Results ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">13(a)</span></a>,<a href="#A2.F13.sf2" title="In Figure 13 ‣ B.1 Answer Position Analysis ‣ Appendix B Controllable Analysis Results ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">13(b)</span></a>). We select a window size of 5 and smooth the data to make a dot plot and a trend line. (2) Grouping calculations (Figure <a href="#A2.F13.sf3" title="In Figure 13 ‣ B.1 Answer Position Analysis ‣ Appendix B Controllable Analysis Results ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">13(c)</span></a>,<a href="#A2.F13.sf4" title="In Figure 13 ‣ B.1 Answer Position Analysis ‣ Appendix B Controllable Analysis Results ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">13(d)</span></a>). We group neighboring rows with granularities of 5, 10, and 20. For instance, with a granularity of 20, we group rows with answers located in the ranges 1-20, 21-40, 41-60, 61-80, and 81-100, resulting in five groups, and compute the average scores.</figcaption>
</figure>
<figure id="A2.F15" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="A2.F15.1" class="ltx_figure ltx_figure_panel ltx_minipage ltx_align_center ltx_align_bottom" style="width:178.9pt;"><img src="/html/2310.15147/assets/x22.png" id="A2.F15.1.g1" class="ltx_graphics ltx_img_landscape" width="423" height="322" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 14: </span>Different input format.</figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="A2.F15.2" class="ltx_figure ltx_figure_panel ltx_minipage ltx_align_center ltx_align_bottom" style="width:178.9pt;"><img src="/html/2310.15147/assets/x23.png" id="A2.F15.2.g1" class="ltx_graphics ltx_img_landscape" width="422" height="334" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 15: </span>Different keywords setting.</figcaption>
</figure>
</div>
</div>
</figure>
</section>
<section id="A2.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.2 </span>Input Format Analysis</h3>

<div id="A2.SS2.p1" class="ltx_para ltx_noindent">
<p id="A2.SS2.p1.1" class="ltx_p">In this section, we focus on comparing two formats of inputting tables, namely <span id="A2.SS2.p1.1.1" class="ltx_text ltx_font_italic">markdown</span> and <span id="A2.SS2.p1.1.2" class="ltx_text ltx_font_italic">flatten</span>, to explore their impact on LLMs performance. Figure <a href="#A2.F15" title="Figure 15 ‣ B.1 Answer Position Analysis ‣ Appendix B Controllable Analysis Results ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">15</span></a> clearly demonstrates a significant improvement in the model’s performance when the <span id="A2.SS2.p1.1.3" class="ltx_text ltx_font_italic">flatten</span> format is used instead of the <span id="A2.SS2.p1.1.4" class="ltx_text ltx_font_italic">markdown</span> input format at any experiments settings.</p>
</div>
<div id="A2.SS2.p2" class="ltx_para ltx_noindent">
<p id="A2.SS2.p2.1" class="ltx_p">The reason behind this improvement lies in the structure of the SQL template, specifically “select &lt;col1&gt; where &lt;col2&gt; &lt;op&gt; &lt;int2&gt;”. In order to execute this template, the model needs to locate the column corresponding to col2 and then identify the row where “int2” is found. This process involves 2-hop reasoning. In <span id="A2.SS2.p2.1.1" class="ltx_text ltx_font_italic">markdown</span> mode, the challenge lies not only in the LLM’s understanding of the table structure but also in how to navigate to another column in the same row. However, in <span id="A2.SS2.p2.1.2" class="ltx_text ltx_font_italic">flatten</span> mode, redundant columns are added to each row as “Column is value.” This additional information simplifies the LLM’s understanding of the table structure and facilitates reasoning. As a result, the flatten method proves to be more beneficial for LLM performance due to its enhanced structure comprehension and reasoning capabilities.</p>
</div>
</section>
<section id="A2.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.3 </span>SQL Keywords Analysis</h3>

<div id="A2.SS3.p1" class="ltx_para ltx_noindent">
<p id="A2.SS3.p1.1" class="ltx_p">‘SQL statements follow a specific syntax and are a well-established language in the database domain. We first control SQL statements to contain only specific types of keywords from the perspective of SQL keywords and test the performance of different models on <span id="A2.SS3.p1.1.1" class="ltx_text ltx_font_smallcaps">S3Eval</span>. The experimental results are shown in Figure <a href="#A2.F15" title="Figure 15 ‣ B.1 Answer Position Analysis ‣ Appendix B Controllable Analysis Results ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">15</span></a>. The change in the performance of LLMs on SQL statements reflects the trend in the difficulty of reasoning.</p>
</div>
</section>
<section id="A2.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.4 </span>SQL Attribute Analysis</h3>

<div id="A2.SS4.p1" class="ltx_para ltx_noindent">
<p id="A2.SS4.p1.1" class="ltx_p"><span id="A2.SS4.p1.1.1" class="ltx_text ltx_font_smallcaps">S3Eval</span> has the ability to flexibly modify the properties of generated SQL statements, including the length of the statement, the number of computations, and the quantity of filtering numbers. These features can intuitively impact the complexity of SQL. In our experiments, we set the table size to <math id="A2.SS4.p1.1.m1.1" class="ltx_Math" alttext="15\times 10" display="inline"><semantics id="A2.SS4.p1.1.m1.1a"><mrow id="A2.SS4.p1.1.m1.1.1" xref="A2.SS4.p1.1.m1.1.1.cmml"><mn id="A2.SS4.p1.1.m1.1.1.2" xref="A2.SS4.p1.1.m1.1.1.2.cmml">15</mn><mo lspace="0.222em" rspace="0.222em" id="A2.SS4.p1.1.m1.1.1.1" xref="A2.SS4.p1.1.m1.1.1.1.cmml">×</mo><mn id="A2.SS4.p1.1.m1.1.1.3" xref="A2.SS4.p1.1.m1.1.1.3.cmml">10</mn></mrow><annotation-xml encoding="MathML-Content" id="A2.SS4.p1.1.m1.1b"><apply id="A2.SS4.p1.1.m1.1.1.cmml" xref="A2.SS4.p1.1.m1.1.1"><times id="A2.SS4.p1.1.m1.1.1.1.cmml" xref="A2.SS4.p1.1.m1.1.1.1"></times><cn type="integer" id="A2.SS4.p1.1.m1.1.1.2.cmml" xref="A2.SS4.p1.1.m1.1.1.2">15</cn><cn type="integer" id="A2.SS4.p1.1.m1.1.1.3.cmml" xref="A2.SS4.p1.1.m1.1.1.3">10</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.SS4.p1.1.m1.1c">15\times 10</annotation></semantics></math> and adjusted the SQL settings for examining the effect of different SQL attributes on model performance. For example, in the analysis of ”Calculation Times,” we employed 500 samples with 0, 1, 2, and 3 calculation times respectively. The experimental outcomes of all SQL attributes are illustrated in Figure<a href="#A2.F16" title="Figure 16 ‣ B.4 SQL Attribute Analysis ‣ Appendix B Controllable Analysis Results ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">16</span></a>. While it might be expected that model performance would decline as these factor values increase, the performance actually fluctuates. Upon combining Column number, Row number, Calculation times, and Filter times in the statistical analysis, we identified a significant downward trend in the model, as demonstrated in Figure<a href="#A2.F16.sf6" title="In Figure 16 ‣ B.4 SQL Attribute Analysis ‣ Appendix B Controllable Analysis Results ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">16(f)</span></a>.</p>
</div>
<figure id="A2.F16" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_3">
<figure id="A2.F16.sf1" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2310.15147/assets/x24.png" id="A2.F16.sf1.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="423" height="319" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">(a) </span>SQL length</figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_3">
<figure id="A2.F16.sf2" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2310.15147/assets/x25.png" id="A2.F16.sf2.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="422" height="316" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">(b) </span>Column number.</figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_3">
<figure id="A2.F16.sf3" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2310.15147/assets/x26.png" id="A2.F16.sf3.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="422" height="310" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">(c) </span>Selected row number</figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_3">
<figure id="A2.F16.sf4" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2310.15147/assets/x27.png" id="A2.F16.sf4.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="422" height="310" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">(d) </span>Calculation times</figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_3">
<figure id="A2.F16.sf5" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2310.15147/assets/x28.png" id="A2.F16.sf5.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="422" height="316" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">(e) </span>Filter times.</figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_3">
<figure id="A2.F16.sf6" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2310.15147/assets/x29.png" id="A2.F16.sf6.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="423" height="319" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">(f) </span>Total analysis.</figcaption>
</figure>
</div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 16: </span>Effect of SQL Attribute Settings on model performance.</figcaption>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
</section>
<section id="A3" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix C </span>Data Demostration</h2>

<section id="A3.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">C.1 </span>Dense and Sparse Examples</h3>

<div id="A3.SS1.p1" class="ltx_para ltx_noindent">
<p id="A3.SS1.p1.1" class="ltx_p">SQL: select boarfish from w where sixties = ’jcrbb’</p>
</div>
<div id="A3.SS1.p2" class="ltx_para ltx_noindent">
<p id="A3.SS1.p2.1" class="ltx_p">Answer: [’qxgd’, ’lorfaljob’, ’qytocp’, ’vkfzhqwj’, ’xwijyubr’]</p>
</div>
<div id="A3.SS1.p3" class="ltx_para ltx_noindent">
<p id="A3.SS1.p3.1" class="ltx_p">We can find that <span id="A3.SS1.p3.1.1" class="ltx_text ltx_font_italic">Dense</span> Setting is better than <span id="A3.SS1.p3.1.2" class="ltx_text ltx_font_italic">Sparse</span> Setting in all cases.</p>
</div>
<section id="A3.SS1.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Sparse Example:</h4>

<div id="A3.SS1.SSS0.Px1.p1" class="ltx_para ltx_noindent">
<p id="A3.SS1.SSS0.Px1.p1.1" class="ltx_p"><span id="A3.SS1.SSS0.Px1.p1.1.1" class="ltx_text ltx_font_typewriter"><span id="A3.SS1.SSS0.Px1.p1.1.1.1" class="ltx_text ltx_inline-block" style="width:0.0pt;"></span><span id="A3.SS1.SSS0.Px1.p1.1.1.2" class="ltx_text ltx_inline-block" style="width:469.8pt;">|    | boarfish   | tool     | sixties   | phoxinus   | angling   |</span></span><span id="A3.SS1.SSS0.Px1.p1.1.2" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">|---:|:-----------|:---------|:----------|:-----------|:----------|</span><span id="A3.SS1.SSS0.Px1.p1.1.3" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">|  0 | mjdsv      | cwzqkdte | tbwqa     | yuogpbo    | mkxqnrhq  |</span><span id="A3.SS1.SSS0.Px1.p1.1.4" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">|  1 | nrbmyc     | eqciiims | wvfesrtzt | yvvgzj     | mkxqnrhq  |</span><span id="A3.SS1.SSS0.Px1.p1.1.5" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">|  2 | iqdr       | ezhuj    | bndktpe   | yuogpbo    | yjblg     |</span><span id="A3.SS1.SSS0.Px1.p1.1.6" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">|  3 | <span id="A3.SS1.SSS0.Px1.p1.1.6.1" class="ltx_text" style="color:#FF0000;">qxgd</span>       | dtfjqfc  | <span id="A3.SS1.SSS0.Px1.p1.1.6.2" class="ltx_text" style="color:#BF0040;">jcrbb</span>     | haxyaz     | yjblg     |</span><span id="A3.SS1.SSS0.Px1.p1.1.7" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">|  4 | xzrrs      | ezhuj    | bndktpe   | dpimlb     | skbpzyhak |</span><span id="A3.SS1.SSS0.Px1.p1.1.8" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">|  5 | <span id="A3.SS1.SSS0.Px1.p1.1.8.1" class="ltx_text" style="color:#FF0000;">lorfaljob</span>  | eqciiims | <span id="A3.SS1.SSS0.Px1.p1.1.8.2" class="ltx_text" style="color:#BF0040;">jcrbb</span>     | jsvbugac   | bwxihx    |</span><span id="A3.SS1.SSS0.Px1.p1.1.9" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">|  6 | pvugxgdju  | dtfjqfc  | bndktpe   | jsvbugac   | mkxqnrhq  |</span><span id="A3.SS1.SSS0.Px1.p1.1.10" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">|  7 | xpkuautv   | ezhuj    | vyoo      | yvvgzj     | bwxihx    |</span><span id="A3.SS1.SSS0.Px1.p1.1.11" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">|  8 | afzrom     | jzdra    | bndktpe   | jsvbugac   | mkxqnrhq  |</span><span id="A3.SS1.SSS0.Px1.p1.1.12" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">|  9 | ivxpmv     | eqciiims | bndktpe   | jsvbugac   | bwxihx    |</span><span id="A3.SS1.SSS0.Px1.p1.1.13" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">| 10 | ehfvur     | ezhuj    | tbwqa     | yuogpbo    | bwxihx    |</span><span id="A3.SS1.SSS0.Px1.p1.1.14" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">| 11 | bdzsy      | ezhuj    | bndktpe   | yvvgzj     | yjblg     |</span><span id="A3.SS1.SSS0.Px1.p1.1.15" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">| 12 | qruh       | ezhuj    | bndktpe   | dpimlb     | skbpzyhak |</span><span id="A3.SS1.SSS0.Px1.p1.1.16" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">| 13 | <span id="A3.SS1.SSS0.Px1.p1.1.16.1" class="ltx_text" style="color:#FF0000;">qytocp</span>     | jzdra    | <span id="A3.SS1.SSS0.Px1.p1.1.16.2" class="ltx_text" style="color:#BF0040;">jcrbb</span>     | dpimlb     | bwxihx    |</span><span id="A3.SS1.SSS0.Px1.p1.1.17" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">| 14 | eqaja      | ezhuj    | bndktpe   | haxyaz     | yjblg     |</span><span id="A3.SS1.SSS0.Px1.p1.1.18" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">| 15 | kwvzixe    | jzdra    | vyoo      | jsvbugac   | skbpzyhak |</span><span id="A3.SS1.SSS0.Px1.p1.1.19" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">| 16 | edmkxm     | eqciiims | vyoo      | haxyaz     | mkxqnrhq  |</span><span id="A3.SS1.SSS0.Px1.p1.1.20" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">| 17 | fdsdlcpxj  | eqciiims | vyoo      | dpimlb     | blqoislm  |</span><span id="A3.SS1.SSS0.Px1.p1.1.21" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">| 18 | ipprxzzlv  | cwzqkdte | bndktpe   | yuogpbo    | yjblg     |</span><span id="A3.SS1.SSS0.Px1.p1.1.22" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">| 19 | gqyxjtbz   | eqciiims | tbwqa     | dpimlb     | yjblg     |</span><span id="A3.SS1.SSS0.Px1.p1.1.23" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">| 20 | noqfw      | ezhuj    | vyoo      | haxyaz     | blqoislm  |</span><span id="A3.SS1.SSS0.Px1.p1.1.24" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">| 21 | <span id="A3.SS1.SSS0.Px1.p1.1.24.1" class="ltx_text" style="color:#FF0000;">vkfzhqwj</span>   | dtfjqfc  | <span id="A3.SS1.SSS0.Px1.p1.1.24.2" class="ltx_text" style="color:#BF0040;">jcrbb</span>     | yuogpbo    | mkxqnrhq  |</span><span id="A3.SS1.SSS0.Px1.p1.1.25" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">| 22 | konftq     | eqciiims | vyoo      | dpimlb     | bwxihx    |</span><span id="A3.SS1.SSS0.Px1.p1.1.26" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">| 23 | ymcwhu     | jzdra    | wvfesrtzt | dpimlb     | blqoislm  |</span><span id="A3.SS1.SSS0.Px1.p1.1.27" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">| 24 | kpygsu     | eqciiims | wvfesrtzt | yuogpbo    | yjblg     |</span><span id="A3.SS1.SSS0.Px1.p1.1.28" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">| 25 | tiwfvqgmt  | ezhuj    | bndktpe   | dpimlb     | mkxqnrhq  |</span><span id="A3.SS1.SSS0.Px1.p1.1.29" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">| 26 | ovomhf     | dtfjqfc  | bndktpe   | yuogpbo    | blqoislm  |</span><span id="A3.SS1.SSS0.Px1.p1.1.30" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">| 27 | lokwxn     | cwzqkdte | tbwqa     | yuogpbo    | mkxqnrhq  |</span><span id="A3.SS1.SSS0.Px1.p1.1.31" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">| 28 | <span id="A3.SS1.SSS0.Px1.p1.1.31.1" class="ltx_text" style="color:#FF0000;">xwijyubr</span>   | jzdra    | <span id="A3.SS1.SSS0.Px1.p1.1.31.2" class="ltx_text" style="color:#BF0040;">jcrbb</span>     | yuogpbo    | mkxqnrhq  |</span><span id="A3.SS1.SSS0.Px1.p1.1.32" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">| 29 | ttonww     | dtfjqfc  | wvfesrtzt | haxyaz     | blqoislm  |</span></p>
</div>
</section>
<section id="A3.SS1.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Dense Example:</h4>

<div id="A3.SS1.SSS0.Px2.p1" class="ltx_para ltx_noindent">
<p id="A3.SS1.SSS0.Px2.p1.1" class="ltx_p"><span id="A3.SS1.SSS0.Px2.p1.1.1" class="ltx_text ltx_font_typewriter"><span id="A3.SS1.SSS0.Px2.p1.1.1.1" class="ltx_text ltx_inline-block" style="width:0.0pt;"></span><span id="A3.SS1.SSS0.Px2.p1.1.1.2" class="ltx_text ltx_inline-block" style="width:469.8pt;">|    | boarfish   | tool     | sixties   | phoxinus   | angling   |</span></span><span id="A3.SS1.SSS0.Px2.p1.1.2" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">|---:|:-----------|:---------|:----------|:-----------|:----------|</span><span id="A3.SS1.SSS0.Px2.p1.1.3" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">|  0 | mjdsv      | cwzqkdte | tbwqa     | yuogpbo    | mkxqnrhq  |</span><span id="A3.SS1.SSS0.Px2.p1.1.4" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">|  1 | nrbmyc     | eqciiims | wvfesrtzt | yvvgzj     | mkxqnrhq  |</span><span id="A3.SS1.SSS0.Px2.p1.1.5" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">|  2 | iqdr       | ezhuj    | bndktpe   | yuogpbo    | yjblg     |</span><span id="A3.SS1.SSS0.Px2.p1.1.6" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">|  3 | xzrrs      | ezhuj    | bndktpe   | dpimlb     | skbpzyhak |</span><span id="A3.SS1.SSS0.Px2.p1.1.7" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">|  4 | pvugxgdju  | dtfjqfc  | bndktpe   | jsvbugac   | mkxqnrhq  |</span><span id="A3.SS1.SSS0.Px2.p1.1.8" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">|  5 | xpkuautv   | ezhuj    | vyoo      | yvvgzj     | bwxihx    |</span><span id="A3.SS1.SSS0.Px2.p1.1.9" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">|  6 | afzrom     | jzdra    | bndktpe   | jsvbugac   | mkxqnrhq  |</span><span id="A3.SS1.SSS0.Px2.p1.1.10" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">|  7 | ivxpmv     | eqciiims | bndktpe   | jsvbugac   | bwxihx    |</span><span id="A3.SS1.SSS0.Px2.p1.1.11" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">|  8 | ehfvur     | ezhuj    | tbwqa     | yuogpbo    | bwxihx    |</span><span id="A3.SS1.SSS0.Px2.p1.1.12" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">|  9 | bdzsy      | ezhuj    | bndktpe   | yvvgzj     | yjblg     |</span><span id="A3.SS1.SSS0.Px2.p1.1.13" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">| 10 | qruh       | ezhuj    | bndktpe   | dpimlb     | skbpzyhak |</span><span id="A3.SS1.SSS0.Px2.p1.1.14" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">| 11 | eqaja      | ezhuj    | bndktpe   | haxyaz     | yjblg     |</span><span id="A3.SS1.SSS0.Px2.p1.1.15" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">| 12 | kwvzixe    | jzdra    | vyoo      | jsvbugac   | skbpzyhak |</span><span id="A3.SS1.SSS0.Px2.p1.1.16" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">| 13 | <span id="A3.SS1.SSS0.Px2.p1.1.16.1" class="ltx_text" style="color:#FF0000;">qxgd</span>       | dtfjqfc  | <span id="A3.SS1.SSS0.Px2.p1.1.16.2" class="ltx_text" style="color:#BF0040;">jcrbb</span>     | haxyaz     | yjblg     |</span><span id="A3.SS1.SSS0.Px2.p1.1.17" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">| 14 | <span id="A3.SS1.SSS0.Px2.p1.1.17.1" class="ltx_text" style="color:#FF0000;">lorfaljob</span>  | eqciiims | <span id="A3.SS1.SSS0.Px2.p1.1.17.2" class="ltx_text" style="color:#BF0040;">jcrbb</span>     | jsvbugac   | bwxihx    |</span><span id="A3.SS1.SSS0.Px2.p1.1.18" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">| 15 | <span id="A3.SS1.SSS0.Px2.p1.1.18.1" class="ltx_text" style="color:#FF0000;">qytocp</span>     | jzdra    | <span id="A3.SS1.SSS0.Px2.p1.1.18.2" class="ltx_text" style="color:#BF0040;">jcrbb</span>     | dpimlb     | bwxihx    |</span><span id="A3.SS1.SSS0.Px2.p1.1.19" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">| 16 | <span id="A3.SS1.SSS0.Px2.p1.1.19.1" class="ltx_text" style="color:#FF0000;">vkfzhqwj</span>   | dtfjqfc  | <span id="A3.SS1.SSS0.Px2.p1.1.19.2" class="ltx_text" style="color:#BF0040;">jcrbb</span>     | yuogpbo    | mkxqnrhq  |</span><span id="A3.SS1.SSS0.Px2.p1.1.20" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">| 17 | <span id="A3.SS1.SSS0.Px2.p1.1.20.1" class="ltx_text" style="color:#FF0000;">xwijyubr</span>   | jzdra    | <span id="A3.SS1.SSS0.Px2.p1.1.20.2" class="ltx_text" style="color:#BF0040;">jcrbb</span>     | yuogpbo    | mkxqnrhq  |</span><span id="A3.SS1.SSS0.Px2.p1.1.21" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">| 18 | edmkxm     | eqciiims | vyoo      | haxyaz     | mkxqnrhq  |</span><span id="A3.SS1.SSS0.Px2.p1.1.22" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">| 19 | fdsdlcpxj  | eqciiims | vyoo      | dpimlb     | blqoislm  |</span><span id="A3.SS1.SSS0.Px2.p1.1.23" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">| 20 | ipprxzzlv  | cwzqkdte | bndktpe   | yuogpbo    | yjblg     |</span><span id="A3.SS1.SSS0.Px2.p1.1.24" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">| 21 | gqyxjtbz   | eqciiims | tbwqa     | dpimlb     | yjblg     |</span><span id="A3.SS1.SSS0.Px2.p1.1.25" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">| 22 | noqfw      | ezhuj    | vyoo      | haxyaz     | blqoislm  |</span><span id="A3.SS1.SSS0.Px2.p1.1.26" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">| 23 | konftq     | eqciiims | vyoo      | dpimlb     | bwxihx    |</span><span id="A3.SS1.SSS0.Px2.p1.1.27" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">| 24 | ymcwhu     | jzdra    | wvfesrtzt | dpimlb     | blqoislm  |</span><span id="A3.SS1.SSS0.Px2.p1.1.28" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">| 25 | kpygsu     | eqciiims | wvfesrtzt | yuogpbo    | yjblg     |</span><span id="A3.SS1.SSS0.Px2.p1.1.29" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">| 26 | tiwfvqgmt  | ezhuj    | bndktpe   | dpimlb     | mkxqnrhq  |</span><span id="A3.SS1.SSS0.Px2.p1.1.30" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">| 27 | ovomhf     | dtfjqfc  | bndktpe   | yuogpbo    | blqoislm  |</span><span id="A3.SS1.SSS0.Px2.p1.1.31" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">| 28 | lokwxn     | cwzqkdte | tbwqa     | yuogpbo    | mkxqnrhq  |</span><span id="A3.SS1.SSS0.Px2.p1.1.32" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">| 29 | ttonww     | dtfjqfc  | wvfesrtzt | haxyaz     | blqoislm  |</span></p>
</div>
</section>
</section>
<section id="A3.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">C.2 </span>SQL Template</h3>

<section id="A3.SS2.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">General:</h4>

<div id="A3.SS2.SSS0.Px1.p1" class="ltx_para ltx_noindent">
<pre id="A3.SS2.SSS0.Px1.p1.1" class="ltx_verbatim ltx_font_typewriter">
"s1": "select &lt;select_condition&gt; from my_table",
"s2": "select &lt;select_condition&gt; from my_table &lt;where_condition&gt;",
"s3": "select &lt;select_condition&gt; from my_table &lt;order_condition&gt;",
"s4": "select &lt;select_condition&gt; from my_table &lt;where_condition&gt; &lt;order_condition&gt;",
"s5": "select &lt;select_condition&gt; from my_table &lt;group_condition&gt; &lt;having_condition&gt;",
"s6": "select &lt;select_condition&gt; from my_table &lt;where_condition&gt; &lt;group_condition&gt; &lt;having_condition&gt;",
"s7": "select &lt;select_condition&gt; from my_table &lt;where_condition&gt;
&lt;group_condition&gt; &lt;having_condition&gt; &lt;order_condition&gt;",
"s8": "select &lt;select_condition&gt; from my_table &lt;group_condition&gt; &lt;having_condition&gt; &lt;order_condition&gt;",
"d1": "select ( [s1,s2] ) &lt;op_and_value&gt;",
"d2": "select &lt;select_condition&gt; from my_table where &lt;col_and_op&gt; ( [s1,s2,s3,s4] )",
"d3": "select &lt;select_condition&gt; from my_table where &lt;col_and_op&gt; ( [s1,s2,s3] ) &lt;order_condition&gt;",
"d4": "select &lt;select_condition&gt; from my_table &lt;where_condition&gt; and &lt;col_and_op&gt; ( [s1,s2,s3,s4] )",
"t1": "select ( [s1,s2,s3] ) &lt;op&gt; ( [s1,s2,s3] )"
</pre>
</div>
</section>
<section id="A3.SS2.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Where Condition:</h4>

<div id="A3.SS2.SSS0.Px2.p1" class="ltx_para ltx_noindent">
<pre id="A3.SS2.SSS0.Px2.p1.1" class="ltx_verbatim ltx_font_typewriter">
select &lt;text_col1&gt; from my_table where &lt;text_col2&gt; = &lt;text_2&gt;
</pre>
</div>
</section>
<section id="A3.SS2.SSS0.Px3" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Count:</h4>

<div id="A3.SS2.SSS0.Px3.p1" class="ltx_para ltx_noindent">
<pre id="A3.SS2.SSS0.Px3.p1.1" class="ltx_verbatim ltx_font_typewriter">
Select Count(&lt;text_col1&gt;) from table where &lt;text_col1&gt; = &lt;text_1&gt;
</pre>
</div>
</section>
<section id="A3.SS2.SSS0.Px4" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Easy:</h4>

<div id="A3.SS2.SSS0.Px4.p1" class="ltx_para ltx_noindent">
<pre id="A3.SS2.SSS0.Px4.p1.1" class="ltx_verbatim ltx_font_typewriter">
select &lt;text_col1&gt; from my_table where &lt;int_col1&gt; = &lt;int_1&gt;
select &lt;int_col1&gt; from my_table where &lt;text_col1&gt; = &lt;text_1&gt;
select &lt;int_col1&gt; from my_table where &lt;int_col2&gt; = &lt;int_2&gt;
select &lt;text_col1&gt; from my_table where &lt;text_col2&gt; = &lt;text_2&gt;
</pre>
</div>
</section>
<section id="A3.SS2.SSS0.Px5" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Filter:</h4>

<div id="A3.SS2.SSS0.Px5.p1" class="ltx_para ltx_noindent">
<pre id="A3.SS2.SSS0.Px5.p1.1" class="ltx_verbatim ltx_font_typewriter">
select &lt;text_col1&gt; from my_table where &lt;text_col2&gt; = &lt;text_2&gt;
select &lt;text_col1&gt; from my_table where &lt;int_col2&gt; &lt;op2&gt; &lt;int_2&gt;
select &lt;text_col1&gt; from my_table where &lt;text_col2&gt; = &lt;text_2&gt; and &lt;int_col1&gt; &lt;op1&gt; &lt;int_1&gt;
select &lt;text_col1&gt; from my_table where &lt;text_col2&gt; = &lt;text_2&gt; and &lt;text_col3&gt; = &lt;text_3&gt;
select &lt;text_col1&gt; from my_table where &lt;int_col1&gt; &lt;op1&gt; &lt;int_1&gt; and &lt;int_col2&gt; &lt;op2&gt; &lt;int_2&gt;
select &lt;int_col1&gt; from my_table where &lt;text_col1&gt; = &lt;text_1&gt;
select &lt;int_col1&gt; from my_table where &lt;int_col2&gt; &lt;op2&gt; &lt;int_2&gt;
select &lt;int_col1&gt; from my_table where &lt;text_col2&gt; = &lt;text_2&gt; and &lt;int_col2&gt; &lt;op2&gt; &lt;int_2&gt;
select &lt;int_col1&gt; from my_table where &lt;text_col2&gt; = &lt;text_2&gt; and &lt;text_col3&gt; = &lt;text_3&gt;
select &lt;int_col1&gt; from my_table where &lt;int_col2&gt; &lt;op2&gt; &lt;int_2&gt; and &lt;int_col3&gt; &lt;op3&gt; &lt;int_3&gt;

</pre>
</div>
</section>
<section id="A3.SS2.SSS0.Px6" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Aggregate:</h4>

<div id="A3.SS2.SSS0.Px6.p1" class="ltx_para ltx_noindent">
<pre id="A3.SS2.SSS0.Px6.p1.1" class="ltx_verbatim ltx_font_typewriter">
select count ( &lt;text_col1&gt; ) from my_table where &lt;text_col2&gt; = &lt;text_2&gt;
select count ( &lt;text_col1&gt; ) from my_table where &lt;int_col2&gt; &lt;op2&gt; &lt;int_2&gt;
select sum ( &lt;int_col1&gt; ) from my_table
select sum ( &lt;int_col1&gt; ) from my_table where &lt;text_col2&gt; = &lt;text_2&gt;
select max ( &lt;int_col1&gt; ) from my_table
select max ( &lt;int_col1&gt; ) from my_table where &lt;text_col2&gt; = &lt;text_2&gt;
select min ( &lt;int_col1&gt; ) from my_table
select min ( &lt;int_col1&gt; ) from my_table where &lt;text_col2&gt; = &lt;text_2&gt;
</pre>
</div>
</section>
<section id="A3.SS2.SSS0.Px7" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Arithmetic:</h4>

<div id="A3.SS2.SSS0.Px7.p1" class="ltx_para ltx_noindent">
<pre id="A3.SS2.SSS0.Px7.p1.1" class="ltx_verbatim ltx_font_typewriter">
select &lt;int_col1&gt; + &lt;int_col2&gt; from my_table where &lt;text_col1&gt; = &lt;text_1&gt;
select &lt;int_col1&gt; + &lt;int_col2&gt; from my_table where &lt;text_col1&gt; = &lt;text_1&gt; and &lt;text_col2&gt; = &lt;text_2&gt;
select &lt;int_col1&gt; - &lt;int_col2&gt; from my_table where &lt;text_col1&gt; = &lt;text_1&gt;
select &lt;int_col1&gt; - &lt;int_col2&gt; from my_table where &lt;text_col1&gt; = &lt;text_1&gt; and &lt;text_col2&gt; = &lt;text_2&gt;
</pre>
</div>
</section>
<section id="A3.SS2.SSS0.Px8" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Superlative:</h4>

<div id="A3.SS2.SSS0.Px8.p1" class="ltx_para ltx_noindent">
<pre id="A3.SS2.SSS0.Px8.p1.1" class="ltx_verbatim ltx_font_typewriter">
select &lt;int_col1&gt; from my_table order by &lt;int_col1&gt; asc limit 1
select &lt;int_col1&gt; from my_table order by &lt;int_col1&gt; desc limit 1
select &lt;text_col1&gt; from my_table order by &lt;int_col1&gt; asc limit 1
select &lt;text_col1&gt; from my_table order by &lt;int_col1&gt; desc limit 1
select &lt;int_col1&gt; from my_table order by &lt;int_col2&gt; asc limit 1
select &lt;int_col1&gt; from my_table order by &lt;int_col2&gt; desc limit 1
</pre>
</div>
</section>
<section id="A3.SS2.SSS0.Px9" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Comparative:</h4>

<div id="A3.SS2.SSS0.Px9.p1" class="ltx_para ltx_noindent">
<pre id="A3.SS2.SSS0.Px9.p1.1" class="ltx_verbatim ltx_font_typewriter">
select ( select &lt;int_col1&gt; from my_table where &lt;text_col1&gt; = &lt;text_1&gt; )
&gt; ( select &lt;int_col1&gt; from my_table where &lt;text_col2&gt; = &lt;text_2&gt; )
select ( select &lt;int_col1&gt; from my_table where &lt;int_col2&gt; &lt;op2&gt; &lt;int_2&gt; )
&gt; ( select &lt;int_col1&gt; from my_table where &lt;int_col3&gt; &lt;op3&gt; &lt;int_3&gt; )
select ( select &lt;int_col1&gt; from my_table where &lt;text_col1&gt; = &lt;text_1&gt; )
&lt; ( select &lt;int_col1&gt; from my_table where &lt;text_col2&gt; = &lt;text_2&gt; )
select ( select &lt;int_col1&gt; from my_table where &lt;int_col2&gt; &lt;op2&gt; &lt;int_2&gt; )
&lt; ( select &lt;int_col1&gt; from my_table where &lt;int_col3&gt; &lt;op3&gt; &lt;int_3&gt; )
select &lt;int_col1&gt; &gt; &lt;int_col2&gt; from my_table where &lt;text_col1&gt; = &lt;text_1&gt;
select &lt;int_col1&gt; &lt; &lt;int_col2&gt; from my_table where &lt;text_col1&gt; = &lt;text_1&gt;
select &lt;int_col1&gt; &gt; &lt;int_col2&gt; from my_table where &lt;int_col3&gt; &lt;op3&gt; &lt;int_3&gt;
select &lt;int_col1&gt; &lt; &lt;int_col2&gt; from my_table where &lt;int_col3&gt; &lt;op3&gt; &lt;int_3&gt;
</pre>
</div>
</section>
</section>
<section id="A3.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">C.3 </span>Table Input Format</h3>

<section id="A3.SS3.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Markdown Table:</h4>

<div id="A3.SS3.SSS0.Px1.p1" class="ltx_para ltx_noindent">
<pre id="A3.SS3.SSS0.Px1.p1.1" class="ltx_verbatim ltx_font_typewriter">
|    |   ercilla |   shucks | liter    | taenia    | dorado    |
|---:|----------:|---------:|:---------|:----------|:----------|
|  0 |        68 |       12 | gcrdvo   | qoath     | katfuw    |
|  1 |       129 |      151 | zmvltkk  | jpcglcjzk | vwqqey    |
|  2 |       248 |      188 | zmdlfbhb | cvhqotsys | wzunmaa   |
|  3 |       267 |      104 | gcrdvo   | ytywunvf  | pjlbo     |
|  4 |       135 |      262 | gcrdvo   | dtnvfp    | ajzpsaoy  |
|  5 |       309 |      119 | zmdlfbhb | klcenmugk | hriunhf   |
|  6 |        25 |      152 | zmvltkk  | cjgcergv  | shrbvrd   |
|  7 |       298 |       18 | zmvltkk  | scvuuc    | ahunvcx   |
|  8 |       321 |      217 | gcrdvo   | ezlp      | hasjaznm  |
|  9 |       139 |      310 | gcrdvo   | ghhjea    | atqvtgoa  |
| 10 |        99 |       34 | zmvltkk  | ecdmpruq  | cfitvz    |
| 11 |       142 |      167 | gcrdvo   | acii      | oenmuezip |
| 12 |       273 |      156 | gcrdvo   | nnvnteh   | tulh      |
| 13 |       197 |       44 | gcrdvo   | pqdbhevkh | dfxuwxz   |
| 14 |       144 |      123 | gcrdvo   | bxrgo     | ccbj      |
</pre>
</div>
</section>
<section id="A3.SS3.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Flatten Table:</h4>

<div id="A3.SS3.SSS0.Px2.p1" class="ltx_para ltx_noindent">
<pre id="A3.SS3.SSS0.Px2.p1.1" class="ltx_verbatim ltx_font_typewriter">
Flatten Table Examples:
The table have 5 columns: ercilla | shucks | liter | taenia | dorado
row 1 : ercilla is 68. shucks is 12. liter is gcrdvo. taenia is qoath. dorado is katfuw.
row 2 : ercilla is 129. shucks is 151. liter is zmvltkk. taenia is jpcglcjzk. dorado is vwqqey.
row 3 : ercilla is 248. shucks is 188. liter is zmdlfbhb. taenia is cvhqotsys. dorado is wzunmaa.
row 4 : ercilla is 267. shucks is 104. liter is gcrdvo. taenia is ytywunvf. dorado is pjlbo.
row 5 : ercilla is 135. shucks is 262. liter is gcrdvo. taenia is dtnvfp. dorado is ajzpsaoy.
row 6 : ercilla is 309. shucks is 119. liter is zmdlfbhb. taenia is klcenmugk. dorado is hriunhf.
row 7 : ercilla is 25. shucks is 152. liter is zmvltkk. taenia is cjgcergv. dorado is shrbvrd.
row 8 : ercilla is 298. shucks is 18. liter is zmvltkk. taenia is scvuuc. dorado is ahunvcx.
row 9 : ercilla is 321. shucks is 217. liter is gcrdvo. taenia is ezlp. dorado is hasjaznm.
row 10 : ercilla is 139. shucks is 310. liter is gcrdvo. taenia is ghhjea. dorado is atqvtgoa.
row 11 : ercilla is 99. shucks is 34. liter is zmvltkk. taenia is ecdmpruq. dorado is cfitvz.
row 12 : ercilla is 142. shucks is 167. liter is gcrdvo. taenia is acii. dorado is oenmuezip.
row 13 : ercilla is 273. shucks is 156. liter is gcrdvo. taenia is nnvnteh. dorado is tulh.
row 14 : ercilla is 197. shucks is 44. liter is gcrdvo. taenia is pqdbhevkh. dorado is dfxuwxz.
row 15 : ercilla is 144. shucks is 123. liter is gcrdvo. taenia is bxrgo. dorado is ccbj.
</pre>
</div>
</section>
</section>
<section id="A3.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">C.4 </span>SQL Execution Examples (Few-shot)</h3>

<section id="A3.SS4.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph"></h4>

<div id="A3.SS4.SSS0.Px1.p1" class="ltx_para ltx_noindent">
<pre id="A3.SS4.SSS0.Px1.p1.1" class="ltx_verbatim ltx_font_typewriter">
You are an SQL executor, you need to execute SQL based on the give table and SQL statement
to obtain the execution results.
Only give me the execution results and do not output any other words.
Table:
|    |   puccoon |   tiepolo |   scope | mutinus    |   intrados | huggins   |   barye |   wear |
|---:|----------:|----------:|--------:|:-----------|-----------:|:----------|--------:|-------:|
|  0 |       171 |       225 |     145 | 2007-04-27 |        322 | yefihroyn |      79 |    207 |
|  1 |       213 |       116 |     319 | 2016-01-15 |        288 | ytyayrvj  |     246 |    272 |
|  2 |       191 |       229 |      95 | 2022-11-08 |        218 | gpmvax    |     167 |     73 |
|  3 |        97 |       155 |     189 | 2013-10-30 |         79 | gpmvax    |      24 |    233 |
|  4 |        56 |        11 |     295 | 2018-12-10 |         81 | yefihroyn |     187 |    198 |
|  5 |       285 |       304 |     168 | 2017-03-24 |         75 | gpmvax    |     111 |     77 |
|  6 |       233 |       325 |      31 | 2014-01-22 |        114 | ytyayrvj  |      20 |    219 |
|  7 |        19 |       146 |     164 | 2021-12-07 |        311 | ytyayrvj  |     188 |      3 |
|  8 |       112 |       255 |      30 | 2015-12-07 |        214 | gpmvax    |      16 |    271 |
|  9 |       175 |        62 |     181 | 2012-04-21 |        182 | gpmvax    |     105 |     76 |
| 10 |       200 |        90 |     101 | 2008-04-28 |        168 | gpmvax    |      70 |    119 |
| 11 |        31 |       180 |      95 | 2004-06-23 |         62 | yefihroyn |     314 |     97 |
| 12 |       297 |       251 |     249 | 2022-02-02 |        185 | yefihroyn |     278 |    313 |
| 13 |        36 |        17 |      67 | 2016-04-14 |        243 | ytyayrvj  |     213 |      4 |
| 14 |        45 |       215 |     182 | 2012-06-15 |        251 | yefihroyn |     221 |     83 |
Now you need to execute SQL based on the given table and SQL statement to obtain the execution result.
Only give me the result and do not output any other words or SQL statement.
The following are some examples.

SQL:select avg ( intrados ) from my_table where tiepolo &gt; 146 group by huggins
having count ( huggins ) &gt; 1 order by count ( tiepolo ) asc limit 1
Answer:146.5
SQL:select wear from my_table where huggins = ’gpmvax’ group by huggins
having wear &lt; 83 order by count ( distinct barye ) asc limit 1
Answer:73
SQL:select mutinus from my_table where tiepolo &gt; 116 group by huggins
having max ( wear ) &gt; 119 order by count ( huggins ) asc limit 1
Answer:2014-01-22
SQL:select tiepolo from my_table where puccoon &lt; 191 and intrados &lt; 79 group by huggins
having intrados &lt; 81 and tiepolo &lt; 255 order by count ( barye ) asc limit 1
Answer:180
SQL:select tiepolo from my_table where scope &gt; 31 group by huggins
having min ( tiepolo ) = 62 order by count ( distinct mutinus ) asc limit 1
Answer:62
SQL:select wear from my_table where huggins = ’ytyayrvj’ group by huggins
having count ( huggins ) &lt; 5 order by count ( distinct mutinus ) desc limit 1
Answer:
</pre>
</div>
</section>
</section>
<section id="A3.SS5" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">C.5 </span>Multi-step Instruction (Few-shot)</h3>

<section id="A3.SS5.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph"></h4>

<div id="A3.SS5.SSS0.Px1.p1" class="ltx_para ltx_noindent">
<pre id="A3.SS5.SSS0.Px1.p1.1" class="ltx_verbatim ltx_font_typewriter">
You need to obtain the final answer based on the table and instructions.
Only give me the result and do not output any other words.
Table:
|    |   puccoon |   tiepolo |   scope | mutinus    |   intrados | huggins   |   barye |   wear |
|---:|----------:|----------:|--------:|:-----------|-----------:|:----------|--------:|-------:|
|  0 |       171 |       225 |     145 | 2007-04-27 |        322 | yefihroyn |      79 |    207 |
|  1 |       213 |       116 |     319 | 2016-01-15 |        288 | ytyayrvj  |     246 |    272 |
|  2 |       191 |       229 |      95 | 2022-11-08 |        218 | gpmvax    |     167 |     73 |
|  3 |        97 |       155 |     189 | 2013-10-30 |         79 | gpmvax    |      24 |    233 |
|  4 |        56 |        11 |     295 | 2018-12-10 |         81 | yefihroyn |     187 |    198 |
|  5 |       285 |       304 |     168 | 2017-03-24 |         75 | gpmvax    |     111 |     77 |
|  6 |       233 |       325 |      31 | 2014-01-22 |        114 | ytyayrvj  |      20 |    219 |
|  7 |        19 |       146 |     164 | 2021-12-07 |        311 | ytyayrvj  |     188 |      3 |
|  8 |       112 |       255 |      30 | 2015-12-07 |        214 | gpmvax    |      16 |    271 |
|  9 |       175 |        62 |     181 | 2012-04-21 |        182 | gpmvax    |     105 |     76 |
| 10 |       200 |        90 |     101 | 2008-04-28 |        168 | gpmvax    |      70 |    119 |
| 11 |        31 |       180 |      95 | 2004-06-23 |         62 | yefihroyn |     314 |     97 |
| 12 |       297 |       251 |     249 | 2022-02-02 |        185 | yefihroyn |     278 |    313 |
| 13 |        36 |        17 |      67 | 2016-04-14 |        243 | ytyayrvj  |     213 |      4 |
| 14 |        45 |       215 |     182 | 2012-06-15 |        251 | yefihroyn |     221 |     83 |
Now you need to get the answer based on the instruction,
only give me the result and do not output any other words.
The following are some examples.

Instruction:Please filter the rows by the column conditions, which need to be met:
The value of column tiepolo needs to be greater than 146.
The rows are then grouped according to the value of the huggins in the remaining rows.
Then filter some groups by the following condition:the number of column huggins is greater than 1.
Select the average of values of intrados column in filtered rows.
Sort the obtained values in ascending order of the number of tiepolo
and select the smallest value to get the answer.
Answer:146.5

Instruction:Please filter the rows by the column conditions, which need to be met:
The value of column huggins is ’gpmvax’.
The rows are then grouped according to the value of the huggins in the remaining rows.
Then filter some groups by the following condition:the column wear is less than 83.
Select values of wear column in filtered rows.
Sort the obtained values in ascending order of the number of non-repeating barye
and select the smallest value to get the answer.
Answer:73

Instruction:Please filter the rows by the column conditions, which need to be met:
The value of column huggins is ’ytyayrvj’.
The rows are then grouped according to the value of the huggins in the remaining rows.
Then filter some groups by the following condition:the number of column huggins is less than 5.
Select values of wear column in filtered rows.
Sort the obtained values in descending order of the number of non-repeating mutinus
and select the largest value to get the answer.
Answer:
</pre>
</div>
</section>
</section>
<section id="A3.SS6" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">C.6 </span>Chain-of-Thought SQL Execution Prompting Examples</h3>

<div id="A3.SS6.p1" class="ltx_para ltx_noindent">
<pre id="A3.SS6.p1.1" class="ltx_verbatim ltx_font_typewriter">
You are an SQL executor, you need to output the execution process and final answer based on table and SQL.
Table:
|    | masthead   | laertes   |   boo | bothrops   | height   | scraper   | trouser   |   lozenge |
|---:|:-----------|:----------|------:|:-----------|:---------|:----------|:----------|----------:|
|  0 | case       | araeswrid |    41 | lyucg      | urbsmxiv | vgxrh     | esauw     |       281 |
|  1 | case       | araeswrid |   138 | lyucg      | tbvg     | oerigocb  | stevw     |       177 |
|  2 | case       | zncmrrvg  |   303 | loclzoglg  | tbvg     | vgxrh     | stevw     |       234 |
|  3 | thyngfwts  | araeswrid |   288 | loclzoglg  | tbvg     | vgxrh     | esauw     |       224 |
|  4 | thyngfwts  | mrehctv   |   177 | loclzoglg  | urbsmxiv | vgxrh     | esauw     |       228 |
|  5 | case       | araeswrid |   163 | loclzoglg  | urbsmxiv | oerigocb  | stevw     |        60 |
|  6 | thyngfwts  | mrehctv   |    45 | loclzoglg  | cidufm   | oerigocb  | esauw     |       289 |
|  7 | thyngfwts  | zncmrrvg  |    42 | loclzoglg  | tbvg     | ffljyxb   | stevw     |       296 |
|  8 | case       | araeswrid |   275 | lyucg      | cidufm   | vgxrh     | stevw     |       172 |
|  9 | case       | mrehctv   |    20 | loclzoglg  | tbvg     | vgxrh     | esauw     |       147 |
| 10 | thyngfwts  | araeswrid |   302 | lyucg      | urbsmxiv | vgxrh     | stevw     |       297 |
| 11 | thyngfwts  | zncmrrvg  |   137 | loclzoglg  | tbvg     | vgxrh     | esauw     |        63 |
| 12 | case       | araeswrid |   186 | loclzoglg  | cidufm   | ffljyxb   | esauw     |       268 |
| 13 | case       | araeswrid |   194 | loclzoglg  | cidufm   | vgxrh     | esauw     |        98 |
| 14 | case       | araeswrid |   234 | lyucg      | urbsmxiv | vgxrh     | stevw     |       276 |
Now you need to get the answer based on the instruction,
only give me the intermedium results and the final answer.
SQL:
select masthead from my_table where height = ’tbvg’ group by masthead order by count ( laertes ) desc limit 1
Execution process:
You need to execute 3 steps.
Step 0:
Please filter the rows by the column conditions, which need to be met: The value of column butcher is ’jxys’.
Intermediate results 0:
|    | encyclia   | butcher   | bowdler   | nuthatch   |   cachexia | claret   |   cortina |   strombus |
|---:|:-----------|:----------|:----------|:-----------|-----------:|:---------|----------:|-----------:|
|  0 | adnh       | jxys      | cxjvfz    | clmb       |          2 | oqmdmbfg |       251 |        184 |
|  1 | xvoxfjbm   | jxys      | cxjvfz    | clmb       |        275 | oqmdmbfg |       140 |        303 |
|  2 | adnh       | jxys      | eohdpivo  | clmb       |        298 | oqmdmbfg |       142 |         28 |
|  3 | adnh       | jxys      | eohdpivo  | rcyixdl    |        153 | oqmdmbfg |        50 |        306 |
|  4 | xvoxfjbm   | jxys      | eohdpivo  | rcyixdl    |        315 | rxbttbm  |       201 |         86 |
Step 1: Select values of strombus column in filtered rows.
Intermediate results 1:
184,303,28,306,86
Step 2: Sort the obtained values in ascending order of claret and select the smallest value to get the answer.
Answer: 184
</pre>
</div>
</section>
<section id="A3.SS7" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">C.7 </span>Real Table SQL Execution (Few-shot)</h3>

<div id="A3.SS7.p1" class="ltx_para ltx_noindent">
<pre id="A3.SS7.p1.1" class="ltx_verbatim ltx_font_typewriter">
You are an SQL executor, you need to execute SQL based on the give table
and SQL statement to obtain the execution results.
Only give me the execution results and do not output any other words.
Table:
|    |   id |   agg |   rank | nation             |   gold |   silver |   bronze |   total |
|---:|-----:|------:|-------:|:-------------------|-------:|---------:|---------:|--------:|
|  0 |    1 |     0 |      1 | soviet union       |     50 |       27 |       22 |      99 |
|  1 |    2 |     0 |      2 | united states      |     33 |       31 |       30 |      94 |
|  2 |    3 |     0 |      3 | east germany (gdr) |     20 |       23 |       23 |      66 |
|  3 |    4 |     0 |      4 | west germany (frg) |     13 |       11 |       16 |      40 |
|  4 |    5 |     0 |      5 | japan              |     13 |        8 |        8 |      29 |
|  5 |    6 |     0 |      6 | australia          |      8 |        7 |        2 |      17 |
|  6 |    7 |     0 |      7 | poland             |      7 |        5 |        9 |      21 |
|  7 |    8 |     0 |      8 | hungary            |      6 |       13 |       16 |      35 |
|  8 |    9 |     0 |      9 | bulgaria           |      6 |       10 |        5 |      21 |
|  9 |   10 |     0 |     10 | italy              |      5 |        3 |       10 |      18 |
Now you need to execute SQL based on the given table and SQL statement to obtain the execution result.
Only give me the result and do not output any other words or SQL statement.
The following are some examples.

SQL:select nation from table where rank = 1
Answer:Soviet Union
SQL:select nation from table where nation != ’bulgaria’
and total = ( select total from table where nation = ’bulgaria’ )
Answer:Poland
SQL:select nation from table order by bronze limit 1
Answer:Australia
SQL:select nation from table order by bronze limit 1
Answer:Australia
SQL:select silver from table order by gold desc limit 1
Answer:
</pre>
</div>
</section>
<section id="A3.SS8" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">C.8 </span>Real Table Question Answering (Few-shot)</h3>

<div id="A3.SS8.p1" class="ltx_para ltx_noindent">
<p id="A3.SS8.p1.1" class="ltx_p"><span id="A3.SS8.p1.1.1" class="ltx_text ltx_font_typewriter"><span id="A3.SS8.p1.1.1.1" class="ltx_text ltx_inline-block" style="width:0.0pt;"></span><span id="A3.SS8.p1.1.1.2" class="ltx_text ltx_inline-block" style="width:469.8pt;">You need to obtain the final answer based on the table and questions.</span></span><span id="A3.SS8.p1.1.2" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">Only give me the answer and do not output any other words.</span><span id="A3.SS8.p1.1.3" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">Table:</span><span id="A3.SS8.p1.1.4" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">|    |   id |   agg |   rank | nation             |   gold |   silver |   bronze |   total |</span><span id="A3.SS8.p1.1.5" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">|---:|-----:|------:|-------:|:-------------------|-------:|---------:|---------:|--------:|</span><span id="A3.SS8.p1.1.6" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">|  0 |    1 |     0 |      1 | soviet union       |     50 |       27 |       22 |      99 |</span><span id="A3.SS8.p1.1.7" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">|  1 |    2 |     0 |      2 | united states      |     33 |       31 |       30 |      94 |</span><span id="A3.SS8.p1.1.8" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">|  2 |    3 |     0 |      3 | east germany (gdr) |     20 |       23 |       23 |      66 |</span><span id="A3.SS8.p1.1.9" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">|  3 |    4 |     0 |      4 | west germany (frg) |     13 |       11 |       16 |      40 |</span><span id="A3.SS8.p1.1.10" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">|  4 |    5 |     0 |      5 | japan              |     13 |        8 |        8 |      29 |</span><span id="A3.SS8.p1.1.11" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">|  5 |    6 |     0 |      6 | australia          |      8 |        7 |        2 |      17 |</span><span id="A3.SS8.p1.1.12" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">|  6 |    7 |     0 |      7 | poland             |      7 |        5 |        9 |      21 |</span><span id="A3.SS8.p1.1.13" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">|  7 |    8 |     0 |      8 | hungary            |      6 |       13 |       16 |      35 |</span><span id="A3.SS8.p1.1.14" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">|  8 |    9 |     0 |      9 | bulgaria           |      6 |       10 |        5 |      21 |</span><span id="A3.SS8.p1.1.15" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">|  9 |   10 |     0 |     10 | italy              |      5 |        3 |       10 |      18 |</span><span id="A3.SS8.p1.1.16" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">Now you need to get the answer based on the question,</span><span id="A3.SS8.p1.1.17" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">only give me the answer and do not output any other words.</span><span id="A3.SS8.p1.1.18" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">The following are some examples.</span><span id="A3.SS8.p1.1.19" class="ltx_text ltx_inline-block" style="width:469.8pt;"></span><span id="A3.SS8.p1.1.20" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">Question:which country was first in rank at the 1972 olympics ?</span><span id="A3.SS8.p1.1.21" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">Answer:Soviet Union</span><span id="A3.SS8.p1.1.22" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">Question:which country won the same amount of medals as bulgaria in these olympics ?</span><span id="A3.SS8.p1.1.23" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">Answer:Poland</span><span id="A3.SS8.p1.1.24" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">Question:which nation won the least number of bronze medals ?</span><span id="A3.SS8.p1.1.25" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">Answer:Australia</span><span id="A3.SS8.p1.1.26" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">Question:which nation received the least bronze medals</span><span id="A3.SS8.p1.1.27" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">Answer:Australia</span><span id="A3.SS8.p1.1.28" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">Question:what number of silver medals was won by the nation with the most gold medals ?</span><span id="A3.SS8.p1.1.29" class="ltx_text ltx_font_typewriter ltx_inline-block" style="width:469.8pt;">Answer:</span></p>
</div>
</section>
</section>
<section id="A4" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix D </span>Experiments Settings Details</h2>

<section id="A4.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">D.1 </span>Setting Description</h3>

<section id="A4.SS1.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Table Config</h4>

<div id="A4.SS1.SSS0.Px1.p1" class="ltx_para ltx_noindent">
<pre id="A4.SS1.SSS0.Px1.p1.1" class="ltx_verbatim ltx_font_typewriter">
"col_min": 5, // the min number of cols
"col_max": 8, // the max number of cols
"row_min": 15,  // the min number of rows
"row_max": 40,  // the max number of rows
"text_int_date": [0.55, 0.35, 0.1], // text,int,date  header ratio
"text_int_date_fix": ["TEXT", "TEXT", "INT", "INT", "INT"], // Specify the type of each header
// Probability of duplicate values in each column
"value_repeat_ratio": [0, 0.2, 0.3, 0, 0, 0, 0, 0, 0.2, 0.5],
"value_repeat_ratio_fix": ["random", "random"], // Specify the duplicate values of each column
</pre>
</div>
</section>
<section id="A4.SS1.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">SQL Config</h4>

<div id="A4.SS1.SSS0.Px2.p1" class="ltx_para ltx_noindent">
<pre id="A4.SS1.SSS0.Px2.p1.1" class="ltx_verbatim ltx_font_typewriter">
"nest": [1],  // Number of SQL nestings. options: [1], [2], [1,2],[1,2, 3]
"keywords_setting": { // if a Keyword is False, then no SQL containing this Keyword is generated.
"select": true,
"where": true,
"group by": true,
"having": true,
"order by": true
},
"length_setting": {      // control the length of sql
"is_available": false, // To enable this setting, you need to adjust "is_available" to true first.
// ’value’ can be set to specific values, such as [13,14,15],
// if value is null, then the range is used [min, max]
"value": [],
"min": 6,
"max": 16
},
"column_ratio": {        // Controlling the ratio of columns involved in SQL
"is_available": false,  // To enable this setting, you need to adjust "is_available" to true first.
// ’value’ can be set to specific values, such as [1,2], Control the number of columns involved in SQL
"value": [],
// if value is null, then the range is used [min, max], it’s the used ratio = (used columns) / (all columns)
"min": 0.1,
"max": 0.3
},
"select_row_ratio":{     // Controlling the ratio of rows involved in select keyword
"is_available": false,  // To enable this setting, you need to adjust "is_available" to true first.
// ’value’ can be set to specific values, such as [1,2,3,4], Control the number of rows involved in SQL
"value": [],
// if value is null, then the range is used [min, max], it’s the used ratio = (select rows) / (all rows)
"min": 0.1,
"max": 0.2
},
// Controlling the calculate times of the sql [’+’,’-’,’*’,’/’,’sum’,’count’,’min’,’max’,’avg’]
"calculate_times": {
"is_available": false,   // To enable this setting, you need to adjust "is_available" to true first.
"value": [1,2,3,4]      // ’value’ can be set to specific values, means the calculate times
},
// Controlling the filter times of the sql [’=’,’&gt;’,’&lt;’,’in’,’like’]
"filter_times": {
"is_available": false,   // To enable this setting, you need to adjust "is_available" to true first.
"value": [1,2,3,4,5]  // ’value’ can be set to specific values, means the calculate times
},
// Controlling the location of answer in the table, usually used in long-context understanding
"answer_location": {
"is_available": false,   // To enable this setting, you need to adjust "is_available" to true first.
"value": null,
"min": 0.1, // if value is null, then the range is used [min, max],
means that  0.1 &lt;  (Row where answer is located ) / (Row number) &lt; 0.9
"max": 0.9
},
// usually remains 1 in this repo, we often just test the sql whose answer is from one cell.
"answer_cells_number": 1,
"include": [],
"exclude": [],
"n_shot": 5
</pre>
</div>
</section>
</section>
<section id="A4.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">D.2 </span>General Setting</h3>

<section id="A4.SS2.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Table Config</h4>

<div id="A4.SS2.SSS0.Px1.p1" class="ltx_para ltx_noindent">
<pre id="A4.SS2.SSS0.Px1.p1.1" class="ltx_verbatim ltx_font_typewriter">
"col_min": 5,
"col_max": 5,
"row_min": 30,
"row_max": 30,
"text_int_date": [0.5, 0.45, 0.05],
"value_repeat_ratio": [0, 0.2, 0.3, 0, 0, 0, 0, 0, 0, 0.5]
</pre>
</div>
</section>
<section id="A4.SS2.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">SQL Config</h4>

<div id="A4.SS2.SSS0.Px2.p1" class="ltx_para ltx_noindent">
<pre id="A4.SS2.SSS0.Px2.p1.1" class="ltx_verbatim ltx_font_typewriter">
  "nest": [1,2,3],
  "select_grammar": [],
  "keywords_setting": { "select": true,
  "where": true,
    "group by": true,
    "having": true,
    "order by": true
  },
  "length_setting": {
    "is_available": false,
    "value": [],
    "min": 6,
    "max": 16
  },
  "column_ratio": {
    "is_available": false,
    "value": [],
    "min": 0.1,
    "max": 0.3
  },
  "select_row_ratio":{
    "is_available": false,
    "value": [],
    "min": 0,
    "max": 0.2
  },
  "calculate_times": {
    "is_available": false,
    "value": [0]
  },
  "filter_times": {
    "is_available": false,
    "value": [0]
  },
  "answer_location": {
    "is_available": false,
    "row_value": [],
    "column_value":[0],
    "min": 0,
    "max": 1
  },
  "answer_cells_number": 1,
  "multi_test": false,
  "include": [],
  "exclude": [],
  "n_shot": 5
</pre>
</div>
</section>
</section>
<section id="A4.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">D.3 </span>LLMs Used In This Paper</h3>

<div id="A4.SS3.p1" class="ltx_para ltx_noindent">
<p id="A4.SS3.p1.1" class="ltx_p"><span id="A4.SS3.p1.1.1" class="ltx_text ltx_font_bold">LLMs.</span> LLaMA2 <cite class="ltx_cite ltx_citemacro_citep">(Touvron et al., <a href="#bib.bib39" title="" class="ltx_ref">2023a</a>)</cite>, Qwen <cite class="ltx_cite ltx_citemacro_citep">(Bai et al., <a href="#bib.bib4" title="" class="ltx_ref">2023a</a>)</cite>, InternLM <cite class="ltx_cite ltx_citemacro_citep">(Team, <a href="#bib.bib38" title="" class="ltx_ref">2023</a>)</cite>, Mistral, XGen <cite class="ltx_cite ltx_citemacro_citep">(Nijkamp et al., <a href="#bib.bib28" title="" class="ltx_ref">2023</a>)</cite>, Falcon <cite class="ltx_cite ltx_citemacro_citep">(Penedo et al., <a href="#bib.bib31" title="" class="ltx_ref">2023</a>)</cite>, phi-1_5 <cite class="ltx_cite ltx_citemacro_citep">(Li et al., <a href="#bib.bib22" title="" class="ltx_ref">2023b</a>)</cite>, StableLM <cite class="ltx_cite ltx_citemacro_citep">(Andonian et al., <a href="#bib.bib2" title="" class="ltx_ref">2021</a>)</cite>, Pythia <cite class="ltx_cite ltx_citemacro_citep">(Biderman et al., <a href="#bib.bib7" title="" class="ltx_ref">2023</a>)</cite>,
CodeLlama <cite class="ltx_cite ltx_citemacro_citep">(Rozière et al., <a href="#bib.bib33" title="" class="ltx_ref">2023</a>)</cite>, StarCoder <cite class="ltx_cite ltx_citemacro_citep">(Li et al., <a href="#bib.bib20" title="" class="ltx_ref">2023a</a>)</cite>, CodeGen <cite class="ltx_cite ltx_citemacro_citep">(Nijkamp et al., <a href="#bib.bib27" title="" class="ltx_ref">2022</a>)</cite>.</p>
</div>
<div id="A4.SS3.p2" class="ltx_para ltx_noindent">
<p id="A4.SS3.p2.1" class="ltx_p">We all use the official model weight from the Huggingface Models<span id="footnote3" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">3</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">3</sup><span class="ltx_tag ltx_tag_note">3</span><a target="_blank" href="https://huggingface.co/models" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://huggingface.co/models</a></span></span></span>. Above we used the model’s abbreviation, we list the model’s huggingface official label in Table <a href="#A4.T6" title="Table 6 ‣ D.3 LLMs Used In This Paper ‣ Appendix D Experiments Settings Details ‣ S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>.</p>
</div>
<figure id="A4.T6" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 6: </span>LLMs used in our experiments and their corresponding names in Huggingface Hub.</figcaption>
<table id="A4.T6.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="A4.T6.1.1" class="ltx_tr">
<td id="A4.T6.1.1.1" class="ltx_td ltx_align_center ltx_border_tt">Model</td>
<td id="A4.T6.1.1.2" class="ltx_td ltx_align_center ltx_border_tt">Name</td>
</tr>
<tr id="A4.T6.1.2" class="ltx_tr">
<td id="A4.T6.1.2.1" class="ltx_td ltx_align_center ltx_border_t">Mistral-7B</td>
<td id="A4.T6.1.2.2" class="ltx_td ltx_align_center ltx_border_t">mistralai/Mistral-7B-v0.1</td>
</tr>
<tr id="A4.T6.1.3" class="ltx_tr">
<td id="A4.T6.1.3.1" class="ltx_td ltx_align_center">Llama-2-13B</td>
<td id="A4.T6.1.3.2" class="ltx_td ltx_align_center">meta-llama/Llama-2-13b-hf</td>
</tr>
<tr id="A4.T6.1.4" class="ltx_tr">
<td id="A4.T6.1.4.1" class="ltx_td ltx_align_center">InternLM-20B</td>
<td id="A4.T6.1.4.2" class="ltx_td ltx_align_center">internlm/internlm-20b</td>
</tr>
<tr id="A4.T6.1.5" class="ltx_tr">
<td id="A4.T6.1.5.1" class="ltx_td ltx_align_center">Qwen-14B</td>
<td id="A4.T6.1.5.2" class="ltx_td ltx_align_center">Qwen/Qwen-14B</td>
</tr>
<tr id="A4.T6.1.6" class="ltx_tr">
<td id="A4.T6.1.6.1" class="ltx_td ltx_align_center">Llama-2-7B</td>
<td id="A4.T6.1.6.2" class="ltx_td ltx_align_center">meta-llama/Llama-2-7b-hf</td>
</tr>
<tr id="A4.T6.1.7" class="ltx_tr">
<td id="A4.T6.1.7.1" class="ltx_td ltx_align_center">Qwen-7B</td>
<td id="A4.T6.1.7.2" class="ltx_td ltx_align_center">Qwen/Qwen-7B</td>
</tr>
<tr id="A4.T6.1.8" class="ltx_tr">
<td id="A4.T6.1.8.1" class="ltx_td ltx_align_center">XGen-7B</td>
<td id="A4.T6.1.8.2" class="ltx_td ltx_align_center">Salesforce/xgen-7b-8k-base</td>
</tr>
<tr id="A4.T6.1.9" class="ltx_tr">
<td id="A4.T6.1.9.1" class="ltx_td ltx_align_center">Internlm-7B</td>
<td id="A4.T6.1.9.2" class="ltx_td ltx_align_center">internlm/internlm-7b</td>
</tr>
<tr id="A4.T6.1.10" class="ltx_tr">
<td id="A4.T6.1.10.1" class="ltx_td ltx_align_center">Phi-1_5</td>
<td id="A4.T6.1.10.2" class="ltx_td ltx_align_center">microsoft/phi-1_5</td>
</tr>
<tr id="A4.T6.1.11" class="ltx_tr">
<td id="A4.T6.1.11.1" class="ltx_td ltx_align_center">Stablelm-7B</td>
<td id="A4.T6.1.11.2" class="ltx_td ltx_align_center">stabilityai/stablelm-base-alpha-7b</td>
</tr>
<tr id="A4.T6.1.12" class="ltx_tr">
<td id="A4.T6.1.12.1" class="ltx_td ltx_align_center">Stablelm-3B</td>
<td id="A4.T6.1.12.2" class="ltx_td ltx_align_center">stabilityai/stablelm-base-alpha-3b</td>
</tr>
<tr id="A4.T6.1.13" class="ltx_tr">
<td id="A4.T6.1.13.1" class="ltx_td ltx_align_center">Pythia-12B</td>
<td id="A4.T6.1.13.2" class="ltx_td ltx_align_center">EleutherAI/pythia-12b</td>
</tr>
<tr id="A4.T6.1.14" class="ltx_tr">
<td id="A4.T6.1.14.1" class="ltx_td ltx_align_center">Pythia-6.9B</td>
<td id="A4.T6.1.14.2" class="ltx_td ltx_align_center">EleutherAI/pythia-6.9b</td>
</tr>
<tr id="A4.T6.1.15" class="ltx_tr">
<td id="A4.T6.1.15.1" class="ltx_td ltx_align_center">Pythia-2.8B</td>
<td id="A4.T6.1.15.2" class="ltx_td ltx_align_center">EleutherAI/pythia-2.8b</td>
</tr>
<tr id="A4.T6.1.16" class="ltx_tr">
<td id="A4.T6.1.16.1" class="ltx_td ltx_align_center">Pythia-1B</td>
<td id="A4.T6.1.16.2" class="ltx_td ltx_align_center">EleutherAI/pythia-1b</td>
</tr>
<tr id="A4.T6.1.17" class="ltx_tr">
<td id="A4.T6.1.17.1" class="ltx_td ltx_align_center">Llama-2-70B</td>
<td id="A4.T6.1.17.2" class="ltx_td ltx_align_center">meta-llama/Llama-2-70b-hf</td>
</tr>
<tr id="A4.T6.1.18" class="ltx_tr">
<td id="A4.T6.1.18.1" class="ltx_td ltx_align_center">CodeLlama-34B</td>
<td id="A4.T6.1.18.2" class="ltx_td ltx_align_center">codellama/CodeLlama-34b-hf</td>
</tr>
<tr id="A4.T6.1.19" class="ltx_tr">
<td id="A4.T6.1.19.1" class="ltx_td ltx_align_center">CodeLlama-13B</td>
<td id="A4.T6.1.19.2" class="ltx_td ltx_align_center">codellama/CodeLlama-13b-hf</td>
</tr>
<tr id="A4.T6.1.20" class="ltx_tr">
<td id="A4.T6.1.20.1" class="ltx_td ltx_align_center">CodeLlama-7B</td>
<td id="A4.T6.1.20.2" class="ltx_td ltx_align_center">codellama/CodeLlama-7b-hf</td>
</tr>
<tr id="A4.T6.1.21" class="ltx_tr">
<td id="A4.T6.1.21.1" class="ltx_td ltx_align_center">StarCoder-15B</td>
<td id="A4.T6.1.21.2" class="ltx_td ltx_align_center">bigcode/starcoderbase</td>
</tr>
<tr id="A4.T6.1.22" class="ltx_tr">
<td id="A4.T6.1.22.1" class="ltx_td ltx_align_center">StarCoder-7B</td>
<td id="A4.T6.1.22.2" class="ltx_td ltx_align_center">bigcode/starcoderbase-7b</td>
</tr>
<tr id="A4.T6.1.23" class="ltx_tr">
<td id="A4.T6.1.23.1" class="ltx_td ltx_align_center">StarCoder-3B</td>
<td id="A4.T6.1.23.2" class="ltx_td ltx_align_center">bigcode/starcoderbase-3b</td>
</tr>
<tr id="A4.T6.1.24" class="ltx_tr">
<td id="A4.T6.1.24.1" class="ltx_td ltx_align_center">StarCoder-1B</td>
<td id="A4.T6.1.24.2" class="ltx_td ltx_align_center">bigcode/starcoderbase-1b</td>
</tr>
<tr id="A4.T6.1.25" class="ltx_tr">
<td id="A4.T6.1.25.1" class="ltx_td ltx_align_center">CodeGen-15B</td>
<td id="A4.T6.1.25.2" class="ltx_td ltx_align_center">Salesforce/codegen-16B-multi</td>
</tr>
<tr id="A4.T6.1.26" class="ltx_tr">
<td id="A4.T6.1.26.1" class="ltx_td ltx_align_center">CodeGen-6B</td>
<td id="A4.T6.1.26.2" class="ltx_td ltx_align_center">Salesforce/codegen-6B-multi</td>
</tr>
<tr id="A4.T6.1.27" class="ltx_tr">
<td id="A4.T6.1.27.1" class="ltx_td ltx_align_center">CodeGen-2B</td>
<td id="A4.T6.1.27.2" class="ltx_td ltx_align_center">Salesforce/codegen-2B-multi</td>
</tr>
<tr id="A4.T6.1.28" class="ltx_tr">
<td id="A4.T6.1.28.1" class="ltx_td ltx_align_center">Yarn-LLaMA2-13B</td>
<td id="A4.T6.1.28.2" class="ltx_td ltx_align_center">NousResearch/Yarn-Llama-2-7b-64k</td>
</tr>
<tr id="A4.T6.1.29" class="ltx_tr">
<td id="A4.T6.1.29.1" class="ltx_td ltx_align_center">LongChat-13B</td>
<td id="A4.T6.1.29.2" class="ltx_td ltx_align_center">lmsys/longchat-7b-16k</td>
</tr>
<tr id="A4.T6.1.30" class="ltx_tr">
<td id="A4.T6.1.30.1" class="ltx_td ltx_align_center ltx_border_bb">RWKV-Raven-14B</td>
<td id="A4.T6.1.30.2" class="ltx_td ltx_align_center ltx_border_bb">lmsys/longchat-7b-16k</td>
</tr>
</table>
</figure>
</section>
<section id="A4.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">D.4 </span>Markdown vs. Flatten Setting Experiments</h3>

<div id="A4.SS4.p1" class="ltx_para ltx_noindent">
<pre id="A4.SS4.p1.1" class="ltx_verbatim ltx_font_typewriter">
"0": Size: 100 * 5, Template: Easy, Model: GPT-3.5
"1": Size: 50 * 5,  Template: Easy, Model: GPT-3.5
"2": Size: 20 * 6,  Template: Count, Model: GPT-3.5
"3": Size: 40 * 10, Template: Where Condition Text, Model: GPT-3.5
"4": Size: 10 * 20, Template: Where Condition Text, Model: GPT-3.5
"5": Size: 10 * 15, Template: Where Condition Text, Model: GPT-3.5
"6": Size: 50 * 5,  Template: Easy, Model: Llama-2-13B
"7": Size: 100 * 5, Template: Easy, Model: Yarn-Llama-2-13B
"8": Size: 50 * 5,  Template: Easy, Model: Yarn-Llama-2-13B
"9": Size: 25 * 7, Template: General, Model: Llama-2-13B
"10": Size: (15~40) * (6~9), Template: General, Model: Llama-2-13B
"11": Size: (15~40) * (6~9), Template: General, Model: Llama-2-13B
"12": Size: (15~40) * (6~9), Template: Easy, Model: Llama-2-13B
"13": Size: (15~40) * (6~9), Template: Easy, Model: Llama-2-13B
"14": Size: (15~40) * (6~9), Template: Easy, Model: Llama-2-13B
</pre>
</div>
</section>
</section>
<div class="ltx_pagination ltx_role_newpage"></div>
</article>
</div>
<div class="ar5iv-footer"><a href="/html/2310.15146" class="ar5iv-nav-button ar5iv-nav-button-prev">◄</a>
    <a class="ar5iv-home-button" href="/"><img height="40" alt="ar5iv homepage" src="/assets/ar5iv.png"></a>
    <a href="/feeling_lucky" class="ar5iv-text-button">Feeling<br>lucky?</a>
    <a href="/log/2310.15147" class="ar5iv-text-button ar5iv-severity-error">Conversion<br>report</a>
    <a class="ar5iv-text-button" target="_blank" href="https://github.com/dginev/ar5iv/issues/new?template=improve-article--arxiv-id-.md&title=Improve+article+2310.15147">Report<br>an issue</a>
    <a href="https://arxiv.org/abs/2310.15147" class="ar5iv-text-button arxiv-ui-theme">View&nbsp;original<br>on&nbsp;arXiv</a><a href="/html/2310.15148" class="ar5iv-nav-button ar5iv-nav-button-next">►</a>
</div><footer class="ltx_page_footer">
<a class="ar5iv-toggle-color-scheme" href="javascript:toggleColorScheme()" title="Toggle ar5iv color scheme"><span class="color-scheme-icon"></span></a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/license" target="_blank">Copyright</a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/policies/privacy_policy" target="_blank">Privacy Policy</a>

<div class="ltx_page_logo">Generated  on Tue Feb 27 23:36:40 2024 by <a target="_blank" href="http://dlmf.nist.gov/LaTeXML/" class="ltx_LaTeXML_logo"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg==" alt="Mascot Sammy"></a>
</div></footer>
</div>

    <script>
      var canMathML = typeof(MathMLElement) == "function";
      if (!canMathML) {
        var body = document.querySelector("body");
        body.firstElementChild.setAttribute('style', 'opacity: 0;');
        var loading = document.createElement("div");
        loading.setAttribute("id", "mathjax-loading-spinner");
        var message = document.createElement("div");
        message.setAttribute("id", "mathjax-loading-message");
        message.innerText = "Typesetting Equations...";
        body.prepend(loading);
        body.prepend(message);

        var el = document.createElement("script");
        el.src = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js";
        document.querySelector("head").appendChild(el);

        window.MathJax = {
          startup: {
            pageReady: () => {
              return MathJax.startup.defaultPageReady().then(() => {
                body.removeChild(loading);
                body.removeChild(message);
                body.firstElementChild.removeAttribute('style');
              }); } } };
      }
    </script>
    <script>
    // Auxiliary function, building the preview feature when
    // an inline citation is clicked
    function clicked_cite(e) {
      e.preventDefault();
      let cite = this.closest('.ltx_cite');
      let next = cite.nextSibling;
      if (next && next.nodeType == Node.ELEMENT_NODE && next.getAttribute('class') == "ar5iv-bibitem-preview") {
        next.remove();
        return; }
      // Before adding a preview modal,
      // cleanup older previews, in case they're still open
      document.querySelectorAll('span.ar5iv-bibitem-preview').forEach(function(node) {
        node.remove();
      })

      // Create the preview
      preview = document.createElement('span');
      preview.setAttribute('class','ar5iv-bibitem-preview');
      let target = document.getElementById(this.getAttribute('href').slice(1));
      target.childNodes.forEach(function (child) {
        preview.append(child.cloneNode(true));
      });
      let close_x = document.createElement('button');
      close_x.setAttribute("aria-label","Close modal for bibliography item preview");
      close_x.textContent = "×";
      close_x.setAttribute('class', 'ar5iv-button-close-preview');
      close_x.setAttribute('onclick','this.parentNode.remove()');
      preview.append(close_x);
      preview.querySelectorAll('.ltx_tag_bibitem').forEach(function(node) {
        node.remove();
      });
      cite.parentNode.insertBefore(preview, cite.nextSibling);
      return;
    }
    // Global Document initialization:
    // - assign the preview feature to all inline citation links
    document.querySelectorAll(".ltx_cite .ltx_ref").forEach(function (link) {
      link.addEventListener("click", clicked_cite);
    });
    </script>
    </body>
</html>

2310.15147
The rapid development of Large Language Models (LLMs)  has led to great strides in model capabilities like reasoning and long-context understanding.
However, as LLMs are able to process longer contexts, it becomes more challenging to evaluate whether they have acquired certain capabilities, since the length of text (e.g., 100K tokens)  they can process far exceeds what humans can reliably assess in a reasonable duration.
In this paper, we propose using complex synthetic tasks as a proxy evaluation method, and presentS3Eval, aSynthetic,Scalable,Systematic evaluation suite for LLMs evaluation.
As a synthetic benchmark,S3Evalenables the creation of any number of evaluation examples that are theoretically invisible to LLMs, mitigating the test set contamination issue.
The synthetic nature ofS3Evalprovides users full control over the dataset, allowing them to systematically probe LLM capabilities by scaling text length and varying task difficulty across diverse scenarios.
The strong correlation betweenS3Evalperformance and scores of real-world benchmarks like Big-Bench Hard (BBH)  demonstrates the soundness of usingS3Evalfor evaluation of LLMs.
The in-depth analysis also uncover additional insights, including performance drop when the answer is sparsely distributed or located in the middle context, as well as some counter-intuitive trends of model performance. Our code is available athttps://github.com/lfy79001/SQLEval.
abstract
S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models
Abstract
2310.15147
1IntroductionLarge Language Models (LLMs)  have greatly propelled significant advancements in Natural Language Processing (NLP) , such as OpenAI GPT, Llama, StarCoder, and others. These models perform well in many NLP tasks and claim to have made progress in advanced capabilities such as reasoning, long-context understanding, and so on. However, existing benchmarksoften fail when it comes to evaluating extremely long-context LLMs or analysing the controllable characteristics and limitations of LLMs.For long-context understanding, previous work has often evaluated LLMs using the scope of language modeling metrics (i.e., perplexity) or the performance on simple artificial tasks.
However, these evaluation tasks tend to lack complexity and are narrowly focused on simple comprehension, which is misaligned with the sophistication required for real-world downstream applications.
While recent work has made great progress on building evaluation benchmarks at longer context lengths with real-world use cases (e.g, question answering) , these manually annotated datasets often lack the scale and diversity to thoroughly assess performance on extended context lengths.
For example, existing benchmarks struggle to effectively evaluate LLMs that claim an ability to process contexts up to 100K tokens, due to the limited capacity of human annotation for very long text.
Developing more scalable and diverse evaluation datasets, potentially leveraging automated supervision, remains an open challenge.For reasoning analysis, conducting both qualitative and quantitative analysis of answers and reasoning processes provides important insights.
However, existing benchmarks lack the ability to precisely control the distribution of the dataset, limiting their utility for in-depth research analysis.
In other words, the nature of these benchmarks makes it challenging for developers to identify the specific weaknesses of their LLMs.
More configurable and granular benchmarks are needed to enable detailed analysis of model performance.
In addition, these benchmarks often draw their evaluation data from NLP tasks that have been extensively studied and are likely to be used in the training corpus of LLMs.
The potential data leakage makes the evaluation less convincing.Figure 1:The illustration demonstrates theS3Evalpipeline, where the capabilities of LLMs are assessed by evaluating their ability to execute SQL queries over randomly generated tables.In this paper, we propose a new evaluation suite calledS3Eval, which addresses the aforementioned issues by using a complex synthetic task - SQL execution - as a proxy for the performance of LLMs on realistic reasoning tasks.
As shown in Figure1, inspired by the work ofTaPEx,S3Evalis based on the SQL execution task.
Specially, given a randomly generated table and a random SQL query,S3Evalevaluates whether LLMs can return the correct execution results.S3Evalhas three notable characteristics: (1)  It is synthetic, with no table or SQL query present in the LLM training corpus. The tasks use complex, grammatically correct SQL syntax, making them very challenging.
(2)  It is scalable, allowing users to customize the benchmark to any length and difficulty.
(3)  It is systematic, containing diverse reasoning types and operations. This enables comprehensive evaluation of LLM capabilities.With these powerful features, developers can extend the context to really long lengths and generate meaningful SQL statements usingS3Eval.
As demonstrated in Table1, developers can modify the configuration settings to conduct focused studies on particular reasoning abilities (e.g., aggregation)  of LLMs.
We conducted comprehensive multi-perspective experiments on several popular LLMs usingS3Eval.
Experimental results demonstrated that the performance of LLMs onS3Evalaligns closely with their performance on mainstream LLM benchmarks.
While LLMs have shown impressive capabilities, our work reveals limitations in their ability to leverage long contexts, since we observe performance degradation of almost all LLMs in long-context settings.
By carefully studying experimental results, we can work to pinpoint situations where LLMs tend to fail and summarize valuable insights. For example, LLMs often encounter challenges when the answer lies in the middle of the context, similar to findings from, or when the answers are sparsely distributed across the input. We believe that these observations fromS3Evalprovide valuable guidance for the development of LLMs and dynamic benchmarks.
introduction
S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models
1Introduction
2310.15147
2Synthetic Suite: Alignment with Realistic BenchmarkIn this section, we describe the details of synthesizing the evaluation data (Section2.1)  and verify the alignment between our synthetic suiteS3Evaland real-world benchmark results.Table 1:OurS3Evalmethod allows users to customize configuration settings and provides descriptions for each parameter that can be adjusted. More configurations can be found in AppendixD.1.ConfigurationDescriptionTable Control# of RowsThe number of rows in the generated tables# of ColumnsThe number of columns in the generated tablesHeader Type RatioThe proportion of table column types that are TEXT, INT, DATECell UniquenessThe proportion of duplicate cells in each columnString / Int LengthThe string length or numeric range of cell valuesInstructionControlSQL KeywordsSELECT, WHERE, GROUP BY, HAVING, ORDER BYSQL LengthThe number of tokens after SQL split by spaceColumn CoverageThe ratio of columns involved in SQL execution to total columns.Row CoverageThe ratio of rows involved in SQL execution to total rowsCalculate TimesThe number of SQL numerical calculations.Filter TimesThe number of SQL filtering operations.AggregatorCOUNT, MAX, MIN, SUM, AVGFilter Operator>, <, =, IN, LIKEOutput ControlAnswer LocationThe location of SQL answers in the input table# of Answer CellsThe number of selected cells in the answerAnswer LengthThe total number of tokens in the answer2.1Suite ConstructionTask FormulationFollowing previous work, each example inS3Evalgenerally contain an SQL query and a (semi-) structured tableas the input.
Each tableconsists ofrows, in which each rowcontainscell values. Each cellcorresponds to a table header. Each SQL query consists oftokens as. Each tokenoriginates from SQL keywords, table schema, or table cells. Each multi-step instruction is transformed from SQL query. The task prompts LLMto obtain the execution resultof the SQL on the table. Our main focus is on analyzing the accuracy of LLM in executing SQL queries.Random Table GenerationAll tables inS3Evalare randomly generated and do not contain any real data or overlap with existing public tables. The tables haverows andcolumns, with adjustable parametersand. The column headers are sampled from English nouns, falling into three types: TEXT, INT, and DATE. INT columns contain random integers from 1 to 1000, which is an adjustable range. DATE columns have values in year-month-day format. TEXT columns have random strings of length 5 to 12 characters, which is also adjustable.
To simulate real-world data where the same value may recur in a column frequently, the data generator includes a parameter to set the probability of duplicating values within a specific column.Random SQL GenerationThe SQL language includes a variety of statements to query and manage data.S3Evaluse context free grammar to generate a specific number of examples with controllable attributes.
As Table1shows, theS3Evaltool allows configuring several parameters of generated SQL statements, including nesting depth, keywords used, length, coverage of SQL features, computational complexity, and more.
For example,calculate timescan be modified to control the complexity of numerical reasoning for each dataset. Except these configures, users can also manually write the specified SQL template to generate fine-grained evaluation data (AppendixC.2) .Evaluation MethodsS3Evalincludes both zero-shot and few-shot prompting methods. For each few-shot setting, all examples share one table. N-shot is formalized as. For the input format of table, we designed several alternative ways, including markdown, flatten, tapex-style, etc.To evaluate the performance of LLMs, we use Exact Match (EM)  as the evaluation metric.
Details are shown in AppendixC.3.Evaluation SettingsModels are evaluated on two settings:EasyandGeneral.Easyis the simplest data thatS3Evalcan generate and is used to evaluate LLM’s ability to understand the most basic instructions. It contains only one template, “SELECT <col1> WHERE <col2> <op> <value>”.Generalis a more difficult setting, containing extensive SQL syntax, and its generating setting is described in AppendixD.2.Generalrandomly generates SQL queries of varying difficulty from an extensive grammar and provides a comprehensive evaluation of LLM performance. All experiments were run fortimes, using 1000 randomly generated queries per trial, with tables of 15 rows and 8 columns and an average of 1200 tokens per input. Details on the LLMs are provided in AppendixD.3.
ConsideringSQL executionis a difficult task, some models may have a poor understanding of symbolic language, which makes it difficult to execute SQL, so we propose an alternative taskSQL multi-step taskto remove this potential bias. Specifically, it converts an SQL query into a multi-step table operation instruction as shown in AppendixC.5. SQL has a fixed execution flow for the query statement: FROMONJOINWHEREGROUP BYHAVINGSELECTORDER BYLIMIT. This is not consistent with the order in which it is written. With this processing, it can also generate chain-of-thought prompting data.2.2Alignment on Scaling LawPrevious workshows a positive correlation between the cross-entropy loss of LLMs and the amount of computing resources used for training, as described by the empirical scaling law.
To verify whether the scaling law holds for ourS3Eval, we employ a set of checkpoints of Pythia-12Bthat are open-sourced at different training steps, corresponding to different amounts of compute.
Using both General and Easy settings, we observe a consistent pattern as illustrated in Figure3: the scores show a smooth progression of improvement that aligns with the scaling law with increasing the training steps.
The steady, incremental performance gains over time, lacking any spikes, demonstrateS3Eval’s reliability as a evaluation suite.
Overall, these experimental results confirm the scaling law’s accuracy in forecasting model gains during training across diverse evaluation settings.Figure 2:The performance of Pythia-12B on the Easy and General settings ofS3Evalwas evaluated across different training steps.Figure 3:The performance of different LLMs onS3Evaland WikiTableQuestions.2.3Alignment on Benchmark PerformanceIn the above, we validated that the LLMs also exhibits the scaling law observed in NL on theS3Evalsuite. A natural question that arises is whether its performance onS3Evalis correlated with the performance on real-world, NL benchmarks.
To examine the hypothesis, we first compare the performance of different LLMs onS3Evaland on WikiTableQuestions, a table question answering dataset consisting of questions and answers.
It is worth noting that to align the difficulty, we use the SQL queries from WikiTableQuestionsas ourS3Evalevaluation set.To systematically compare the performance, following previous work,
we consider two correlation measures: the Pearson correlation coefficient () , which evaluates the linear relationship between model scores on the two benchmarks, and the Kendall rank correlation coefficient () , which assesses whether the relative ranking of models is consistent across the benchmarks.
The specific steps for computing the correlation coefficients are as follows:(1) (2) whereandare the evaluation results of two different methods for the same model, respectively.
The strong correlation between LLMs’ performance on the SQL execution task and the table question answering task, as evidenced by the high(e.g., 99.1)  and high(e.g., 93.6)  in Figure3.(a) Performance of large language models(b) Performance of code large language modelsFigure 4:Each point in the scatterplot represents the LLM performance on the benchmarks corresponding to the horizontal and vertical coordinates. The black straight line is the trend line. The larger the values ofand, the higher the correlation between the two benchmarks. We consider> 0.8 to be high concurrence.AlthoughS3Evalhas shown significant correlation with WikiTableQuestions, the fact that they are both tasks on tables may cause one to question whetherS3Evalcan serve as a proxy task to evaluate LLMs’ capabilities on generic reasoning tasks.
Therefore, we also compare the performance onS3Evalwith the results of generic popular benchmarks like BBHand HumanEval.
The results depicted in Figure4(a) demonstrate a strong correlation between LLM performance onS3Evaland the BBH benchmark, with BBH performance obtained from the OpenCompass platform using few-shot chain-of-thought prompting.
Similarly, Figure4(b) illustrates the alignment betweenS3Evalperformance and pass@1 scores on HumanEvalfor code LLMs.
The results demonstrate thatS3Evalserves as a robust proxy task for assessing the reasoning capabilities of LLMs on realistic benchmarks.
Concrete experimental results are provided in Table3.Figure 5:SQL execution training experiments onS3Eval.Figure 6:Experiment results of different LLMs on different context lengths.
body
S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models
2Synthetic Suite: Alignment with Realistic Benchmark
2310.15147
3Scalable Suite: Unlimited Evaluation ResourcesS3Evalprovides a unique capability to generate infinite number of examples (Section3.1)  with infinite length (Section3.2) .3.1Scalable Number of Evaluation ExamplesThe strength ofS3Evalis its ability to generate unlimited number of examples for evaluation. This stems from two key design choices inS3Eval: (1)  the synthetic table size can be scaled to different number of rows and columns, and (2)  the table cells are synthesized from randomly generated strings. Combined with the provided large library of SQL query templates, these features enable the creation of a near-infinite set of unique evaluation examples.
This kind of capacity enables the continuous creation of novel examples unseen during training, which helps safeguard test data integrity by preventing leakage of the evaluation set into the training corpus.However, the absence of data leakage does not necessarily mean thatS3Eval’s performance always represents the model’s out-of-distribution generalization ability. It is because the model may perform well onS3Evalvia domain-specific training on the SQL execution task, rather than acquiring more general abilities.
To investigate whether LLMs can “hack”S3Evalvia domain-specific training, we fine-tuned StarCoder-1B, which is not able to solve SQL execution tasks, on a randomly generated dataset of one million examples.
The performance of the fine-tuned StarCoder-1B is illustrated in Figure6, where it is evaluated on three types of test datasets:Seen Table(same tables as training) ,Unseen Table(new tables in same format as training tables) , andUnseen Templates(new SQL query templates) .
For the unseen table setting, we explore different table shapes, wheremeans the table consists ofrows andcolumns.The experimental results demonstrate that for Unseen Tables with different shapes, regardless of their size, the performance of the fine-tuned StarCoder experiences a substantial decline compared to Seen Tables. Likewise, when faced with Unseen Templates, the performance of the fine-tuned StarCoder exhibits a significant drop.
The results indicate that even if LLMs have been heavily trained on SQL execution tasks, their out-of-distribution performance can still be accurately evaluated by using novel SQL templates. These new SQL templates can be easily generated thanks to the vast grammar of SQL queries. Additionally, evaluating LLMs on larger tables that they were not trained on can also reveal part of their out-of-distribution capabilities.3.2Scalable Length of Evaluation ExamplesOne advantage ofS3Evalis its scalability and adjustable context length per example.
The flexibility allowsS3Evalto rigorously evaluate LLMs that claim capability with long contexts.
To clearly expose limitations of current LLMs, we intentionally chose theEasysetting inS3Evalto evaluate their performance. Specifically, we establish table configurations with approximately 2K, 4K, 8K, and 16K tokens, by using different numbers of rows and fixing the number of columns.
We generate a dataset consisting of 500 samples for each evaluation setting.
The experimental results on up to 16K context length are plotted in Figure6.
As observed, the performance of almost all LLMs, significantly decreases as the context length increases.
Of all the models, Claude-1.3-100K is the only one that maintains a relatively strong performance trend.
Detailed results can be found in AppendixA.4.Given the limitation of existing LLMs on long-context tasks, we are curious about the bottleneck of them.
By usingS3Eval, we can systematically investigate the long-context modeling capabilities of LLMs by controlling the distribution of answers in the evaluation suite.
Specifically, we use theEasysetting and fix the number of answers to four cells (i.e., the result of the SQL execution is always spanning four cells) .
As illustrated in Figure7, we introduce two distribution patterns,DenseandSparse111Examples of these two patterns can be found in AppendixC.1.to probe the limitations of current LLMs.
The dense mode only requires the model to understand the local context, whereas the sparse mode requires the model to have a broader, global understanding of the context across multiple blocks.
The sparse mode intuitively poses more challenges and demands more complex reasoning across a broader scope of the provided context.
We conduct experiments on ChatGPT and Yarn-llama2-13B.
The experimental results indicate that both models perform significantly better in dense mode compared to sparse mode, as shown in Figure7.
This indicates that LLMs struggle to retrieve information over long sequences, even though their pre-training included lengthy contexts.
This may be caused by the fact that the training data does not contain sufficient examples of long-distance dependencies for the model to learn effectively.
Furthermore, the steep drop in performance from 4K to 8K tokens for both ChatGPT and Yarn-Llama2 in dense mode indicates that current length extension techniques may not be as effective as hoped.
In summary, we believe thatS3Evalprovides a valuable framework for evaluating long-context language models, as it allows testing models on dialogues of arbitrary length. This establishes a solid foundation for advancing research on large language models that can leverage long-term context.Figure 7:Experiment results of ChatGPT and Yarn-Llama 2 onDenseandSparseSettings.Densemeans that the answer cells (i.e.,Liu,Chen,Lei,Wang)  lie in adjacent rows, andSparsemeans that the answer cells are separated. The model performs better on local queries which only involves adjacent cells.
body
S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models
3Scalable Suite: Unlimited Evaluation Resources
2310.15147
4Systematic Suite: Controllable AnalysisS3Evalprovides a comprehensive framework that empowers developers to synthesize diverse evaluation examples for systematically assessing LLMs from multiple perspectives.
In this section, inspired by the work of lost in the middle, we first analyze the impact of answer position on performance (Section4.1) .
Then we evaluate LLMs from different viewpoints, and we have conducted some initial explorations on the reasoning types analysis (Section4.2) .
Last, we provide some insights by analyzing LLMs on three selected SQL templates (Section4.3) .
These experiments reveal counter-intuitive performance trends and new discoveries that may inspire further research and extension of the work.4.1Answer Position AnalysisWe investigate the influence of the answer’s position on the performance of LLMs, which is generally considered important.
Unlike standard NLP benchmarks where it is difficult to control the position of the answer,S3Evalallows for fine-grained control of answer position at the token level.
To mitigate the influence of long contexts, we only analyzed answers that fell within a limited context window (i.e., less than 4K tokens) .Echoing the findings of, “lost in the middle”, our results in Figure8demonstrate that both ChatGPT and CodeLlama achieve higher performance when the answer is located at the beginning or end of the context, compared to when it appears in the middle.
In addition, we found a periodic fluctuation trend in the performance of both models as the position of the answer shifts within the context.
For example, the performance of ChatGPT increases from 0 to around 200, then starts to decrease from around 200 to 500.
This wave-like pattern in performance appears to correlate with the position embedding approach used by LLMs.In contrast to previous studies that used long-context question answering tasksfor analysis and are thus limited to controlling answer positions at the paragraph level,S3Evalprovides a more precise approach by focusing on token level. This key difference enablesS3Evalto offer fine-grained control and promote the exploration of relevant phenomena.(a) ChatGPT(b) CodeLlamaFigure 8:The relationship between LLMs performance and the position of the answer token. The purple data points in figure represent the performance under the ‘sliding window=50’, while the orange curve represents the trend line. ‘sliding window’ means that each data point represents the performance of the answer position within its window range. This is advantageous for smoothing.Table 2:Reasoning types experiments examples of different LLMs.OperatorExample SQLChatGPTClaudeMistralCodeLlamaFilterSELECT lyonnais FROM table WHERE farmer = ’mijl’ AND lashing >28879.679.264.872.8ArithmeticSELECT synset + refuge FROM table WHERE blender = ’owxdbzjg’67.259.45.410.6ComparativeSELECT upsetter < jollity FROM table WHERE kelp = 15045.246.444.846.6AggregateSELECT MIN(skeptic)  FROM table38.439.428.433.8GroupSELECT lats FROM table GROUP BY shastan HAVING sum ( logbook )  = 5638.128.231.037.8SuperlativeSELECT severity FROM table ORDER BY bierce DESC Limit 124.841.419.228.34.2Reasoning Type AnalysisS3Evalenables the creation of multiple templates to generate different SQL statements, with each statement representing a distinct reasoning type.
We selected six common reasoning types to investigate the reasoning capabilities of LLMs and examined four different LLMs: ChatGPT, Claude, Mistral-7B, and CodeLlama-34B.
Following, the six reasoning types222Detailed templates for each type can be found in AppendixC.2.we considered areFilter,Aggregate,Arithmetic,Superlative,Comparative, andGroup. The example SQL and the experimental results of different LLMs are presented in Table2.
The expressive power of SQL queries enablesS3Evalto be used for evaluating diverse scenarios such as numerical reasoning, multi-hop reasoning, complex code understanding, and multi-turn interaction with intermediate execution results.4.3Template Controlled AnalysisEach data template inS3Evalincludes corresponding reasoning types, and thus it provides fine-grained control over the evaluation examples.
To stimulate new insights and uncover counter-intuitive performance phenomena of LLMs, we present several controlled analysis examples using simple templates as a starting point.Template1: SELECT[text_col1]FROM table WHERE ([text_col2]=[text2]) Figure 9:ChatGPT performance with different locations of [text_col1] and [text_col2]. The performance improves when the example has the location of [text_col1] before [text_col2].We first explore the relationship between the model performance and the locations of [text_col1] and [text_col2].
To begin with, we generated a set oftables, each comprisingdistinct columns.
We created 400 unique combinations by pairing each value in text_col1 with each value in text_col2. For each of the 400 pairs, we generated 40 evaluation examples, resulting in a total of 16,000 evaluation examples.
After SQL execution experiments, we calculated the scores of each pair and constructed a heatmap, which is illustrated in Figure9. The heatmap indicates that the performance is overall better when [text_col1] is the previous column. And the model performance is also better when the [text_col1] column is before [text_col2] column.
It indicates that the model tends to focus on the beginning of a specific paragraph. Moreover, in multi-hop reasoning, LLMs excel at hopping to the context preceding a intermediate hop, but struggles when it comes to searching backward.Template2: SELECT[text_col1]FROM table WHERE ([text_col2]=[text2]) NWe then investigate the impact of the number of WHERE conditions on LLM performance. Intuitively, more conditions should make it harder for LLM to execute SQL since the instruction becomes more complex.
However, the experimental results contradict this intuition, as shown in blue in Figure11. We speculate that this counter-intuitive result stems from how LLMs actually reason: by looking up string co-occurrences rather than logically considering all conditions.Figure 10:Trend of ChatGPT performance with where condition number using Template2.Figure 11:Trend of ChatGPT performance with the COUNT value in Template3. Only when the COUNT value is the largest or smallest, the model have good performance.Template3: SELECT COUNT([text_col])  FROM table WHERE[text_col]=[text].We analyze the counting ability of LLMs, which is an important numerical reasoning capability.
To avoid potential symbolic effects of SQLs, we also use the instruction style (Section2.1)  to prompt the model (e.g. Please count the number of “[text_col]is[text]”) . As shown in Figure11, whether it is zero-shot or few-shot, SQL style or instruction style, the performance of LLMs is best when the COUNT value is the smallest or the largest. When the COUNT value is in the middle, the performance of the model is almost zero.In the future, developers can employ theS3Evalsuite to analyze the performance of LLMs with various complex SQL queries and discover new insights. They can also investigate more on the multi-step instruction prompting (SectionC.5)  and chain-of-thought prompting (SectionC.6)  to better understand LLMs.
body
S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models
4Systematic Suite: Controllable Analysis
2310.15147
5Related WorkLanguage Model EvaluationEvaluating large language models (LLMs)  has garnered significant interest in the NLP community. This allows us to gain a deeper understanding of the specific capabilities and limitations of LLMs while guiding further research.
Researchers proposed MMLUto measure the knowledge acquired by a language model during pre-training. In recent years, with the development of LLMs, a series of general evaluation benchmarks have emerged. For instance, BBHand AGIEvalassess the reasoning ablitities. GSM8Kevalutes the math reasoning, HumanEval, MBPPand DS-1000measure code capalities. Additionally, MINTand InterCodehave been proposed to evaluate both code and multi-turn capability of LLMs. Our work aims to provide an evaluation suite for measuring reasoning ability.
Many previous works on long-text modeling rely on the perplexityor performance on simple artificial tasks. Concurrently, ZeroSCROLLS, L-Evaland LongBenchare proposed as evaluation benchmarks for long-text modeling. However, these benchmarks are built from existing public datasets and have fixed evaluation types. In contrast,S3Evalcan effectively assess comprehension of infinitely long-context. Furthermore,S3Evalallows customization of settings to generate evaluation data that meets specific needs, enabling effective evaluation of model deficiencies and discovery of new insights into LLMs.Synthetic DataInvestigating the use of synthetic data to explore NLP problems is worthwhile. Previous research has explored the effectiveness of synthetic tasks in various ways, such as table pretraining, applying synchronous context-free grammars in text-to-SQL parsing, and utilizing synthetic data for numerical reasoning. In our work, we utilize the controllability of SQL synthesis to allow users to customize the performance analysis of LLM. This approach represents the first scalable and infinite LLM benchmark. DyValis relevant to our work as it harnesses the structural advantage of directed acyclic graphs to dynamically generate evaluation samples with controllable complexities.
body
S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models
5Related Work
2310.15147
6ConclusionIn this paper, we have introducedS3Eval, a novel synthetic evaluation suite for LLMs using SQL execution.S3Evalrepresents a scalable and systematic approach to evaluate LLMs on a dynamic task.
Our experiments demonstrate strong alignment betweenS3Evaland traditional evaluation benchmarks.
The key innovations ofS3Evalare its flexibility, allowing unlimited context length and unlimited evaluation examples, and its fine-grained, systematic nature which enables detailed analysis of model capabilities and flaws.
We believeS3Evalcan serve as a valuable benchmark for LLM development and shed light on the dynamic synthetic benchmark construction.
conclusion
S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models
6Conclusion