<!DOCTYPE html><html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>[2312.07104] Efficiently Programming Large Language Models using SGLang</title><meta property="og:description" content="Large language models (LLMs) are increasingly used for complex tasks requiring multiple chained generation calls, advanced prompting techniques, control flow, and interaction with external environments. However, effici…">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="Efficiently Programming Large Language Models using SGLang">
<meta name="twitter:image:src" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta name="twitter:image:alt" content="ar5iv logo">
<meta property="og:title" content="Efficiently Programming Large Language Models using SGLang">
<meta property="og:site_name" content="ar5iv">
<meta property="og:image" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta property="og:type" content="article">
<meta property="og:url" content="https://ar5iv.labs.arxiv.org/html/2312.07104">

<!--Generated on Tue Feb 27 11:25:01 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.-->
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

<script>
  function detectColorScheme(){
    var theme="light";
    var current_theme = localStorage.getItem("ar5iv_theme");
    if(current_theme){
      if(current_theme == "dark"){
        theme = "dark";
      } }
    else if(!window.matchMedia) { return false; }
    else if(window.matchMedia("(prefers-color-scheme: dark)").matches) {
      theme = "dark"; }
    if (theme=="dark") {
      document.documentElement.setAttribute("data-theme", "dark");
    } else {
      document.documentElement.setAttribute("data-theme", "light"); } }

  detectColorScheme();

  function toggleColorScheme(){
    var current_theme = localStorage.getItem("ar5iv_theme");
    if (current_theme) {
      if (current_theme == "light") {
        localStorage.setItem("ar5iv_theme", "dark"); }
      else {
        localStorage.setItem("ar5iv_theme", "light"); } }
    else {
        localStorage.setItem("ar5iv_theme", "dark"); }
    detectColorScheme(); }
</script>
<link media="all" rel="stylesheet" href="/assets/ar5iv-fonts.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv-site.0.2.2.css">
</head>
<body>
<div class="ltx_page_main">
<div class="ltx_page_content">
<article class="ltx_document ltx_authors_1line">
<h1 class="ltx_title ltx_title_document">Efficiently Programming Large Language Models using SGLang</h1>
<div class="ltx_authors">
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">

Lianmin Zheng<math id="id1.1.m1.2" class="ltx_Math" alttext="{}^{\text{2},*}" display="inline"><semantics id="id1.1.m1.2a"><msup id="id1.1.m1.2.2" xref="id1.1.m1.2.2.cmml"><mi id="id1.1.m1.2.2a" xref="id1.1.m1.2.2.cmml"></mi><mrow id="id1.1.m1.2.2.2.4" xref="id1.1.m1.2.2.2.3.cmml"><mtext id="id1.1.m1.1.1.1.1" xref="id1.1.m1.1.1.1.1a.cmml">2</mtext><mo rspace="0em" id="id1.1.m1.2.2.2.4.1" xref="id1.1.m1.2.2.2.3.cmml">,</mo><mo lspace="0em" id="id1.1.m1.2.2.2.2" xref="id1.1.m1.2.2.2.2.cmml">∗</mo></mrow></msup><annotation-xml encoding="MathML-Content" id="id1.1.m1.2b"><apply id="id1.1.m1.2.2.cmml" xref="id1.1.m1.2.2"><list id="id1.1.m1.2.2.2.3.cmml" xref="id1.1.m1.2.2.2.4"><ci id="id1.1.m1.1.1.1.1a.cmml" xref="id1.1.m1.1.1.1.1"><mtext mathsize="70%" id="id1.1.m1.1.1.1.1.cmml" xref="id1.1.m1.1.1.1.1">2</mtext></ci><times id="id1.1.m1.2.2.2.2.cmml" xref="id1.1.m1.2.2.2.2"></times></list></apply></annotation-xml><annotation encoding="application/x-tex" id="id1.1.m1.2c">{}^{\text{2},*}</annotation></semantics></math>  Liangsheng Yin<math id="id2.2.m2.1" class="ltx_Math" alttext="{}^{\text{3}}" display="inline"><semantics id="id2.2.m2.1a"><msup id="id2.2.m2.1.1" xref="id2.2.m2.1.1.cmml"><mi id="id2.2.m2.1.1a" xref="id2.2.m2.1.1.cmml"></mi><mtext id="id2.2.m2.1.1.1" xref="id2.2.m2.1.1.1a.cmml">3</mtext></msup><annotation-xml encoding="MathML-Content" id="id2.2.m2.1b"><apply id="id2.2.m2.1.1.cmml" xref="id2.2.m2.1.1"><ci id="id2.2.m2.1.1.1a.cmml" xref="id2.2.m2.1.1.1"><mtext mathsize="70%" id="id2.2.m2.1.1.1.cmml" xref="id2.2.m2.1.1.1">3</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="id2.2.m2.1c">{}^{\text{3}}</annotation></semantics></math>  Zhiqiang Xie<math id="id3.3.m3.1" class="ltx_Math" alttext="{}^{\text{1}}" display="inline"><semantics id="id3.3.m3.1a"><msup id="id3.3.m3.1.1" xref="id3.3.m3.1.1.cmml"><mi id="id3.3.m3.1.1a" xref="id3.3.m3.1.1.cmml"></mi><mtext id="id3.3.m3.1.1.1" xref="id3.3.m3.1.1.1a.cmml">1</mtext></msup><annotation-xml encoding="MathML-Content" id="id3.3.m3.1b"><apply id="id3.3.m3.1.1.cmml" xref="id3.3.m3.1.1"><ci id="id3.3.m3.1.1.1a.cmml" xref="id3.3.m3.1.1.1"><mtext mathsize="70%" id="id3.3.m3.1.1.1.cmml" xref="id3.3.m3.1.1.1">1</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="id3.3.m3.1c">{}^{\text{1}}</annotation></semantics></math>  Jeff Huang<math id="id4.4.m4.1" class="ltx_Math" alttext="{}^{\text{4}}" display="inline"><semantics id="id4.4.m4.1a"><msup id="id4.4.m4.1.1" xref="id4.4.m4.1.1.cmml"><mi id="id4.4.m4.1.1a" xref="id4.4.m4.1.1.cmml"></mi><mtext id="id4.4.m4.1.1.1" xref="id4.4.m4.1.1.1a.cmml">4</mtext></msup><annotation-xml encoding="MathML-Content" id="id4.4.m4.1b"><apply id="id4.4.m4.1.1.cmml" xref="id4.4.m4.1.1"><ci id="id4.4.m4.1.1.1a.cmml" xref="id4.4.m4.1.1.1"><mtext mathsize="70%" id="id4.4.m4.1.1.1.cmml" xref="id4.4.m4.1.1.1">4</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="id4.4.m4.1c">{}^{\text{4}}</annotation></semantics></math>  Chuyue Sun<math id="id5.5.m5.1" class="ltx_Math" alttext="{}^{\text{1}}" display="inline"><semantics id="id5.5.m5.1a"><msup id="id5.5.m5.1.1" xref="id5.5.m5.1.1.cmml"><mi id="id5.5.m5.1.1a" xref="id5.5.m5.1.1.cmml"></mi><mtext id="id5.5.m5.1.1.1" xref="id5.5.m5.1.1.1a.cmml">1</mtext></msup><annotation-xml encoding="MathML-Content" id="id5.5.m5.1b"><apply id="id5.5.m5.1.1.cmml" xref="id5.5.m5.1.1"><ci id="id5.5.m5.1.1.1a.cmml" xref="id5.5.m5.1.1.1"><mtext mathsize="70%" id="id5.5.m5.1.1.1.cmml" xref="id5.5.m5.1.1.1">1</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="id5.5.m5.1c">{}^{\text{1}}</annotation></semantics></math>  Cody Hao Yu<math id="id6.6.m6.1" class="ltx_Math" alttext="{}^{\text{5}}" display="inline"><semantics id="id6.6.m6.1a"><msup id="id6.6.m6.1.1" xref="id6.6.m6.1.1.cmml"><mi id="id6.6.m6.1.1a" xref="id6.6.m6.1.1.cmml"></mi><mtext id="id6.6.m6.1.1.1" xref="id6.6.m6.1.1.1a.cmml">5</mtext></msup><annotation-xml encoding="MathML-Content" id="id6.6.m6.1b"><apply id="id6.6.m6.1.1.cmml" xref="id6.6.m6.1.1"><ci id="id6.6.m6.1.1.1a.cmml" xref="id6.6.m6.1.1.1"><mtext mathsize="70%" id="id6.6.m6.1.1.1.cmml" xref="id6.6.m6.1.1.1">5</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="id6.6.m6.1c">{}^{\text{5}}</annotation></semantics></math>  Shiyi Cao<math id="id7.7.m7.1" class="ltx_Math" alttext="{}^{\text{2}}" display="inline"><semantics id="id7.7.m7.1a"><msup id="id7.7.m7.1.1" xref="id7.7.m7.1.1.cmml"><mi id="id7.7.m7.1.1a" xref="id7.7.m7.1.1.cmml"></mi><mtext id="id7.7.m7.1.1.1" xref="id7.7.m7.1.1.1a.cmml">2</mtext></msup><annotation-xml encoding="MathML-Content" id="id7.7.m7.1b"><apply id="id7.7.m7.1.1.cmml" xref="id7.7.m7.1.1"><ci id="id7.7.m7.1.1.1a.cmml" xref="id7.7.m7.1.1.1"><mtext mathsize="70%" id="id7.7.m7.1.1.1.cmml" xref="id7.7.m7.1.1.1">2</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="id7.7.m7.1c">{}^{\text{2}}</annotation></semantics></math>  

<br class="ltx_break">

Christos Kozyrakis<math id="id8.8.m8.1" class="ltx_Math" alttext="{}^{\text{1}}" display="inline"><semantics id="id8.8.m8.1a"><msup id="id8.8.m8.1.1" xref="id8.8.m8.1.1.cmml"><mi id="id8.8.m8.1.1a" xref="id8.8.m8.1.1.cmml"></mi><mtext id="id8.8.m8.1.1.1" xref="id8.8.m8.1.1.1a.cmml">1</mtext></msup><annotation-xml encoding="MathML-Content" id="id8.8.m8.1b"><apply id="id8.8.m8.1.1.cmml" xref="id8.8.m8.1.1"><ci id="id8.8.m8.1.1.1a.cmml" xref="id8.8.m8.1.1.1"><mtext mathsize="70%" id="id8.8.m8.1.1.1.cmml" xref="id8.8.m8.1.1.1">1</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="id8.8.m8.1c">{}^{\text{1}}</annotation></semantics></math>  Ion Stoica<math id="id9.9.m9.1" class="ltx_Math" alttext="{}^{\text{2}}" display="inline"><semantics id="id9.9.m9.1a"><msup id="id9.9.m9.1.1" xref="id9.9.m9.1.1.cmml"><mi id="id9.9.m9.1.1a" xref="id9.9.m9.1.1.cmml"></mi><mtext id="id9.9.m9.1.1.1" xref="id9.9.m9.1.1.1a.cmml">2</mtext></msup><annotation-xml encoding="MathML-Content" id="id9.9.m9.1b"><apply id="id9.9.m9.1.1.cmml" xref="id9.9.m9.1.1"><ci id="id9.9.m9.1.1.1a.cmml" xref="id9.9.m9.1.1.1"><mtext mathsize="70%" id="id9.9.m9.1.1.1.cmml" xref="id9.9.m9.1.1.1">2</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="id9.9.m9.1c">{}^{\text{2}}</annotation></semantics></math>  Joseph E. Gonzalez<math id="id10.10.m10.1" class="ltx_Math" alttext="{}^{\text{2}}" display="inline"><semantics id="id10.10.m10.1a"><msup id="id10.10.m10.1.1" xref="id10.10.m10.1.1.cmml"><mi id="id10.10.m10.1.1a" xref="id10.10.m10.1.1.cmml"></mi><mtext id="id10.10.m10.1.1.1" xref="id10.10.m10.1.1.1a.cmml">2</mtext></msup><annotation-xml encoding="MathML-Content" id="id10.10.m10.1b"><apply id="id10.10.m10.1.1.cmml" xref="id10.10.m10.1.1"><ci id="id10.10.m10.1.1.1a.cmml" xref="id10.10.m10.1.1.1"><mtext mathsize="70%" id="id10.10.m10.1.1.1.cmml" xref="id10.10.m10.1.1.1">2</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="id10.10.m10.1c">{}^{\text{2}}</annotation></semantics></math>  Clark Barrett<math id="id11.11.m11.1" class="ltx_Math" alttext="{}^{\text{1}}" display="inline"><semantics id="id11.11.m11.1a"><msup id="id11.11.m11.1.1" xref="id11.11.m11.1.1.cmml"><mi id="id11.11.m11.1.1a" xref="id11.11.m11.1.1.cmml"></mi><mtext id="id11.11.m11.1.1.1" xref="id11.11.m11.1.1.1a.cmml">1</mtext></msup><annotation-xml encoding="MathML-Content" id="id11.11.m11.1b"><apply id="id11.11.m11.1.1.cmml" xref="id11.11.m11.1.1"><ci id="id11.11.m11.1.1.1a.cmml" xref="id11.11.m11.1.1.1"><mtext mathsize="70%" id="id11.11.m11.1.1.1.cmml" xref="id11.11.m11.1.1.1">1</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="id11.11.m11.1c">{}^{\text{1}}</annotation></semantics></math>  Ying Sheng<math id="id12.12.m12.1" class="ltx_Math" alttext="{}^{\text{1,*}}" display="inline"><semantics id="id12.12.m12.1a"><msup id="id12.12.m12.1.1" xref="id12.12.m12.1.1.cmml"><mi id="id12.12.m12.1.1a" xref="id12.12.m12.1.1.cmml"></mi><mtext id="id12.12.m12.1.1.1" xref="id12.12.m12.1.1.1a.cmml">1,*</mtext></msup><annotation-xml encoding="MathML-Content" id="id12.12.m12.1b"><apply id="id12.12.m12.1.1.cmml" xref="id12.12.m12.1.1"><ci id="id12.12.m12.1.1.1a.cmml" xref="id12.12.m12.1.1.1"><mtext mathsize="70%" id="id12.12.m12.1.1.1.cmml" xref="id12.12.m12.1.1.1">1,*</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="id12.12.m12.1c">{}^{\text{1,*}}</annotation></semantics></math>  

<br class="ltx_break">
<br class="ltx_break">

<math id="id13.13.m13.1" class="ltx_Math" alttext="{}^{\text{1}}" display="inline"><semantics id="id13.13.m13.1a"><msup id="id13.13.m13.1.1" xref="id13.13.m13.1.1.cmml"><mi id="id13.13.m13.1.1a" xref="id13.13.m13.1.1.cmml"></mi><mtext id="id13.13.m13.1.1.1" xref="id13.13.m13.1.1.1a.cmml">1</mtext></msup><annotation-xml encoding="MathML-Content" id="id13.13.m13.1b"><apply id="id13.13.m13.1.1.cmml" xref="id13.13.m13.1.1"><ci id="id13.13.m13.1.1.1a.cmml" xref="id13.13.m13.1.1.1"><mtext mathsize="70%" id="id13.13.m13.1.1.1.cmml" xref="id13.13.m13.1.1.1">1</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="id13.13.m13.1c">{}^{\text{1}}</annotation></semantics></math>Stanford University <math id="id14.14.m14.1" class="ltx_Math" alttext="{}^{\text{2}}" display="inline"><semantics id="id14.14.m14.1a"><msup id="id14.14.m14.1.1" xref="id14.14.m14.1.1.cmml"><mi id="id14.14.m14.1.1a" xref="id14.14.m14.1.1.cmml"></mi><mtext id="id14.14.m14.1.1.1" xref="id14.14.m14.1.1.1a.cmml">2</mtext></msup><annotation-xml encoding="MathML-Content" id="id14.14.m14.1b"><apply id="id14.14.m14.1.1.cmml" xref="id14.14.m14.1.1"><ci id="id14.14.m14.1.1.1a.cmml" xref="id14.14.m14.1.1.1"><mtext mathsize="70%" id="id14.14.m14.1.1.1.cmml" xref="id14.14.m14.1.1.1">2</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="id14.14.m14.1c">{}^{\text{2}}</annotation></semantics></math>UC Berkeley <math id="id15.15.m15.1" class="ltx_Math" alttext="{}^{\text{3}}" display="inline"><semantics id="id15.15.m15.1a"><msup id="id15.15.m15.1.1" xref="id15.15.m15.1.1.cmml"><mi id="id15.15.m15.1.1a" xref="id15.15.m15.1.1.cmml"></mi><mtext id="id15.15.m15.1.1.1" xref="id15.15.m15.1.1.1a.cmml">3</mtext></msup><annotation-xml encoding="MathML-Content" id="id15.15.m15.1b"><apply id="id15.15.m15.1.1.cmml" xref="id15.15.m15.1.1"><ci id="id15.15.m15.1.1.1a.cmml" xref="id15.15.m15.1.1.1"><mtext mathsize="70%" id="id15.15.m15.1.1.1.cmml" xref="id15.15.m15.1.1.1">3</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="id15.15.m15.1c">{}^{\text{3}}</annotation></semantics></math>Shanghai Jiao Tong University 

<br class="ltx_break">

<math id="id16.16.m16.1" class="ltx_Math" alttext="{}^{\text{4}}" display="inline"><semantics id="id16.16.m16.1a"><msup id="id16.16.m16.1.1" xref="id16.16.m16.1.1.cmml"><mi id="id16.16.m16.1.1a" xref="id16.16.m16.1.1.cmml"></mi><mtext id="id16.16.m16.1.1.1" xref="id16.16.m16.1.1.1a.cmml">4</mtext></msup><annotation-xml encoding="MathML-Content" id="id16.16.m16.1b"><apply id="id16.16.m16.1.1.cmml" xref="id16.16.m16.1.1"><ci id="id16.16.m16.1.1.1a.cmml" xref="id16.16.m16.1.1.1"><mtext mathsize="70%" id="id16.16.m16.1.1.1.cmml" xref="id16.16.m16.1.1.1">4</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="id16.16.m16.1c">{}^{\text{4}}</annotation></semantics></math>Texas A&amp;M University <math id="id17.17.m17.1" class="ltx_Math" alttext="{}^{\text{5}}" display="inline"><semantics id="id17.17.m17.1a"><msup id="id17.17.m17.1.1" xref="id17.17.m17.1.1.cmml"><mi id="id17.17.m17.1.1a" xref="id17.17.m17.1.1.cmml"></mi><mtext id="id17.17.m17.1.1.1" xref="id17.17.m17.1.1.1a.cmml">5</mtext></msup><annotation-xml encoding="MathML-Content" id="id17.17.m17.1b"><apply id="id17.17.m17.1.1.cmml" xref="id17.17.m17.1.1"><ci id="id17.17.m17.1.1.1a.cmml" xref="id17.17.m17.1.1.1"><mtext mathsize="70%" id="id17.17.m17.1.1.1.cmml" xref="id17.17.m17.1.1.1">5</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="id17.17.m17.1c">{}^{\text{5}}</annotation></semantics></math>Independent Researcher
</span></span>
</div>

<div class="ltx_abstract">
<h6 class="ltx_title ltx_title_abstract">Abstract</h6>
<p id="id18.1" class="ltx_p">Large language models (LLMs) are increasingly used for complex tasks requiring multiple chained generation calls, advanced prompting techniques, control flow, and interaction with external environments. However, efficient systems for programming and executing these applications are lacking.
To bridge this gap, we introduce SGLang, a <span id="id18.1.1" class="ltx_text ltx_framed ltx_framed_underline">S</span>tructured <span id="id18.1.2" class="ltx_text ltx_framed ltx_framed_underline">G</span>eneration <span id="id18.1.3" class="ltx_text ltx_framed ltx_framed_underline">Lang</span>uage for LLMs. SGLang is designed for the efficient programming of LLMs and incorporates primitives for common LLM programming patterns. We have implemented SGLang as a domain-specific language embedded in Python, and we developed an interpreter, a compiler, and a high-performance runtime for SGLang.
These components work together to enable optimizations such as parallelism, batching, caching, sharing, and other compilation techniques. Additionally, we propose RadixAttention, a novel technique that maintains a Least Recently Used (LRU) cache of the Key-Value (KV) cache for all requests in a radix tree, enabling automatic KV cache reuse across multiple generation calls at runtime. SGLang simplifies the writing of LLM programs and boosts execution efficiency.
Our experiments demonstrate that SGLang can speed up common LLM tasks by up to <math id="id18.1.m1.1" class="ltx_math_unparsed" alttext="5\times" display="inline"><semantics id="id18.1.m1.1a"><mrow id="id18.1.m1.1b"><mn id="id18.1.m1.1.1">5</mn><mo lspace="0.222em" id="id18.1.m1.1.2">×</mo></mrow><annotation encoding="application/x-tex" id="id18.1.m1.1c">5\times</annotation></semantics></math>, while reducing code complexity and enhancing control.</p>
</div>
<span id="footnote1" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">†</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">†</sup><sup id="footnote1.1" class="ltx_sup">∗</sup>Equal contribution.</span></span></span>
<section id="S1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">1 </span>Introduction</h2>

<div id="S1.p1" class="ltx_para">
<p id="S1.p1.1" class="ltx_p">Recent increases in the capabilities of LLMs have broadened their utility, enabling them to tackle a wider range of general tasks and act as autonomous agents <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib40" title="" class="ltx_ref">40</a>, <a href="#bib.bib6" title="" class="ltx_ref">6</a>, <a href="#bib.bib42" title="" class="ltx_ref">42</a>, <a href="#bib.bib61" title="" class="ltx_ref">61</a>]</cite>.
In such applications, LLMs engage in <em id="S1.p1.1.1" class="ltx_emph ltx_font_italic">multi-round</em> planning, reasoning, and interaction with external environments.
This is accomplished through tool usage <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib49" title="" class="ltx_ref">49</a>]</cite>, multiple input modalities <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib1" title="" class="ltx_ref">1</a>]</cite>, and a wide-range of prompting techniques <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib31" title="" class="ltx_ref">31</a>]</cite>, like few-shot learning <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib5" title="" class="ltx_ref">5</a>]</cite>,
self-consistency <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib63" title="" class="ltx_ref">63</a>]</cite>, skeleton-of-thought <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib37" title="" class="ltx_ref">37</a>]</cite>, and tree-of-thought <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib70" title="" class="ltx_ref">70</a>]</cite>.
All of these new use cases require multiple, often dependent, LLM generation calls, and critically, the control flow of these programs may depend on the output of LLMs <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib71" title="" class="ltx_ref">71</a>]</cite>.
<em id="S1.p1.1.2" class="ltx_emph ltx_font_italic">The emergence of these complex patterns signifies a shift in our interaction with LLMs, moving from simple turn-based chatting to a more sophisticated form of programmatic usage of LLMs (see <a href="#S2.SS2" title="2.2 The Paradigm of Programming LLMs ‣ 2 Background ‣ Efficiently Programming Large Language Models using SGLang" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Section 2.2</span></a>).</em>
<a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ Efficiently Programming Large Language Models using SGLang" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 1</span></a> shows an example of such an LLM program, where multiple generation calls are used to accomplish a task. The example uses SGLang, a new domain-specific language we introduce in this paper and explain in <a href="#S4" title="4 Language, Interpreter, and Compiler ‣ Efficiently Programming Large Language Models using SGLang" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Section 4</span></a>.</p>
</div>
<figure id="S1.F1" class="ltx_figure"><img src="/html/2312.07104/assets/x1.png" id="S1.F1.g1" class="ltx_graphics ltx_centering ltx_img_square" width="461" height="376" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S1.F1.7.1.1" class="ltx_text" style="font-size:90%;">Figure 1</span>: </span><span id="S1.F1.8.2" class="ltx_text" style="font-size:90%;">
The implementation of a multi-dimensional essay judge in SGLang utilizes the branch-solve-merge prompting technique <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib48" title="" class="ltx_ref">48</a>]</cite>.
This function uses LLMs to evaluate the quality of an essay from multiple dimensions, merges the judgments, generates a summary, and assigns a final grade.
The highlighted regions illustrate the use of SGLang APIs.
(1) <span id="S1.F1.8.2.1" class="ltx_text ltx_font_typewriter">fork</span> creates multiple parallel copies of a prompt.
(2) <span id="S1.F1.8.2.2" class="ltx_text ltx_font_typewriter">gen</span> invokes an LLM generation and stores the result in a variable.
(3) <span id="S1.F1.8.2.3" class="ltx_text ltx_font_typewriter">[variable_name]</span> retrieves the result of the generation.
(4) <span id="S1.F1.8.2.4" class="ltx_text ltx_font_typewriter">choices</span> imposes constraints on the generation.
(5) <span id="S1.F1.8.2.5" class="ltx_text ltx_font_typewriter">run</span> executes a SGLang function with its arguments.
</span></figcaption>
</figure>
<div id="S1.p2" class="ltx_para">
<p id="S1.p2.1" class="ltx_p">To align with emerging trends, the community has seen the development and rising popularity of new interfaces such as LangChain <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib24" title="" class="ltx_ref">24</a>]</cite>, LMQL <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib4" title="" class="ltx_ref">4</a>]</cite>, Guidance <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib16" title="" class="ltx_ref">16</a>]</cite>, and DSPy <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib21" title="" class="ltx_ref">21</a>]</cite>.
These tools help users manage prompt templates and program pipelines with ease. However, their primary focus on front-end design often results in compromised runtime performance.</p>
</div>
<div id="S1.p3" class="ltx_para">
<p id="S1.p3.1" class="ltx_p">Conversely, while back-end inference engines like NVIDIA TensorRT-LLM <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib38" title="" class="ltx_ref">38</a>]</cite>, Hugging Face TGI <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib19" title="" class="ltx_ref">19</a>]</cite>, Orca <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib74" title="" class="ltx_ref">74</a>]</cite>, and vLLM <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib23" title="" class="ltx_ref">23</a>]</cite> continue to reduce latency and improve throughput, their design largely remains anchored in traditional single-generation call interfaces. They typically support APIs similar to the OpenAI Completion API <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib39" title="" class="ltx_ref">39</a>]</cite>, where a user prompt is processed and a response is returned in a stateless manner.</p>
</div>
<div id="S1.p4" class="ltx_para">
<p id="S1.p4.1" class="ltx_p">While both front-end and back-end technologies are advancing rapidly, they are evolving along separate trajectories. The front end largely remains oblivious to the LLM inference processes, while the back end often lacks awareness of the application’s structure and is unable to optimize across multiple calls. This disjointed development leads to numerous missed opportunities for performance optimization, such as KV cache sharing and parallelism, as discussed in <a href="#S2.SS3" title="2.3 Challenges and Opportunities ‣ 2 Background ‣ Efficiently Programming Large Language Models using SGLang" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Section 2.3</span></a>.</p>
</div>
<div id="S1.p5" class="ltx_para">
<p id="S1.p5.1" class="ltx_p">In this paper, we present a holistic approach for an efficient LLM programming system by co-designing both the front-end language and the back-end runtime.
The process involves three key challenges: (i) creating a flexible, domain-specific language with primitives suitable for LLM programming and ecosystem integration; (ii)
ensuring efficient program execution via automatic parallelism, batching, caching, and sharing across multiple calls and multiple programs; and (iii)
enhancing program performance through compiler optimizations like code movement and prefetching annotations.</p>
</div>
<div id="S1.p6" class="ltx_para">
<p id="S1.p6.1" class="ltx_p">On the front end, we introduce SGLang, a domain-specific language embedded in Python. It provides a set of primitives specifically designed to facilitate programming LLMs. These primitives enable the manipulation of prompts and generations (e.g., <span id="S1.p6.1.1" class="ltx_text ltx_font_typewriter">extend</span>, <span id="S1.p6.1.2" class="ltx_text ltx_font_typewriter">gen</span>, <span id="S1.p6.1.3" class="ltx_text ltx_font_typewriter">select</span>) and the control of parallelism (e.g., <span id="S1.p6.1.4" class="ltx_text ltx_font_typewriter">fork</span>, <span id="S1.p6.1.5" class="ltx_text ltx_font_typewriter">join</span>). Furthermore, SGLang is compatible with Python’s control flow and libraries.
Leveraging these primitives along with native Python syntax, users can
intuitively develop sophisticated prompting strategies and agents, while obviating the need for manual optimization.
Moreover, the unified language enhances portability, enabling users to transition easily between different open-source and proprietary models, such as those from OpenAI and Anthropic.</p>
</div>
<div id="S1.p7" class="ltx_para">
<p id="S1.p7.1" class="ltx_p">We develop both an interpreter and a compiler for SGLang. The interpreter is responsible for managing the prompt state and executing SGLang programs.
Specifically, it treats a prompt as a stream and submits primitive operations to this stream for asynchronous execution, ensuring proper control over synchronization and intra-program parallelism.
Additionally, we can trace a SGLang program and utilize the compiler to compile it into a computational graph.
This approach opens opportunities for more aggressive optimizations. We explore several classical compiler optimizations, e.g., code movement, in the context of LLM programming.</p>
</div>
<div id="S1.p8" class="ltx_para">
<p id="S1.p8.1" class="ltx_p">On the back end, we propose a novel technique called RadixAttention to automatically reuse the KV cache across multiple generation calls. In existing systems, the KV cache of a request is discarded after processing is completed, which prevents KV cache from being reused across multiple generation calls.
Instead, our system maintains an LRU cache of the KV cache for all requests within a radix tree. This approach allows the runtime to efficiently achieve automatic sharing for various reuse patterns.
Additionally, to increase the cache hit rate, we utilize a cache-aware scheduling policy.</p>
</div>
<div id="S1.p9" class="ltx_para">
<p id="S1.p9.1" class="ltx_p">In summary, we make the following contributions:</p>
<ul id="S1.I1" class="ltx_itemize">
<li id="S1.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S1.I1.i1.p1" class="ltx_para">
<p id="S1.I1.i1.p1.1" class="ltx_p">We present SGLang, a domain-specific language with primitives for LLM programming, supporting parallelism, control flow, nested calls, and external calls.</p>
</div>
</li>
<li id="S1.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S1.I1.i2.p1" class="ltx_para">
<p id="S1.I1.i2.p1.1" class="ltx_p">We develop an interpreter capable of executing SGLang programs in tandem with the Python interpreter and managing prompt states as asynchronous streams.</p>
</div>
</li>
<li id="S1.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S1.I1.i3.p1" class="ltx_para">
<p id="S1.I1.i3.p1.1" class="ltx_p">We build a compiler capable of compiling programs into computational graphs and explore classical compiler techniques in the context of LLM programming.</p>
</div>
</li>
<li id="S1.I1.i4" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S1.I1.i4.p1" class="ltx_para">
<p id="S1.I1.i4.p1.1" class="ltx_p">We propose RadixAttention, a novel technique for automatic KV cache reuse across multiple generation calls at runtime.</p>
</div>
</li>
<li id="S1.I1.i5" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S1.I1.i5.p1" class="ltx_para">
<p id="S1.I1.i5.p1.1" class="ltx_p">Using SGLang, we implement various LLM applications, including agent control, logical reasoning, content generation, benchmarking, and long document processing.
Our experiments demonstrate that SGLang can accelerate common LLM tasks by up to <math id="S1.I1.i5.p1.1.m1.1" class="ltx_math_unparsed" alttext="5\times" display="inline"><semantics id="S1.I1.i5.p1.1.m1.1a"><mrow id="S1.I1.i5.p1.1.m1.1b"><mn id="S1.I1.i5.p1.1.m1.1.1">5</mn><mo lspace="0.222em" id="S1.I1.i5.p1.1.m1.1.2">×</mo></mrow><annotation encoding="application/x-tex" id="S1.I1.i5.p1.1.m1.1c">5\times</annotation></semantics></math>, while reducing code complexity and enhancing control.</p>
</div>
</li>
</ul>
</div>
</section>
<section id="S2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">2 </span>Background</h2>

<section id="S2.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.1 </span>The Inference Process of LLMs</h3>

<div id="S2.SS1.p1" class="ltx_para">
<p id="S2.SS1.p1.1" class="ltx_p">Most LLMs in use today, such as GPT-3 <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib5" title="" class="ltx_ref">5</a>]</cite>, PaLM <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib8" title="" class="ltx_ref">8</a>]</cite>, and LLaMA <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib58" title="" class="ltx_ref">58</a>]</cite>, are based on the autoregressive Transformer architecture <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib60" title="" class="ltx_ref">60</a>]</cite>.
The models predict the probability of the next token in a sequence based on the preceding tokens.
During inference, the model first processes a sequence of input tokens through a forward pass.
It then sequentially decodes output tokens, with each token depending on prior tokens.
We refer to the process of taking a sequence of input tokens and generating a sequence of output tokens as a single generation call.
Throughout this process, each token generates some intermediate tensors, which are used for decoding further tokens.
These intermediate tensors, known as the "KV Cache", are named for the key-value pairs in the self-attention mechanism.
An observation that will be important when discussing optimizations later is that the computation of the KV cache only depends on all previous tokens, so different sequences with the same prefix can reuse the KV cache of the prefix tokens and avoid redundant computation.</p>
</div>
</section>
<section id="S2.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.2 </span>The Paradigm of Programming LLMs</h3>

<div id="S2.SS2.p1" class="ltx_para">
<p id="S2.SS2.p1.1" class="ltx_p">As LLMs become more powerful, they are increasingly employed as versatile task solvers in complex systems. In these contexts, it is imperative that LLMs operate as dependable and intelligent components.
This evolves the use of LLMs from single, straightforward generation calls to multiple, interdependent generation calls interleaved with control flow, leading to the emergence of the "Programming LLMs" paradigm.
Programming LLMs refers to the development of computer programs that control the generation processes of LLMs. This concept can be exemplified in several ways.</p>
</div>
<div id="S2.SS2.p2" class="ltx_para">
<p id="S2.SS2.p2.1" class="ltx_p">First, advanced prompting techniques are examples of programming LLMs. An exemplary case is meta-reasoning <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib73" title="" class="ltx_ref">73</a>]</cite>. To solve a problem, the LLM initially generates multiple solutions through chain-of-thought prompting. It then performs meta-reasoning across these solutions to derive the final resolution. Another example is skeleton-of-thought <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib37" title="" class="ltx_ref">37</a>]</cite>. To answer a question, the LLM first generates a skeleton of the answer. It then issues multiple parallel calls to elaborate the skeleton into a detailed explanation. A more complex case is the tree-of-thought approach <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib70" title="" class="ltx_ref">70</a>]</cite>, which integrates LLM calls within tree search algorithms (e.g., breadth-first search).</p>
</div>
<div id="S2.SS2.p3" class="ltx_para">
<p id="S2.SS2.p3.1" class="ltx_p">Second, in LLM-based agents, the programmatic use of LLMs is crucial. Take the ReAct agent <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib71" title="" class="ltx_ref">71</a>]</cite>, for example. The behavior of the agent is controlled by loops and conditional statements. The agent performs one action per iteration. In each iteration, the agent chooses the action type (search or finish), generates action parameters, and processes observations from the external environment.</p>
</div>
<div id="S2.SS2.p4" class="ltx_para">
<p id="S2.SS2.p4.1" class="ltx_p">Third, simpler prompting forms also fall under the scope of programming LLMs. Techniques like few-shot prompting <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib5" title="" class="ltx_ref">5</a>]</cite>, chain-of-thought prompting <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib64" title="" class="ltx_ref">64</a>]</cite>, and retrieval-augmented generation <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib26" title="" class="ltx_ref">26</a>]</cite> are basic elements in programming LLMs.</p>
</div>
<figure id="S2.F2" class="ltx_figure"><img src="/html/2312.07104/assets/x2.png" id="S2.F2.g1" class="ltx_graphics ltx_centering ltx_img_portrait" width="461" height="665" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S2.F2.2.1.1" class="ltx_text" style="font-size:90%;">Figure 2</span>: </span><span id="S2.F2.3.2" class="ltx_text" style="font-size:90%;">KV cache sharing examples. Blue boxes represent shareable prompt parts, green boxes indicate non-shareable parts, and yellow boxes mark non-shareable model outputs. Shareable elements include few-shot learning examples, questions in self-consistency <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib63" title="" class="ltx_ref">63</a>]</cite>, chat history in multi-turn chats, and search history in tree-of-thought <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib70" title="" class="ltx_ref">70</a>]</cite>.</span></figcaption>
</figure>
<figure id="S2.T1" class="ltx_table">
<table id="S2.T1.2" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S2.T1.2.3.1" class="ltx_tr">
<th id="S2.T1.2.3.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt"><span id="S2.T1.2.3.1.1.1" class="ltx_text" style="font-size:80%;">Language</span></th>
<th id="S2.T1.2.3.1.2" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt"><span id="S2.T1.2.3.1.2.1" class="ltx_text" style="font-size:80%;">Syntax</span></th>
<th id="S2.T1.2.3.1.3" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt"><span id="S2.T1.2.3.1.3.1" class="ltx_text" style="font-size:80%;">Primitives</span></th>
<th id="S2.T1.2.3.1.4" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt"><span id="S2.T1.2.3.1.4.1" class="ltx_text" style="font-size:80%;">Runtime Backends</span></th>
<th id="S2.T1.2.3.1.5" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt"><span id="S2.T1.2.3.1.5.1" class="ltx_text" style="font-size:80%;">Execution Modes</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S2.T1.2.4.1" class="ltx_tr">
<td id="S2.T1.2.4.1.1" class="ltx_td ltx_align_left ltx_border_t"><span id="S2.T1.2.4.1.1.1" class="ltx_text" style="font-size:80%;">LMQL</span></td>
<td id="S2.T1.2.4.1.2" class="ltx_td ltx_align_left ltx_border_t"><span id="S2.T1.2.4.1.2.1" class="ltx_text" style="font-size:80%;">Custom</span></td>
<td id="S2.T1.2.4.1.3" class="ltx_td ltx_align_left ltx_border_t"><span id="S2.T1.2.4.1.3.1" class="ltx_text" style="font-size:80%;">extend, gen, select</span></td>
<td id="S2.T1.2.4.1.4" class="ltx_td ltx_align_left ltx_border_t"><span id="S2.T1.2.4.1.4.1" class="ltx_text" style="font-size:80%;">Hugging Face Transformers, llama.cpp, OpenAI</span></td>
<td id="S2.T1.2.4.1.5" class="ltx_td ltx_align_left ltx_border_t"><span id="S2.T1.2.4.1.5.1" class="ltx_text" style="font-size:80%;">Compiler</span></td>
</tr>
<tr id="S2.T1.2.5.2" class="ltx_tr">
<td id="S2.T1.2.5.2.1" class="ltx_td ltx_align_left"><span id="S2.T1.2.5.2.1.1" class="ltx_text" style="font-size:80%;">Guidance</span></td>
<td id="S2.T1.2.5.2.2" class="ltx_td ltx_align_left"><span id="S2.T1.2.5.2.2.1" class="ltx_text" style="font-size:80%;">Python</span></td>
<td id="S2.T1.2.5.2.3" class="ltx_td ltx_align_left"><span id="S2.T1.2.5.2.3.1" class="ltx_text" style="font-size:80%;">extend, gen, select</span></td>
<td id="S2.T1.2.5.2.4" class="ltx_td ltx_align_left"><span id="S2.T1.2.5.2.4.1" class="ltx_text" style="font-size:80%;">Hugging Face Transformers, llama.cpp, OpenAI, VertexAI</span></td>
<td id="S2.T1.2.5.2.5" class="ltx_td ltx_align_left"><span id="S2.T1.2.5.2.5.1" class="ltx_text" style="font-size:80%;">Interpreter</span></td>
</tr>
<tr id="S2.T1.2.2" class="ltx_tr">
<td id="S2.T1.2.2.3" class="ltx_td ltx_align_left ltx_border_bb ltx_border_t"><span id="S2.T1.2.2.3.1" class="ltx_text" style="font-size:80%;">SGLang</span></td>
<td id="S2.T1.2.2.4" class="ltx_td ltx_align_left ltx_border_bb ltx_border_t"><span id="S2.T1.2.2.4.1" class="ltx_text" style="font-size:80%;">Python</span></td>
<td id="S2.T1.2.2.5" class="ltx_td ltx_align_left ltx_border_bb ltx_border_t"><span id="S2.T1.2.2.5.1" class="ltx_text" style="font-size:80%;">extend, gen, select, fork, join</span></td>
<td id="S2.T1.2.2.2" class="ltx_td ltx_align_left ltx_border_bb ltx_border_t">
<span id="S2.T1.2.2.2.1" class="ltx_text" style="font-size:80%;">SGVM, OpenAI, Anthropic, Hugging Face TGI</span><sup id="S2.T1.2.2.2.2" class="ltx_sup"><span id="S2.T1.2.2.2.2.1" class="ltx_text" style="font-size:80%;">∗</span></sup><span id="S2.T1.2.2.2.3" class="ltx_text" style="font-size:80%;">, llama.cpp</span><sup id="S2.T1.2.2.2.4" class="ltx_sup"><span id="S2.T1.2.2.2.4.1" class="ltx_text" style="font-size:80%;">∗</span></sup>
</td>
<td id="S2.T1.2.2.6" class="ltx_td ltx_align_left ltx_border_bb ltx_border_t"><span id="S2.T1.2.2.6.1" class="ltx_text" style="font-size:80%;">Interpreter and compiler</span></td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering" style="font-size:80%;"><span class="ltx_tag ltx_tag_table"><span id="S2.T1.10.2.1" class="ltx_text" style="font-size:113%;">Table 1</span>: </span><span id="S2.T1.4.1" class="ltx_text" style="font-size:113%;">Comparison among LMQL, Guidance, and SGLang. The backends marked with <sup id="S2.T1.4.1.1" class="ltx_sup">∗</sup> are experimental.</span></figcaption>
</figure>
</section>
<section id="S2.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.3 </span>Challenges and Opportunities</h3>

<div id="S2.SS3.p1" class="ltx_para">
<p id="S2.SS3.p1.1" class="ltx_p">Designing programming systems for LLMs presents unique challenges and opportunities.
A good system should help a programmer be productive, offering an intuitive and expressive interface for common programming tasks, including LLM state management, decoding process control, and smooth integration with external tools.
Furthermore, it is essential to ensure performance by optimizing for low latency and high throughput execution.
In particular, the rich structure of LLM programs and the unique properties of LLM inference provide plenty of opportunities for runtime optimizations. These include:</p>
</div>
<div id="S2.SS3.p2" class="ltx_para">
<ul id="S2.I1" class="ltx_itemize">
<li id="S2.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S2.I1.i1.p1" class="ltx_para">
<p id="S2.I1.i1.p1.1" class="ltx_p"><span id="S2.I1.i1.p1.1.1" class="ltx_text ltx_font_bold">Caching:</span> Often in LLM programs, multiple text segments and generation calls are appended to a single prompt. Caching the computed KV cache for previous tokens across multiple chained calls can reduce redundant computation. This optimization, however, is neither free nor trivial, as it requires additional storage and more complex memory management.</p>
</div>
</li>
<li id="S2.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S2.I1.i2.p1" class="ltx_para">
<p id="S2.I1.i2.p1.1" class="ltx_p"><span id="S2.I1.i2.p1.1.1" class="ltx_text ltx_font_bold">Batching:</span> The decoding process in LLMs is primarily memory-bound, meaning increasing the batch size can significantly enhance throughput. Running a large number of LLM programs in a batch, and utilizing the contiguous batching technique <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib74" title="" class="ltx_ref">74</a>]</cite> can be beneficial.</p>
</div>
</li>
<li id="S2.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S2.I1.i3.p1" class="ltx_para">
<p id="S2.I1.i3.p1.1" class="ltx_p"><span id="S2.I1.i3.p1.1.1" class="ltx_text ltx_font_bold">Sharing:</span> It is common in LLM programs to generate multiple outputs from a single prompt or to fork a new prompt from the current state. Basic prefix sharing has been investigated in vLLM <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib23" title="" class="ltx_ref">23</a>]</cite>. More advanced sharing patterns like irregular tree-structured sharing can also be employed. <a href="#S2.F2" title="Figure 2 ‣ 2.2 The Paradigm of Programming LLMs ‣ 2 Background ‣ Efficiently Programming Large Language Models using SGLang" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 2</span></a> shows four typical patterns of KV cache sharing across multiple calls; none of the existing systems can automatically handle all of them.</p>
</div>
</li>
<li id="S2.I1.i4" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S2.I1.i4.p1" class="ltx_para">
<p id="S2.I1.i4.p1.1" class="ltx_p"><span id="S2.I1.i4.p1.1.1" class="ltx_text ltx_font_bold">Parallelism:</span> An LLM program often includes multiple generation calls. By creating a dependency graph for these calls, we can identify independent calls and execute them in parallel. This creates intra-program parallelism, which adds additional parallelism to the straightforward inter-program parallelism.
For example, in <a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ Efficiently Programming Large Language Models using SGLang" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 1</span></a>, the generation of multiple judgments can be parallelized.</p>
</div>
</li>
<li id="S2.I1.i5" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S2.I1.i5.p1" class="ltx_para">
<p id="S2.I1.i5.p1.1" class="ltx_p"><span id="S2.I1.i5.p1.1.1" class="ltx_text ltx_font_bold">Compilation:</span> Once a full program is available, it can be compiled into an optimized intermediate representation for more efficient execution. More aggressive optimizations can also be implemented, such as automatically adjusting the original prompts based on user-provided test cases. These optimizations may include expanding, compressing, rearranging parts of the prompts, and selecting different models for different calls.</p>
</div>
</li>
</ul>
</div>
<div id="S2.SS3.p3" class="ltx_para">
<p id="S2.SS3.p3.1" class="ltx_p">To systematically explore these opportunities, it is essential to co-design the programming interface and runtime. However, inference engines like Orca and vLLM fall short, as they optimize only for single generation calls. This limitation leads to redundant computations and missed opportunities when executing LLM programs that involve multiple calls.</p>
</div>
</section>
<section id="S2.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.4 </span>Existing LLM Programming Systems</h3>

<div id="S2.SS4.p1" class="ltx_para">
<p id="S2.SS4.p1.1" class="ltx_p">Given the significance of LLM programming, specialized programming systems have been developed. The top layer comprises high-level application development libraries with pre-built modules, such as LangChain <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib24" title="" class="ltx_ref">24</a>]</cite>. The middle layer features more flexible pipeline programming frameworks like DSPy <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib21" title="" class="ltx_ref">21</a>]</cite>. At the bottom layer, there are domain-specific languages designed for controlling a single prompt, including LMQL <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib4" title="" class="ltx_ref">4</a>]</cite> and Guidance <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib16" title="" class="ltx_ref">16</a>]</cite>.</p>
</div>
<div id="S2.SS4.p2" class="ltx_para">
<p id="S2.SS4.p2.1" class="ltx_p">SGLang, uniquely positioned between the middle and bottom layers, shares similarities with LMQL and Guidance. However, unlike LMQL and Guidance, which primarily focus on language design while often overlooking runtime efficiency, SGLang thoughtfully integrates front-end language features with back-end runtime optimizations. This approach significantly enhances runtime efficiency, sometimes by an order of magnitude or more.
For its front-end, LMQL employs an SQL-like custom syntax, contrasting with the more intuitive Python syntax. Guidance initially used the Handlebars template language and later transitioned to a Pythonic language embedded within Python. The front-end language of SGLang, inspired by Guidance, is being developed concurrently with the new version of Guidance.</p>
</div>
<div id="S2.SS4.p3" class="ltx_para">
<p id="S2.SS4.p3.1" class="ltx_p">SGLang introduces new primitives for enhanced expressiveness and runtime efficiency. It also offers the flexibility of both interpreter and compiler modes. A comparison among LMQL, Guidance, and SGLang is presented in <a href="#S2.T1" title="Table 1 ‣ 2.2 The Paradigm of Programming LLMs ‣ 2 Background ‣ Efficiently Programming Large Language Models using SGLang" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Table 1</span></a>. We omit the comparison with LangChain and DSPy because they are at a higher level. It is possible to compile high-level pipelines from LangChain and DSPy to SGLang.</p>
</div>
</section>
</section>
<section id="S3" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">3 </span>Overview</h2>

<div id="S3.p1" class="ltx_para">
<p id="S3.p1.1" class="ltx_p"><a href="#S3.F3" title="Figure 3 ‣ 3 Overview ‣ Efficiently Programming Large Language Models using SGLang" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 3</span></a> shows an overview of our system.
At the user level, SGLang programs are written in Python with embedded SGLang primitives for managing prompts and generation calls.
The syntax and primitives of SGLang will be introduced in <a href="#S4.SS1" title="4.1 Syntax and Primitives ‣ 4 Language, Interpreter, and Compiler ‣ Efficiently Programming Large Language Models using SGLang" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Section 4.1</span></a>.
SGLang programs can be executed in two ways: either using an interpreter or a compiler. When using the interpreter, SGLang programs, which are also valid Python programs, are processed by both the Python and SGLang interpreters. The SGLang interpreter handles specific SGLang primitives.
The design of this interpreter is explained in <a href="#S4.SS2" title="4.2 Interpreter ‣ 4 Language, Interpreter, and Compiler ‣ Efficiently Programming Large Language Models using SGLang" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Section 4.2</span></a>.
Alternatively, a limited set of SGLang programs can be compiled, enabling further optimizations.
<a href="#S4.SS3" title="4.3 Compiler ‣ 4 Language, Interpreter, and Compiler ‣ Efficiently Programming Large Language Models using SGLang" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Section 4.3</span></a> is dedicated to the SGLang compiler.
Both the interpreter and the compiler work with the SGVM, which is our custom serving engine for LLM generation calls (<a href="#S5" title="5 Runtime with RadixAttention ‣ Efficiently Programming Large Language Models using SGLang" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Section 5</span></a>).
A key aspect of SGVM is its RadixAttention feature, which automatically reuses KV caches across multiple calls.</p>
</div>
<figure id="S3.F3" class="ltx_figure"><img src="/html/2312.07104/assets/x3.png" id="S3.F3.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="84" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S3.F3.2.1.1" class="ltx_text" style="font-size:90%;">Figure 3</span>: </span><span id="S3.F3.3.2" class="ltx_text" style="font-size:90%;">The overview of the system.</span></figcaption>
</figure>
</section>
<section id="S4" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">4 </span>Language, Interpreter, and Compiler</h2>

<section id="S4.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.1 </span>Syntax and Primitives</h3>

<div id="S4.SS1.p1" class="ltx_para">
<p id="S4.SS1.p1.1" class="ltx_p">SGLang is implemented as an embedded domain-specific language in Python.</p>
</div>
<div id="S4.SS1.p2" class="ltx_para">
<p id="S4.SS1.p2.1" class="ltx_p"><a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ Efficiently Programming Large Language Models using SGLang" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 1</span></a> shows an example of a SGLang program.
This program evaluates the quality of an essay from multiple dimensions using the branch-solve-merge prompting method <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib48" title="" class="ltx_ref">48</a>]</cite>.
The function <span id="S4.SS1.p2.1.1" class="ltx_text ltx_font_typewriter">essay_judge</span> takes two arguments: <span id="S4.SS1.p2.1.2" class="ltx_text ltx_font_typewriter">s</span> and <span id="S4.SS1.p2.1.3" class="ltx_text ltx_font_typewriter">essay</span>.
The variable <span id="S4.SS1.p2.1.4" class="ltx_text ltx_font_typewriter">s</span> manages the state of a prompt, allowing new strings and SGLang primitives to be appended using <span id="S4.SS1.p2.1.5" class="ltx_text ltx_font_typewriter">+=</span>.
In the function, it first adds the essay to the prompt, and then forks the prompt into three copies.
These forks run in parallel, generating judgments from different dimensions.
Finally, it merges these judgments, generates a summary, and assigns a letter grade to the essay.</p>
</div>
<div id="S4.SS1.p3" class="ltx_para">
<p id="S4.SS1.p3.1" class="ltx_p"><a href="#S4.F4" title="Figure 4 ‣ 4.2 Interpreter ‣ 4 Language, Interpreter, and Compiler ‣ Efficiently Programming Large Language Models using SGLang" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 4</span></a> presents another example.
This program generates two tips in parallel for a given topic using the skeleton-of-thought prompting method <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib37" title="" class="ltx_ref">37</a>]</cite>.
It employs two functions: <span id="S4.SS1.p3.1.1" class="ltx_text ltx_font_typewriter">expand</span>, which elaborates a short tip into a detailed paragraph, and <span id="S4.SS1.p3.1.2" class="ltx_text ltx_font_typewriter">tip_suggestion</span>, which orchestrates the entire process.
It first generates two short tips and then calls <span id="S4.SS1.p3.1.3" class="ltx_text ltx_font_typewriter">expand</span> to elaborate them in parallel.
Finally, it generates a summary.</p>
</div>
<div id="S4.SS1.p4" class="ltx_para">
<p id="S4.SS1.p4.1" class="ltx_p">These examples illustrate the use of SGLang. In SGLang, we provide the following primitives:
<span id="S4.SS1.p4.1.1" class="ltx_text ltx_font_typewriter">gen</span> for calling LLM generation,
<span id="S4.SS1.p4.1.2" class="ltx_text ltx_font_typewriter">select</span> for letting the LLM choose the option with the highest probability from a list,
<span id="S4.SS1.p4.1.3" class="ltx_text ltx_font_typewriter">+=</span> or <span id="S4.SS1.p4.1.4" class="ltx_text ltx_font_typewriter">extend</span> for extending the current prompt,
<span id="S4.SS1.p4.1.5" class="ltx_text ltx_font_typewriter">fork</span> for forking the current prompt state, and
<span id="S4.SS1.p4.1.6" class="ltx_text ltx_font_typewriter">join</span> for rejoining the forked prompt states.
Users can interleave these primitives with arbitrary Python control flow and libraries.</p>
</div>
</section>
<section id="S4.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.2 </span>Interpreter</h3>

<div id="S4.SS2.p1" class="ltx_para">
<p id="S4.SS2.p1.1" class="ltx_p">The most basic way to execute a SGLang program is through an interpreter. In SGLang, a prompt is treated as an asynchronous stream. Executing primitives such as extend, gen, and select on a prompt leads to the interpreter submitting these primitives into the stream for asynchronous execution. These submission calls are non-blocking, allowing the Python interpreter to continue executing Python code without waiting for the LLM generation to finish. This process is akin to launching CUDA kernels to CUDA streams asynchronously. Each prompt is managed by a stream executor in a background thread, enabling users to achieve parallelism.</p>
</div>
<div id="S4.SS2.p2" class="ltx_para">
<p id="S4.SS2.p2.1" class="ltx_p">When fetching the generation results from a prompt, the fetch action will be blocked until the desired generation results are ready. This is implemented using event mechanisms, where a generation call sets an event upon producing its output. Thus, the interpreter effectively manages intra-program parallelism and synchronization.</p>
</div>
<figure id="S4.F4" class="ltx_figure"><img src="/html/2312.07104/assets/x4.png" id="S4.F4.g1" class="ltx_graphics ltx_centering ltx_img_square" width="461" height="448" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S4.F4.2.1.1" class="ltx_text" style="font-size:90%;">Figure 4</span>: </span><span id="S4.F4.3.2" class="ltx_text" style="font-size:90%;">The SGLang program for parallel tip suggestion with skeleton-of-thought prompting.</span></figcaption>
</figure>
</section>
<section id="S4.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.3 </span>Compiler</h3>

<figure id="S4.F5" class="ltx_figure"><img src="/html/2312.07104/assets/x5.png" id="S4.F5.g1" class="ltx_graphics ltx_centering ltx_img_square" width="392" height="466" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S4.F5.2.1.1" class="ltx_text" style="font-size:90%;">Figure 5</span>: </span><span id="S4.F5.3.2" class="ltx_text" style="font-size:90%;">A computational graph for the program in <a href="#S4.F4" title="Figure 4 ‣ 4.2 Interpreter ‣ 4 Language, Interpreter, and Compiler ‣ Efficiently Programming Large Language Models using SGLang" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 4</span></a>. The three streams correspond to three function calls.</span></figcaption>
</figure>
<div id="S4.SS3.p1" class="ltx_para">
<p id="S4.SS3.p1.1" class="ltx_p">Another way to run SGLang programs is to compile them as computational graphs and run them with a graph executor. This opens the opportunity for more compilation optimizations, as we can rewrite the graph.</p>
</div>
<section id="S4.SS3.SSS1" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">4.3.1 </span>Tracing and Graph Executor</h4>

<div id="S4.SS3.SSS1.p1" class="ltx_para">
<p id="S4.SS3.SSS1.p1.1" class="ltx_p">We design an intermediate representation (IR) for SGLang, representing SGLang program structures and operations as a computational graph.
This graph includes nodes for primitive operators and edges for dependencies. See <a href="#S4.F5" title="Figure 5 ‣ 4.3 Compiler ‣ 4 Language, Interpreter, and Compiler ‣ Efficiently Programming Large Language Models using SGLang" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 5</span></a> for the graph corresponding to the program in <a href="#S4.F4" title="Figure 4 ‣ 4.2 Interpreter ‣ 4 Language, Interpreter, and Compiler ‣ Efficiently Programming Large Language Models using SGLang" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 4</span></a>.
In the program, each call to a decorated function and or fork will create a new prompt state, or a stream.</p>
</div>
<div id="S4.SS3.SSS1.p2" class="ltx_para">
<p id="S4.SS3.SSS1.p2.1" class="ltx_p">There are several types of nodes. Each operand of <span id="S4.SS3.SSS1.p2.1.1" class="ltx_text ltx_font_typewriter">+=</span> and <span id="S4.SS3.SSS1.p2.1.2" class="ltx_text ltx_font_typewriter">+</span> in a SGLang program is represented by an IR node. These include <span id="S4.SS3.SSS1.p2.1.3" class="ltx_text ltx_font_typewriter">ConstantText</span>, <span id="S4.SS3.SSS1.p2.1.4" class="ltx_text ltx_font_typewriter">Argument</span>, <span id="S4.SS3.SSS1.p2.1.5" class="ltx_text ltx_font_typewriter">Gen</span>, <span id="S4.SS3.SSS1.p2.1.6" class="ltx_text ltx_font_typewriter">Select</span>, <span id="S4.SS3.SSS1.p2.1.7" class="ltx_text ltx_font_typewriter">Variable</span>, <span id="S4.SS3.SSS1.p2.1.8" class="ltx_text ltx_font_typewriter">Fork</span>, <span id="S4.SS3.SSS1.p2.1.9" class="ltx_text ltx_font_typewriter">GetForkItem</span>, and <span id="S4.SS3.SSS1.p2.1.10" class="ltx_text ltx_font_typewriter">Join</span>.
There are two types of dependencies.
The first is intra-stream dependency, where operations submitted into a stream using <span id="S4.SS3.SSS1.p2.1.11" class="ltx_text ltx_font_typewriter">+=</span> must be executed after all preceding operations in that stream. The second is inter-stream dependency, which occurs when one stream needs to fetch variable values from another stream, necessitating synchronization. Operations like fork manipulate multiple streams and thus introduce inter-stream dependencies.</p>
</div>
<div id="S4.SS3.SSS1.p3" class="ltx_para">
<p id="S4.SS3.SSS1.p3.1" class="ltx_p">To generate the graph, we use tracing to run the program with abstract arguments and construct the graph dynamically.
This method is limited to programs without data-dependent control flow, a limitation we plan to address in future work.
Once constructed, the graph can be executed through a graph executor, eliminating the need to reinterpret the original Python program. This results in benefits like graph rewriting optimizations, reduced runtime overhead, and program serialization.
For execution, stream executors are launched for each data stream, dispatching IR nodes to the streams in topological order.</p>
</div>
</section>
<section id="S4.SS3.SSS2" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">4.3.2 </span>Compilation Optimizations</h4>

<div id="S4.SS3.SSS2.p1" class="ltx_para">
<p id="S4.SS3.SSS2.p1.1" class="ltx_p">We explore two compilation optimizations for SGLang IR. The first one is noteworthy as it integrates GPT-4 into compiler optimization processes. The second one is more conventional.
We anticipate more classical compiler techniques can also be applied, such as auto-tuning and instruction selection.</p>
</div>
<div id="S4.SS3.SSS2.p2" class="ltx_para">
<p id="S4.SS3.SSS2.p2.1" class="ltx_p"><span id="S4.SS3.SSS2.p2.1.1" class="ltx_text ltx_font_bold">Code Movement for Improving Prefix Sharing.</span>
This optimization aims to improve prefix sharing by reordering nodes in the graph to increase the length of the constant prefix. It does not strictly preserve the original computation, classifying it as an aggressive optimization. For instance, changing the prompt from
“Here is a question + {question}. Please act as a math expert and solve the given question” to “Please act as a math expert and solve the given question. Here is a question + {question}.”
results in a longer sharable prefix.
This optimization is interesting because traditional program analysis cannot achieve it, due to the presence of natural language instructions in SGLang. Instead, we prompt GPT-4 to reorder graph nodes. We write a prompt with several examples to teach GPT-4 the concepts of SGLang IR, and we find that GPT-4 can successfully apply this optimization for some simple SGLang programs.</p>
</div>
<div id="S4.SS3.SSS2.p3" class="ltx_para">
<p id="S4.SS3.SSS2.p3.1" class="ltx_p"><span id="S4.SS3.SSS2.p3.1.1" class="ltx_text ltx_font_bold">Prefetching Annotations.</span>
This involves automatically adding prefetching nodes to the graph, signaling the runtime to move the cached KV cache from the CPU to the GPU if the cache exists for a long prompt. This can reduce the latency for pipelines with long prompts.</p>
</div>
<figure id="S4.F6" class="ltx_figure"><img src="/html/2312.07104/assets/x6.png" id="S4.F6.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="368" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S4.F6.2.1.1" class="ltx_text" style="font-size:90%;">Figure 6</span>: </span><span id="S4.F6.3.2" class="ltx_text" style="font-size:90%;">
Examples of RadixAttention operations with an LRU eviction policy, illustrated across nine time points.
The figure demonstrates the dynamic evolution of the radix tree in response to various requests. These requests include two chat sessions, a batch of few-shot learning inquiries, and a self-consistency sampling.
Each tree edge carries a label denoting a substring or a sequence of tokens. The nodes are color-coded to reflect different states: green for newly added nodes, blue for cached nodes accessed during the time point, and red for nodes that have been evicted.
In step (1), the radix tree is initially empty.
In step (2), the server processes an incoming user message "Hello" and responds with the LLM output "Hi". The system prompt "You are a helpful assistant", the user message "Hello!", and the LLM reply "Hi!" are consolidated into the tree as a single edge linked to a new node.
In step (3), a new prompt arrives and the server finds the prefix of the prompt (i.e., the first turn of the conversation) in the radix tree and reuses its KV cache. The new turn is appended to the tree as a new node.
In step (4), a new chat session begins. The node “b” from (3) is split into two nodes to allow the two chat sessions to share the system prompt.
In step (5), the second chat session continues. However, due to the memory limit, node "c" from (4) must be evicted. The new turn is appended after node "d" in (4).
In step (6), the server receives a few-shot learning query, processes it, and inserts it into the tree. The root node is split because the new query does not share any prefix with existing nodes.
In step (7), the server receives a batch of additional few-shot learning queries. These queries share the same set of few-shot examples, so we split node ’e’ from (6) to enable sharing.
In step (8), the server receives a new message from the first chat session. It evicts all nodes from the second chat session (node "g" and "h") as they are least recently used.
In step (9), the server receives a request to sample more answers for the questions in node "j" from (8), likely for self-consistency prompting. To make space for these requests, we evict node "i", "k", and "l" in (8)."
</span></figcaption>
</figure>
</section>
</section>
</section>
<section id="S5" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">5 </span>Runtime with RadixAttention</h2>

<div id="S5.p1" class="ltx_para">
<p id="S5.p1.1" class="ltx_p">The primary challenge in optimizing the execution of SGLang programs lies in reusing the KV cache across multiple calls and programs, as discussed in <a href="#S2.SS3" title="2.3 Challenges and Opportunities ‣ 2 Background ‣ Efficiently Programming Large Language Models using SGLang" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Section 2.3</span></a>. While some existing systems can handle KV cache reuse in certain scenarios, this often requires manual configurations and ad-hoc adjustments. However, even with manual configurations, no existing system can automatically handle all scenarios due to the variety of possible reuse patterns.</p>
</div>
<div id="S5.p2" class="ltx_para">
<p id="S5.p2.1" class="ltx_p">This section introduces RadixAttention, a novel technique for automatic KV cache reuse during runtime. Instead of discarding the KV cache after finishing a generation request, our approach retains the KV cache for both prompts and generation results in a radix tree. This structure enables efficient prefix search, reuse, insertion, and eviction. We implement a Least Recently Used (LRU) eviction policy, complemented by a cache-aware scheduling policy, to enhance the cache hit rate.
Furthermore, RadixAttention is compatible with existing techniques like continuous batching and paged attention.</p>
</div>
<section id="S5.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.1 </span>RadixAttention</h3>

<div id="S5.SS1.p1" class="ltx_para">
<p id="S5.SS1.p1.1" class="ltx_p">A radix tree is a data structure that serves as a space-efficient alternative to a trie (prefix tree). Unlike typical trees, the edges of a radix tree can be labeled with not just single elements, but also with sequences of elements of varying lengths.
This feature significantly boosts the efficiency of radix trees.
In our system, we utilize a radix tree to manage a mapping. This mapping is between sequences of tokens, which act as the keys, and their corresponding KV cache tensors, which serve as the values. These KV cache tensors are stored in a non-contiguous, paged layout, where the size of each page is equivalent to one token.</p>
</div>
<div id="S5.SS1.p2" class="ltx_para">
<p id="S5.SS1.p2.1" class="ltx_p">Considering the limited capacity of GPU memory, we cannot retrain infinite KV cache tensors, which necessitates an eviction policy. To tackle this, we implement an LRU eviction policy that recursively evicts leaf nodes.
In the continuous batching setting, we cannot evict nodes that are being used by the currently running batch. Therefore, each node maintains a reference counter indicating how many running requests are using it. A node is evictable if its reference counter is zero.</p>
</div>
<div id="S5.SS1.p3" class="ltx_para">
<p id="S5.SS1.p3.1" class="ltx_p">The front end always sends full prompts to the runtime and the runtime will automatically do prefix matching, reuse, and caching. <a href="#S4.F6" title="Figure 6 ‣ 4.3.2 Compilation Optimizations ‣ 4.3 Compiler ‣ 4 Language, Interpreter, and Compiler ‣ Efficiently Programming Large Language Models using SGLang" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 6</span></a> illustrates how the radix tree is maintained when processing several incoming requests.
The tree structure is stored on the CPU and the maintenance overhead is small.</p>
</div>
</section>
<section id="S5.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.2 </span>Cache-Aware Scheduling</h3>

<div id="S5.SS2.p1" class="ltx_para">
<p id="S5.SS2.p1.1" class="ltx_p">To increase the cache hit rate, the scheduler must be cache-aware. Otherwise, when a large batch of requests arrives, the scheduler may switch between different requests, leading to cache thrashing. <a href="#alg1" title="Algorithm 1 ‣ 5.2 Cache-Aware Scheduling ‣ 5 Runtime with RadixAttention ‣ Efficiently Programming Large Language Models using SGLang" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Algorithm 1</span></a> presents the algorithm for cache-aware scheduling in RadixAttention with contiguous batching. The key idea is to sort the requests by matched prefix length, as opposed to adhering to a first-come-first-serve schedule.
There are further opportunities for the implementation of more advanced scheduling strategies that assign priority based on the weights of subtrees. Essentially, this would involve counting the total number of requests within each sub-tree(prefix-sharing trees) and even considering requests that have been historically evicted. The weighted priority can fully utilize the tree-shape locality of some LLM tasks such as tree-of-thought <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib70" title="" class="ltx_ref">70</a>]</cite>, making the least redundant evictions.</p>
</div>
<figure id="alg1" class="ltx_float ltx_float_algorithm ltx_framed ltx_framed_top">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_float"><span id="alg1.2.1.1" class="ltx_text ltx_font_bold">Algorithm 1</span> </span> Cache-Aware Scheduling for RadixAttention with Continuous Batching.</figcaption>
<div id="alg1.3" class="ltx_listing ltx_listing">
<div id="alg1.l1" class="ltx_listingline">The radix tree <math id="alg1.l1.m1.1" class="ltx_Math" alttext="T" display="inline"><semantics id="alg1.l1.m1.1a"><mi id="alg1.l1.m1.1.1" xref="alg1.l1.m1.1.1.cmml">T</mi><annotation-xml encoding="MathML-Content" id="alg1.l1.m1.1b"><ci id="alg1.l1.m1.1.1.cmml" xref="alg1.l1.m1.1.1">𝑇</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l1.m1.1c">T</annotation></semantics></math>, the memory pool <math id="alg1.l1.m2.1" class="ltx_Math" alttext="P" display="inline"><semantics id="alg1.l1.m2.1a"><mi id="alg1.l1.m2.1.1" xref="alg1.l1.m2.1.1.cmml">P</mi><annotation-xml encoding="MathML-Content" id="alg1.l1.m2.1b"><ci id="alg1.l1.m2.1.1.cmml" xref="alg1.l1.m2.1.1">𝑃</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l1.m2.1c">P</annotation></semantics></math>, the current running batch <math id="alg1.l1.m3.1" class="ltx_Math" alttext="B" display="inline"><semantics id="alg1.l1.m3.1a"><mi id="alg1.l1.m3.1.1" xref="alg1.l1.m3.1.1.cmml">B</mi><annotation-xml encoding="MathML-Content" id="alg1.l1.m3.1b"><ci id="alg1.l1.m3.1.1.cmml" xref="alg1.l1.m3.1.1">𝐵</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l1.m3.1c">B</annotation></semantics></math>, the waiting queue <math id="alg1.l1.m4.1" class="ltx_Math" alttext="Q" display="inline"><semantics id="alg1.l1.m4.1a"><mi id="alg1.l1.m4.1.1" xref="alg1.l1.m4.1.1.cmml">Q</mi><annotation-xml encoding="MathML-Content" id="alg1.l1.m4.1b"><ci id="alg1.l1.m4.1.1.cmml" xref="alg1.l1.m4.1.1">𝑄</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l1.m4.1c">Q</annotation></semantics></math>.

</div>
<div id="alg1.l2" class="ltx_listingline">Finished requests and updated system state.

</div>
<div id="alg1.l3" class="ltx_listingline">
<span id="alg1.l3.1" class="ltx_text">// Get all requests from the waiting queue</span>

</div>
<div id="alg1.l4" class="ltx_listingline">
<math id="alg1.l4.m1.2" class="ltx_Math" alttext="requests\leftarrow Q.\text{get\_all\_requests}()" display="inline"><semantics id="alg1.l4.m1.2a"><mrow id="alg1.l4.m1.2.2.2" xref="alg1.l4.m1.2.2.3.cmml"><mrow id="alg1.l4.m1.1.1.1.1" xref="alg1.l4.m1.1.1.1.1.cmml"><mrow id="alg1.l4.m1.1.1.1.1.2" xref="alg1.l4.m1.1.1.1.1.2.cmml"><mi id="alg1.l4.m1.1.1.1.1.2.2" xref="alg1.l4.m1.1.1.1.1.2.2.cmml">r</mi><mo lspace="0em" rspace="0em" id="alg1.l4.m1.1.1.1.1.2.1" xref="alg1.l4.m1.1.1.1.1.2.1.cmml">​</mo><mi id="alg1.l4.m1.1.1.1.1.2.3" xref="alg1.l4.m1.1.1.1.1.2.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l4.m1.1.1.1.1.2.1a" xref="alg1.l4.m1.1.1.1.1.2.1.cmml">​</mo><mi id="alg1.l4.m1.1.1.1.1.2.4" xref="alg1.l4.m1.1.1.1.1.2.4.cmml">q</mi><mo lspace="0em" rspace="0em" id="alg1.l4.m1.1.1.1.1.2.1b" xref="alg1.l4.m1.1.1.1.1.2.1.cmml">​</mo><mi id="alg1.l4.m1.1.1.1.1.2.5" xref="alg1.l4.m1.1.1.1.1.2.5.cmml">u</mi><mo lspace="0em" rspace="0em" id="alg1.l4.m1.1.1.1.1.2.1c" xref="alg1.l4.m1.1.1.1.1.2.1.cmml">​</mo><mi id="alg1.l4.m1.1.1.1.1.2.6" xref="alg1.l4.m1.1.1.1.1.2.6.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l4.m1.1.1.1.1.2.1d" xref="alg1.l4.m1.1.1.1.1.2.1.cmml">​</mo><mi id="alg1.l4.m1.1.1.1.1.2.7" xref="alg1.l4.m1.1.1.1.1.2.7.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg1.l4.m1.1.1.1.1.2.1e" xref="alg1.l4.m1.1.1.1.1.2.1.cmml">​</mo><mi id="alg1.l4.m1.1.1.1.1.2.8" xref="alg1.l4.m1.1.1.1.1.2.8.cmml">t</mi><mo lspace="0em" rspace="0em" id="alg1.l4.m1.1.1.1.1.2.1f" xref="alg1.l4.m1.1.1.1.1.2.1.cmml">​</mo><mi id="alg1.l4.m1.1.1.1.1.2.9" xref="alg1.l4.m1.1.1.1.1.2.9.cmml">s</mi></mrow><mo stretchy="false" id="alg1.l4.m1.1.1.1.1.1" xref="alg1.l4.m1.1.1.1.1.1.cmml">←</mo><mi id="alg1.l4.m1.1.1.1.1.3" xref="alg1.l4.m1.1.1.1.1.3.cmml">Q</mi></mrow><mo lspace="0em" rspace="0.167em" id="alg1.l4.m1.2.2.2.3" xref="alg1.l4.m1.2.2.3a.cmml">.</mo><mrow id="alg1.l4.m1.2.2.2.2" xref="alg1.l4.m1.2.2.2.2.cmml"><mtext id="alg1.l4.m1.2.2.2.2.2" xref="alg1.l4.m1.2.2.2.2.2a.cmml">get_all_requests</mtext><mo lspace="0em" rspace="0em" id="alg1.l4.m1.2.2.2.2.1" xref="alg1.l4.m1.2.2.2.2.1.cmml">​</mo><mrow id="alg1.l4.m1.2.2.2.2.3.2" xref="alg1.l4.m1.2.2.2.2.cmml"><mo stretchy="false" id="alg1.l4.m1.2.2.2.2.3.2.1" xref="alg1.l4.m1.2.2.2.2.3.1.cmml">(</mo><mo stretchy="false" id="alg1.l4.m1.2.2.2.2.3.2.2" xref="alg1.l4.m1.2.2.2.2.3.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l4.m1.2b"><apply id="alg1.l4.m1.2.2.3.cmml" xref="alg1.l4.m1.2.2.2"><csymbol cd="ambiguous" id="alg1.l4.m1.2.2.3a.cmml" xref="alg1.l4.m1.2.2.2.3">formulae-sequence</csymbol><apply id="alg1.l4.m1.1.1.1.1.cmml" xref="alg1.l4.m1.1.1.1.1"><ci id="alg1.l4.m1.1.1.1.1.1.cmml" xref="alg1.l4.m1.1.1.1.1.1">←</ci><apply id="alg1.l4.m1.1.1.1.1.2.cmml" xref="alg1.l4.m1.1.1.1.1.2"><times id="alg1.l4.m1.1.1.1.1.2.1.cmml" xref="alg1.l4.m1.1.1.1.1.2.1"></times><ci id="alg1.l4.m1.1.1.1.1.2.2.cmml" xref="alg1.l4.m1.1.1.1.1.2.2">𝑟</ci><ci id="alg1.l4.m1.1.1.1.1.2.3.cmml" xref="alg1.l4.m1.1.1.1.1.2.3">𝑒</ci><ci id="alg1.l4.m1.1.1.1.1.2.4.cmml" xref="alg1.l4.m1.1.1.1.1.2.4">𝑞</ci><ci id="alg1.l4.m1.1.1.1.1.2.5.cmml" xref="alg1.l4.m1.1.1.1.1.2.5">𝑢</ci><ci id="alg1.l4.m1.1.1.1.1.2.6.cmml" xref="alg1.l4.m1.1.1.1.1.2.6">𝑒</ci><ci id="alg1.l4.m1.1.1.1.1.2.7.cmml" xref="alg1.l4.m1.1.1.1.1.2.7">𝑠</ci><ci id="alg1.l4.m1.1.1.1.1.2.8.cmml" xref="alg1.l4.m1.1.1.1.1.2.8">𝑡</ci><ci id="alg1.l4.m1.1.1.1.1.2.9.cmml" xref="alg1.l4.m1.1.1.1.1.2.9">𝑠</ci></apply><ci id="alg1.l4.m1.1.1.1.1.3.cmml" xref="alg1.l4.m1.1.1.1.1.3">𝑄</ci></apply><apply id="alg1.l4.m1.2.2.2.2.cmml" xref="alg1.l4.m1.2.2.2.2"><times id="alg1.l4.m1.2.2.2.2.1.cmml" xref="alg1.l4.m1.2.2.2.2.1"></times><ci id="alg1.l4.m1.2.2.2.2.2a.cmml" xref="alg1.l4.m1.2.2.2.2.2"><mtext id="alg1.l4.m1.2.2.2.2.2.cmml" xref="alg1.l4.m1.2.2.2.2.2">get_all_requests</mtext></ci><list id="alg1.l4.m1.2.2.2.2.3.1.cmml" xref="alg1.l4.m1.2.2.2.2.3.2.1"></list></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l4.m1.2c">requests\leftarrow Q.\text{get\_all\_requests}()</annotation></semantics></math>

</div>
<div id="alg1.l5" class="ltx_listingline">
<span id="alg1.l5.1" class="ltx_text">// Search for the prefix matching for all waiting requests</span>

</div>
<div id="alg1.l6" class="ltx_listingline">
<span id="alg1.l6.1" class="ltx_text ltx_font_bold">for</span> <math id="alg1.l6.m1.1" class="ltx_Math" alttext="req\in requests" display="inline"><semantics id="alg1.l6.m1.1a"><mrow id="alg1.l6.m1.1.1" xref="alg1.l6.m1.1.1.cmml"><mrow id="alg1.l6.m1.1.1.2" xref="alg1.l6.m1.1.1.2.cmml"><mi id="alg1.l6.m1.1.1.2.2" xref="alg1.l6.m1.1.1.2.2.cmml">r</mi><mo lspace="0em" rspace="0em" id="alg1.l6.m1.1.1.2.1" xref="alg1.l6.m1.1.1.2.1.cmml">​</mo><mi id="alg1.l6.m1.1.1.2.3" xref="alg1.l6.m1.1.1.2.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l6.m1.1.1.2.1a" xref="alg1.l6.m1.1.1.2.1.cmml">​</mo><mi id="alg1.l6.m1.1.1.2.4" xref="alg1.l6.m1.1.1.2.4.cmml">q</mi></mrow><mo id="alg1.l6.m1.1.1.1" xref="alg1.l6.m1.1.1.1.cmml">∈</mo><mrow id="alg1.l6.m1.1.1.3" xref="alg1.l6.m1.1.1.3.cmml"><mi id="alg1.l6.m1.1.1.3.2" xref="alg1.l6.m1.1.1.3.2.cmml">r</mi><mo lspace="0em" rspace="0em" id="alg1.l6.m1.1.1.3.1" xref="alg1.l6.m1.1.1.3.1.cmml">​</mo><mi id="alg1.l6.m1.1.1.3.3" xref="alg1.l6.m1.1.1.3.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l6.m1.1.1.3.1a" xref="alg1.l6.m1.1.1.3.1.cmml">​</mo><mi id="alg1.l6.m1.1.1.3.4" xref="alg1.l6.m1.1.1.3.4.cmml">q</mi><mo lspace="0em" rspace="0em" id="alg1.l6.m1.1.1.3.1b" xref="alg1.l6.m1.1.1.3.1.cmml">​</mo><mi id="alg1.l6.m1.1.1.3.5" xref="alg1.l6.m1.1.1.3.5.cmml">u</mi><mo lspace="0em" rspace="0em" id="alg1.l6.m1.1.1.3.1c" xref="alg1.l6.m1.1.1.3.1.cmml">​</mo><mi id="alg1.l6.m1.1.1.3.6" xref="alg1.l6.m1.1.1.3.6.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l6.m1.1.1.3.1d" xref="alg1.l6.m1.1.1.3.1.cmml">​</mo><mi id="alg1.l6.m1.1.1.3.7" xref="alg1.l6.m1.1.1.3.7.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg1.l6.m1.1.1.3.1e" xref="alg1.l6.m1.1.1.3.1.cmml">​</mo><mi id="alg1.l6.m1.1.1.3.8" xref="alg1.l6.m1.1.1.3.8.cmml">t</mi><mo lspace="0em" rspace="0em" id="alg1.l6.m1.1.1.3.1f" xref="alg1.l6.m1.1.1.3.1.cmml">​</mo><mi id="alg1.l6.m1.1.1.3.9" xref="alg1.l6.m1.1.1.3.9.cmml">s</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l6.m1.1b"><apply id="alg1.l6.m1.1.1.cmml" xref="alg1.l6.m1.1.1"><in id="alg1.l6.m1.1.1.1.cmml" xref="alg1.l6.m1.1.1.1"></in><apply id="alg1.l6.m1.1.1.2.cmml" xref="alg1.l6.m1.1.1.2"><times id="alg1.l6.m1.1.1.2.1.cmml" xref="alg1.l6.m1.1.1.2.1"></times><ci id="alg1.l6.m1.1.1.2.2.cmml" xref="alg1.l6.m1.1.1.2.2">𝑟</ci><ci id="alg1.l6.m1.1.1.2.3.cmml" xref="alg1.l6.m1.1.1.2.3">𝑒</ci><ci id="alg1.l6.m1.1.1.2.4.cmml" xref="alg1.l6.m1.1.1.2.4">𝑞</ci></apply><apply id="alg1.l6.m1.1.1.3.cmml" xref="alg1.l6.m1.1.1.3"><times id="alg1.l6.m1.1.1.3.1.cmml" xref="alg1.l6.m1.1.1.3.1"></times><ci id="alg1.l6.m1.1.1.3.2.cmml" xref="alg1.l6.m1.1.1.3.2">𝑟</ci><ci id="alg1.l6.m1.1.1.3.3.cmml" xref="alg1.l6.m1.1.1.3.3">𝑒</ci><ci id="alg1.l6.m1.1.1.3.4.cmml" xref="alg1.l6.m1.1.1.3.4">𝑞</ci><ci id="alg1.l6.m1.1.1.3.5.cmml" xref="alg1.l6.m1.1.1.3.5">𝑢</ci><ci id="alg1.l6.m1.1.1.3.6.cmml" xref="alg1.l6.m1.1.1.3.6">𝑒</ci><ci id="alg1.l6.m1.1.1.3.7.cmml" xref="alg1.l6.m1.1.1.3.7">𝑠</ci><ci id="alg1.l6.m1.1.1.3.8.cmml" xref="alg1.l6.m1.1.1.3.8">𝑡</ci><ci id="alg1.l6.m1.1.1.3.9.cmml" xref="alg1.l6.m1.1.1.3.9">𝑠</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l6.m1.1c">req\in requests</annotation></semantics></math> <span id="alg1.l6.2" class="ltx_text ltx_font_bold">do</span>

</div>
<div id="alg1.l7" class="ltx_listingline">     <math id="alg1.l7.m1.3" class="ltx_Math" alttext="req.prefix\_node,req.prefix\_len\leftarrow" display="inline"><semantics id="alg1.l7.m1.3a"><mrow id="alg1.l7.m1.3.3.3" xref="alg1.l7.m1.3.3.4.cmml"><mrow id="alg1.l7.m1.1.1.1.1" xref="alg1.l7.m1.1.1.1.1.cmml"><mi id="alg1.l7.m1.1.1.1.1.2" xref="alg1.l7.m1.1.1.1.1.2.cmml">r</mi><mo lspace="0em" rspace="0em" id="alg1.l7.m1.1.1.1.1.1" xref="alg1.l7.m1.1.1.1.1.1.cmml">​</mo><mi id="alg1.l7.m1.1.1.1.1.3" xref="alg1.l7.m1.1.1.1.1.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l7.m1.1.1.1.1.1a" xref="alg1.l7.m1.1.1.1.1.1.cmml">​</mo><mi id="alg1.l7.m1.1.1.1.1.4" xref="alg1.l7.m1.1.1.1.1.4.cmml">q</mi></mrow><mo lspace="0em" rspace="0.167em" id="alg1.l7.m1.3.3.3.4" xref="alg1.l7.m1.3.3.4a.cmml">.</mo><mrow id="alg1.l7.m1.2.2.2.2.2" xref="alg1.l7.m1.2.2.2.2.3.cmml"><mrow id="alg1.l7.m1.2.2.2.2.1.1" xref="alg1.l7.m1.2.2.2.2.1.1.cmml"><mi id="alg1.l7.m1.2.2.2.2.1.1.2" xref="alg1.l7.m1.2.2.2.2.1.1.2.cmml">p</mi><mo lspace="0em" rspace="0em" id="alg1.l7.m1.2.2.2.2.1.1.1" xref="alg1.l7.m1.2.2.2.2.1.1.1.cmml">​</mo><mi id="alg1.l7.m1.2.2.2.2.1.1.3" xref="alg1.l7.m1.2.2.2.2.1.1.3.cmml">r</mi><mo lspace="0em" rspace="0em" id="alg1.l7.m1.2.2.2.2.1.1.1a" xref="alg1.l7.m1.2.2.2.2.1.1.1.cmml">​</mo><mi id="alg1.l7.m1.2.2.2.2.1.1.4" xref="alg1.l7.m1.2.2.2.2.1.1.4.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l7.m1.2.2.2.2.1.1.1b" xref="alg1.l7.m1.2.2.2.2.1.1.1.cmml">​</mo><mi id="alg1.l7.m1.2.2.2.2.1.1.5" xref="alg1.l7.m1.2.2.2.2.1.1.5.cmml">f</mi><mo lspace="0em" rspace="0em" id="alg1.l7.m1.2.2.2.2.1.1.1c" xref="alg1.l7.m1.2.2.2.2.1.1.1.cmml">​</mo><mi id="alg1.l7.m1.2.2.2.2.1.1.6" xref="alg1.l7.m1.2.2.2.2.1.1.6.cmml">i</mi><mo lspace="0em" rspace="0em" id="alg1.l7.m1.2.2.2.2.1.1.1d" xref="alg1.l7.m1.2.2.2.2.1.1.1.cmml">​</mo><mi id="alg1.l7.m1.2.2.2.2.1.1.7" xref="alg1.l7.m1.2.2.2.2.1.1.7.cmml">x</mi><mo lspace="0em" rspace="0em" id="alg1.l7.m1.2.2.2.2.1.1.1e" xref="alg1.l7.m1.2.2.2.2.1.1.1.cmml">​</mo><mi mathvariant="normal" id="alg1.l7.m1.2.2.2.2.1.1.8" xref="alg1.l7.m1.2.2.2.2.1.1.8.cmml">_</mi><mo lspace="0em" rspace="0em" id="alg1.l7.m1.2.2.2.2.1.1.1f" xref="alg1.l7.m1.2.2.2.2.1.1.1.cmml">​</mo><mi id="alg1.l7.m1.2.2.2.2.1.1.9" xref="alg1.l7.m1.2.2.2.2.1.1.9.cmml">n</mi><mo lspace="0em" rspace="0em" id="alg1.l7.m1.2.2.2.2.1.1.1g" xref="alg1.l7.m1.2.2.2.2.1.1.1.cmml">​</mo><mi id="alg1.l7.m1.2.2.2.2.1.1.10" xref="alg1.l7.m1.2.2.2.2.1.1.10.cmml">o</mi><mo lspace="0em" rspace="0em" id="alg1.l7.m1.2.2.2.2.1.1.1h" xref="alg1.l7.m1.2.2.2.2.1.1.1.cmml">​</mo><mi id="alg1.l7.m1.2.2.2.2.1.1.11" xref="alg1.l7.m1.2.2.2.2.1.1.11.cmml">d</mi><mo lspace="0em" rspace="0em" id="alg1.l7.m1.2.2.2.2.1.1.1i" xref="alg1.l7.m1.2.2.2.2.1.1.1.cmml">​</mo><mi id="alg1.l7.m1.2.2.2.2.1.1.12" xref="alg1.l7.m1.2.2.2.2.1.1.12.cmml">e</mi></mrow><mo id="alg1.l7.m1.2.2.2.2.2.3" xref="alg1.l7.m1.2.2.2.2.3.cmml">,</mo><mrow id="alg1.l7.m1.2.2.2.2.2.2" xref="alg1.l7.m1.2.2.2.2.2.2.cmml"><mi id="alg1.l7.m1.2.2.2.2.2.2.2" xref="alg1.l7.m1.2.2.2.2.2.2.2.cmml">r</mi><mo lspace="0em" rspace="0em" id="alg1.l7.m1.2.2.2.2.2.2.1" xref="alg1.l7.m1.2.2.2.2.2.2.1.cmml">​</mo><mi id="alg1.l7.m1.2.2.2.2.2.2.3" xref="alg1.l7.m1.2.2.2.2.2.2.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l7.m1.2.2.2.2.2.2.1a" xref="alg1.l7.m1.2.2.2.2.2.2.1.cmml">​</mo><mi id="alg1.l7.m1.2.2.2.2.2.2.4" xref="alg1.l7.m1.2.2.2.2.2.2.4.cmml">q</mi></mrow></mrow><mo lspace="0em" rspace="0.167em" id="alg1.l7.m1.3.3.3.5" xref="alg1.l7.m1.3.3.4a.cmml">.</mo><mrow id="alg1.l7.m1.3.3.3.3" xref="alg1.l7.m1.3.3.3.3.cmml"><mrow id="alg1.l7.m1.3.3.3.3.2" xref="alg1.l7.m1.3.3.3.3.2.cmml"><mi id="alg1.l7.m1.3.3.3.3.2.2" xref="alg1.l7.m1.3.3.3.3.2.2.cmml">p</mi><mo lspace="0em" rspace="0em" id="alg1.l7.m1.3.3.3.3.2.1" xref="alg1.l7.m1.3.3.3.3.2.1.cmml">​</mo><mi id="alg1.l7.m1.3.3.3.3.2.3" xref="alg1.l7.m1.3.3.3.3.2.3.cmml">r</mi><mo lspace="0em" rspace="0em" id="alg1.l7.m1.3.3.3.3.2.1a" xref="alg1.l7.m1.3.3.3.3.2.1.cmml">​</mo><mi id="alg1.l7.m1.3.3.3.3.2.4" xref="alg1.l7.m1.3.3.3.3.2.4.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l7.m1.3.3.3.3.2.1b" xref="alg1.l7.m1.3.3.3.3.2.1.cmml">​</mo><mi id="alg1.l7.m1.3.3.3.3.2.5" xref="alg1.l7.m1.3.3.3.3.2.5.cmml">f</mi><mo lspace="0em" rspace="0em" id="alg1.l7.m1.3.3.3.3.2.1c" xref="alg1.l7.m1.3.3.3.3.2.1.cmml">​</mo><mi id="alg1.l7.m1.3.3.3.3.2.6" xref="alg1.l7.m1.3.3.3.3.2.6.cmml">i</mi><mo lspace="0em" rspace="0em" id="alg1.l7.m1.3.3.3.3.2.1d" xref="alg1.l7.m1.3.3.3.3.2.1.cmml">​</mo><mi id="alg1.l7.m1.3.3.3.3.2.7" xref="alg1.l7.m1.3.3.3.3.2.7.cmml">x</mi><mo lspace="0em" rspace="0em" id="alg1.l7.m1.3.3.3.3.2.1e" xref="alg1.l7.m1.3.3.3.3.2.1.cmml">​</mo><mi mathvariant="normal" id="alg1.l7.m1.3.3.3.3.2.8" xref="alg1.l7.m1.3.3.3.3.2.8.cmml">_</mi><mo lspace="0em" rspace="0em" id="alg1.l7.m1.3.3.3.3.2.1f" xref="alg1.l7.m1.3.3.3.3.2.1.cmml">​</mo><mi id="alg1.l7.m1.3.3.3.3.2.9" xref="alg1.l7.m1.3.3.3.3.2.9.cmml">l</mi><mo lspace="0em" rspace="0em" id="alg1.l7.m1.3.3.3.3.2.1g" xref="alg1.l7.m1.3.3.3.3.2.1.cmml">​</mo><mi id="alg1.l7.m1.3.3.3.3.2.10" xref="alg1.l7.m1.3.3.3.3.2.10.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l7.m1.3.3.3.3.2.1h" xref="alg1.l7.m1.3.3.3.3.2.1.cmml">​</mo><mi id="alg1.l7.m1.3.3.3.3.2.11" xref="alg1.l7.m1.3.3.3.3.2.11.cmml">n</mi></mrow><mo stretchy="false" id="alg1.l7.m1.3.3.3.3.1" xref="alg1.l7.m1.3.3.3.3.1.cmml">←</mo><mi id="alg1.l7.m1.3.3.3.3.3" xref="alg1.l7.m1.3.3.3.3.3.cmml"></mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l7.m1.3b"><apply id="alg1.l7.m1.3.3.4.cmml" xref="alg1.l7.m1.3.3.3"><csymbol cd="ambiguous" id="alg1.l7.m1.3.3.4a.cmml" xref="alg1.l7.m1.3.3.3.4">formulae-sequence</csymbol><apply id="alg1.l7.m1.1.1.1.1.cmml" xref="alg1.l7.m1.1.1.1.1"><times id="alg1.l7.m1.1.1.1.1.1.cmml" xref="alg1.l7.m1.1.1.1.1.1"></times><ci id="alg1.l7.m1.1.1.1.1.2.cmml" xref="alg1.l7.m1.1.1.1.1.2">𝑟</ci><ci id="alg1.l7.m1.1.1.1.1.3.cmml" xref="alg1.l7.m1.1.1.1.1.3">𝑒</ci><ci id="alg1.l7.m1.1.1.1.1.4.cmml" xref="alg1.l7.m1.1.1.1.1.4">𝑞</ci></apply><list id="alg1.l7.m1.2.2.2.2.3.cmml" xref="alg1.l7.m1.2.2.2.2.2"><apply id="alg1.l7.m1.2.2.2.2.1.1.cmml" xref="alg1.l7.m1.2.2.2.2.1.1"><times id="alg1.l7.m1.2.2.2.2.1.1.1.cmml" xref="alg1.l7.m1.2.2.2.2.1.1.1"></times><ci id="alg1.l7.m1.2.2.2.2.1.1.2.cmml" xref="alg1.l7.m1.2.2.2.2.1.1.2">𝑝</ci><ci id="alg1.l7.m1.2.2.2.2.1.1.3.cmml" xref="alg1.l7.m1.2.2.2.2.1.1.3">𝑟</ci><ci id="alg1.l7.m1.2.2.2.2.1.1.4.cmml" xref="alg1.l7.m1.2.2.2.2.1.1.4">𝑒</ci><ci id="alg1.l7.m1.2.2.2.2.1.1.5.cmml" xref="alg1.l7.m1.2.2.2.2.1.1.5">𝑓</ci><ci id="alg1.l7.m1.2.2.2.2.1.1.6.cmml" xref="alg1.l7.m1.2.2.2.2.1.1.6">𝑖</ci><ci id="alg1.l7.m1.2.2.2.2.1.1.7.cmml" xref="alg1.l7.m1.2.2.2.2.1.1.7">𝑥</ci><ci id="alg1.l7.m1.2.2.2.2.1.1.8.cmml" xref="alg1.l7.m1.2.2.2.2.1.1.8">_</ci><ci id="alg1.l7.m1.2.2.2.2.1.1.9.cmml" xref="alg1.l7.m1.2.2.2.2.1.1.9">𝑛</ci><ci id="alg1.l7.m1.2.2.2.2.1.1.10.cmml" xref="alg1.l7.m1.2.2.2.2.1.1.10">𝑜</ci><ci id="alg1.l7.m1.2.2.2.2.1.1.11.cmml" xref="alg1.l7.m1.2.2.2.2.1.1.11">𝑑</ci><ci id="alg1.l7.m1.2.2.2.2.1.1.12.cmml" xref="alg1.l7.m1.2.2.2.2.1.1.12">𝑒</ci></apply><apply id="alg1.l7.m1.2.2.2.2.2.2.cmml" xref="alg1.l7.m1.2.2.2.2.2.2"><times id="alg1.l7.m1.2.2.2.2.2.2.1.cmml" xref="alg1.l7.m1.2.2.2.2.2.2.1"></times><ci id="alg1.l7.m1.2.2.2.2.2.2.2.cmml" xref="alg1.l7.m1.2.2.2.2.2.2.2">𝑟</ci><ci id="alg1.l7.m1.2.2.2.2.2.2.3.cmml" xref="alg1.l7.m1.2.2.2.2.2.2.3">𝑒</ci><ci id="alg1.l7.m1.2.2.2.2.2.2.4.cmml" xref="alg1.l7.m1.2.2.2.2.2.2.4">𝑞</ci></apply></list><apply id="alg1.l7.m1.3.3.3.3.cmml" xref="alg1.l7.m1.3.3.3.3"><ci id="alg1.l7.m1.3.3.3.3.1.cmml" xref="alg1.l7.m1.3.3.3.3.1">←</ci><apply id="alg1.l7.m1.3.3.3.3.2.cmml" xref="alg1.l7.m1.3.3.3.3.2"><times id="alg1.l7.m1.3.3.3.3.2.1.cmml" xref="alg1.l7.m1.3.3.3.3.2.1"></times><ci id="alg1.l7.m1.3.3.3.3.2.2.cmml" xref="alg1.l7.m1.3.3.3.3.2.2">𝑝</ci><ci id="alg1.l7.m1.3.3.3.3.2.3.cmml" xref="alg1.l7.m1.3.3.3.3.2.3">𝑟</ci><ci id="alg1.l7.m1.3.3.3.3.2.4.cmml" xref="alg1.l7.m1.3.3.3.3.2.4">𝑒</ci><ci id="alg1.l7.m1.3.3.3.3.2.5.cmml" xref="alg1.l7.m1.3.3.3.3.2.5">𝑓</ci><ci id="alg1.l7.m1.3.3.3.3.2.6.cmml" xref="alg1.l7.m1.3.3.3.3.2.6">𝑖</ci><ci id="alg1.l7.m1.3.3.3.3.2.7.cmml" xref="alg1.l7.m1.3.3.3.3.2.7">𝑥</ci><ci id="alg1.l7.m1.3.3.3.3.2.8.cmml" xref="alg1.l7.m1.3.3.3.3.2.8">_</ci><ci id="alg1.l7.m1.3.3.3.3.2.9.cmml" xref="alg1.l7.m1.3.3.3.3.2.9">𝑙</ci><ci id="alg1.l7.m1.3.3.3.3.2.10.cmml" xref="alg1.l7.m1.3.3.3.3.2.10">𝑒</ci><ci id="alg1.l7.m1.3.3.3.3.2.11.cmml" xref="alg1.l7.m1.3.3.3.3.2.11">𝑛</ci></apply><csymbol cd="latexml" id="alg1.l7.m1.3.3.3.3.3.cmml" xref="alg1.l7.m1.3.3.3.3.3">absent</csymbol></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l7.m1.3c">req.prefix\_node,req.prefix\_len\leftarrow</annotation></semantics></math>

</div>
<div id="alg1.l8" class="ltx_listingline">                 <math id="alg1.l8.m1.2" class="ltx_math_unparsed" alttext="T.\text{match\_prefix}(req.input\_tokens)" display="inline"><semantics id="alg1.l8.m1.2a"><mrow id="alg1.l8.m1.2b"><mi id="alg1.l8.m1.1.1">T</mi><mo lspace="0em" rspace="0.167em" id="alg1.l8.m1.2.3">.</mo><mtext id="alg1.l8.m1.2.2">match_prefix</mtext><mrow id="alg1.l8.m1.2.4"><mo stretchy="false" id="alg1.l8.m1.2.4.1">(</mo><mi id="alg1.l8.m1.2.4.2">r</mi><mi id="alg1.l8.m1.2.4.3">e</mi><mi id="alg1.l8.m1.2.4.4">q</mi><mo lspace="0em" rspace="0.167em" id="alg1.l8.m1.2.4.5">.</mo><mi id="alg1.l8.m1.2.4.6">i</mi><mi id="alg1.l8.m1.2.4.7">n</mi><mi id="alg1.l8.m1.2.4.8">p</mi><mi id="alg1.l8.m1.2.4.9">u</mi><mi id="alg1.l8.m1.2.4.10">t</mi><mi mathvariant="normal" id="alg1.l8.m1.2.4.11">_</mi><mi id="alg1.l8.m1.2.4.12">t</mi><mi id="alg1.l8.m1.2.4.13">o</mi><mi id="alg1.l8.m1.2.4.14">k</mi><mi id="alg1.l8.m1.2.4.15">e</mi><mi id="alg1.l8.m1.2.4.16">n</mi><mi id="alg1.l8.m1.2.4.17">s</mi><mo stretchy="false" id="alg1.l8.m1.2.4.18">)</mo></mrow></mrow><annotation encoding="application/x-tex" id="alg1.l8.m1.2c">T.\text{match\_prefix}(req.input\_tokens)</annotation></semantics></math>

</div>
<div id="alg1.l9" class="ltx_listingline">
<span id="alg1.l9.1" class="ltx_text">// Sort the requests according to the matched prefix lenghts</span>

</div>
<div id="alg1.l10" class="ltx_listingline">
<math id="alg1.l10.m1.2" class="ltx_Math" alttext="requests.\text{sort}()" display="inline"><semantics id="alg1.l10.m1.2a"><mrow id="alg1.l10.m1.2.2.2" xref="alg1.l10.m1.2.2.3.cmml"><mrow id="alg1.l10.m1.1.1.1.1" xref="alg1.l10.m1.1.1.1.1.cmml"><mi id="alg1.l10.m1.1.1.1.1.2" xref="alg1.l10.m1.1.1.1.1.2.cmml">r</mi><mo lspace="0em" rspace="0em" id="alg1.l10.m1.1.1.1.1.1" xref="alg1.l10.m1.1.1.1.1.1.cmml">​</mo><mi id="alg1.l10.m1.1.1.1.1.3" xref="alg1.l10.m1.1.1.1.1.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l10.m1.1.1.1.1.1a" xref="alg1.l10.m1.1.1.1.1.1.cmml">​</mo><mi id="alg1.l10.m1.1.1.1.1.4" xref="alg1.l10.m1.1.1.1.1.4.cmml">q</mi><mo lspace="0em" rspace="0em" id="alg1.l10.m1.1.1.1.1.1b" xref="alg1.l10.m1.1.1.1.1.1.cmml">​</mo><mi id="alg1.l10.m1.1.1.1.1.5" xref="alg1.l10.m1.1.1.1.1.5.cmml">u</mi><mo lspace="0em" rspace="0em" id="alg1.l10.m1.1.1.1.1.1c" xref="alg1.l10.m1.1.1.1.1.1.cmml">​</mo><mi id="alg1.l10.m1.1.1.1.1.6" xref="alg1.l10.m1.1.1.1.1.6.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l10.m1.1.1.1.1.1d" xref="alg1.l10.m1.1.1.1.1.1.cmml">​</mo><mi id="alg1.l10.m1.1.1.1.1.7" xref="alg1.l10.m1.1.1.1.1.7.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg1.l10.m1.1.1.1.1.1e" xref="alg1.l10.m1.1.1.1.1.1.cmml">​</mo><mi id="alg1.l10.m1.1.1.1.1.8" xref="alg1.l10.m1.1.1.1.1.8.cmml">t</mi><mo lspace="0em" rspace="0em" id="alg1.l10.m1.1.1.1.1.1f" xref="alg1.l10.m1.1.1.1.1.1.cmml">​</mo><mi id="alg1.l10.m1.1.1.1.1.9" xref="alg1.l10.m1.1.1.1.1.9.cmml">s</mi></mrow><mo lspace="0em" rspace="0.167em" id="alg1.l10.m1.2.2.2.3" xref="alg1.l10.m1.2.2.3a.cmml">.</mo><mrow id="alg1.l10.m1.2.2.2.2" xref="alg1.l10.m1.2.2.2.2.cmml"><mtext id="alg1.l10.m1.2.2.2.2.2" xref="alg1.l10.m1.2.2.2.2.2a.cmml">sort</mtext><mo lspace="0em" rspace="0em" id="alg1.l10.m1.2.2.2.2.1" xref="alg1.l10.m1.2.2.2.2.1.cmml">​</mo><mrow id="alg1.l10.m1.2.2.2.2.3.2" xref="alg1.l10.m1.2.2.2.2.cmml"><mo stretchy="false" id="alg1.l10.m1.2.2.2.2.3.2.1" xref="alg1.l10.m1.2.2.2.2.3.1.cmml">(</mo><mo stretchy="false" id="alg1.l10.m1.2.2.2.2.3.2.2" xref="alg1.l10.m1.2.2.2.2.3.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l10.m1.2b"><apply id="alg1.l10.m1.2.2.3.cmml" xref="alg1.l10.m1.2.2.2"><csymbol cd="ambiguous" id="alg1.l10.m1.2.2.3a.cmml" xref="alg1.l10.m1.2.2.2.3">formulae-sequence</csymbol><apply id="alg1.l10.m1.1.1.1.1.cmml" xref="alg1.l10.m1.1.1.1.1"><times id="alg1.l10.m1.1.1.1.1.1.cmml" xref="alg1.l10.m1.1.1.1.1.1"></times><ci id="alg1.l10.m1.1.1.1.1.2.cmml" xref="alg1.l10.m1.1.1.1.1.2">𝑟</ci><ci id="alg1.l10.m1.1.1.1.1.3.cmml" xref="alg1.l10.m1.1.1.1.1.3">𝑒</ci><ci id="alg1.l10.m1.1.1.1.1.4.cmml" xref="alg1.l10.m1.1.1.1.1.4">𝑞</ci><ci id="alg1.l10.m1.1.1.1.1.5.cmml" xref="alg1.l10.m1.1.1.1.1.5">𝑢</ci><ci id="alg1.l10.m1.1.1.1.1.6.cmml" xref="alg1.l10.m1.1.1.1.1.6">𝑒</ci><ci id="alg1.l10.m1.1.1.1.1.7.cmml" xref="alg1.l10.m1.1.1.1.1.7">𝑠</ci><ci id="alg1.l10.m1.1.1.1.1.8.cmml" xref="alg1.l10.m1.1.1.1.1.8">𝑡</ci><ci id="alg1.l10.m1.1.1.1.1.9.cmml" xref="alg1.l10.m1.1.1.1.1.9">𝑠</ci></apply><apply id="alg1.l10.m1.2.2.2.2.cmml" xref="alg1.l10.m1.2.2.2.2"><times id="alg1.l10.m1.2.2.2.2.1.cmml" xref="alg1.l10.m1.2.2.2.2.1"></times><ci id="alg1.l10.m1.2.2.2.2.2a.cmml" xref="alg1.l10.m1.2.2.2.2.2"><mtext id="alg1.l10.m1.2.2.2.2.2.cmml" xref="alg1.l10.m1.2.2.2.2.2">sort</mtext></ci><list id="alg1.l10.m1.2.2.2.2.3.1.cmml" xref="alg1.l10.m1.2.2.2.2.3.2.1"></list></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l10.m1.2c">requests.\text{sort}()</annotation></semantics></math>

</div>
<div id="alg1.l11" class="ltx_listingline">
<span id="alg1.l11.1" class="ltx_text">// Select requests for the next batch</span>

</div>
<div id="alg1.l12" class="ltx_listingline">
<math id="alg1.l12.m1.3" class="ltx_Math" alttext="available\_size\leftarrow T.\text{evictable\_size}()+P.\text{available\_size}()" display="inline"><semantics id="alg1.l12.m1.3a"><mrow id="alg1.l12.m1.3.3.3" xref="alg1.l12.m1.3.3.4.cmml"><mrow id="alg1.l12.m1.1.1.1.1" xref="alg1.l12.m1.1.1.1.1.cmml"><mrow id="alg1.l12.m1.1.1.1.1.2" xref="alg1.l12.m1.1.1.1.1.2.cmml"><mi id="alg1.l12.m1.1.1.1.1.2.2" xref="alg1.l12.m1.1.1.1.1.2.2.cmml">a</mi><mo lspace="0em" rspace="0em" id="alg1.l12.m1.1.1.1.1.2.1" xref="alg1.l12.m1.1.1.1.1.2.1.cmml">​</mo><mi id="alg1.l12.m1.1.1.1.1.2.3" xref="alg1.l12.m1.1.1.1.1.2.3.cmml">v</mi><mo lspace="0em" rspace="0em" id="alg1.l12.m1.1.1.1.1.2.1a" xref="alg1.l12.m1.1.1.1.1.2.1.cmml">​</mo><mi id="alg1.l12.m1.1.1.1.1.2.4" xref="alg1.l12.m1.1.1.1.1.2.4.cmml">a</mi><mo lspace="0em" rspace="0em" id="alg1.l12.m1.1.1.1.1.2.1b" xref="alg1.l12.m1.1.1.1.1.2.1.cmml">​</mo><mi id="alg1.l12.m1.1.1.1.1.2.5" xref="alg1.l12.m1.1.1.1.1.2.5.cmml">i</mi><mo lspace="0em" rspace="0em" id="alg1.l12.m1.1.1.1.1.2.1c" xref="alg1.l12.m1.1.1.1.1.2.1.cmml">​</mo><mi id="alg1.l12.m1.1.1.1.1.2.6" xref="alg1.l12.m1.1.1.1.1.2.6.cmml">l</mi><mo lspace="0em" rspace="0em" id="alg1.l12.m1.1.1.1.1.2.1d" xref="alg1.l12.m1.1.1.1.1.2.1.cmml">​</mo><mi id="alg1.l12.m1.1.1.1.1.2.7" xref="alg1.l12.m1.1.1.1.1.2.7.cmml">a</mi><mo lspace="0em" rspace="0em" id="alg1.l12.m1.1.1.1.1.2.1e" xref="alg1.l12.m1.1.1.1.1.2.1.cmml">​</mo><mi id="alg1.l12.m1.1.1.1.1.2.8" xref="alg1.l12.m1.1.1.1.1.2.8.cmml">b</mi><mo lspace="0em" rspace="0em" id="alg1.l12.m1.1.1.1.1.2.1f" xref="alg1.l12.m1.1.1.1.1.2.1.cmml">​</mo><mi id="alg1.l12.m1.1.1.1.1.2.9" xref="alg1.l12.m1.1.1.1.1.2.9.cmml">l</mi><mo lspace="0em" rspace="0em" id="alg1.l12.m1.1.1.1.1.2.1g" xref="alg1.l12.m1.1.1.1.1.2.1.cmml">​</mo><mi id="alg1.l12.m1.1.1.1.1.2.10" xref="alg1.l12.m1.1.1.1.1.2.10.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l12.m1.1.1.1.1.2.1h" xref="alg1.l12.m1.1.1.1.1.2.1.cmml">​</mo><mi mathvariant="normal" id="alg1.l12.m1.1.1.1.1.2.11" xref="alg1.l12.m1.1.1.1.1.2.11.cmml">_</mi><mo lspace="0em" rspace="0em" id="alg1.l12.m1.1.1.1.1.2.1i" xref="alg1.l12.m1.1.1.1.1.2.1.cmml">​</mo><mi id="alg1.l12.m1.1.1.1.1.2.12" xref="alg1.l12.m1.1.1.1.1.2.12.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg1.l12.m1.1.1.1.1.2.1j" xref="alg1.l12.m1.1.1.1.1.2.1.cmml">​</mo><mi id="alg1.l12.m1.1.1.1.1.2.13" xref="alg1.l12.m1.1.1.1.1.2.13.cmml">i</mi><mo lspace="0em" rspace="0em" id="alg1.l12.m1.1.1.1.1.2.1k" xref="alg1.l12.m1.1.1.1.1.2.1.cmml">​</mo><mi id="alg1.l12.m1.1.1.1.1.2.14" xref="alg1.l12.m1.1.1.1.1.2.14.cmml">z</mi><mo lspace="0em" rspace="0em" id="alg1.l12.m1.1.1.1.1.2.1l" xref="alg1.l12.m1.1.1.1.1.2.1.cmml">​</mo><mi id="alg1.l12.m1.1.1.1.1.2.15" xref="alg1.l12.m1.1.1.1.1.2.15.cmml">e</mi></mrow><mo stretchy="false" id="alg1.l12.m1.1.1.1.1.1" xref="alg1.l12.m1.1.1.1.1.1.cmml">←</mo><mi id="alg1.l12.m1.1.1.1.1.3" xref="alg1.l12.m1.1.1.1.1.3.cmml">T</mi></mrow><mo lspace="0em" rspace="0.167em" id="alg1.l12.m1.3.3.3.4" xref="alg1.l12.m1.3.3.4a.cmml">.</mo><mrow id="alg1.l12.m1.2.2.2.2" xref="alg1.l12.m1.2.2.2.2.cmml"><mrow id="alg1.l12.m1.2.2.2.2.2" xref="alg1.l12.m1.2.2.2.2.2.cmml"><mtext id="alg1.l12.m1.2.2.2.2.2.2" xref="alg1.l12.m1.2.2.2.2.2.2a.cmml">evictable_size</mtext><mo lspace="0em" rspace="0em" id="alg1.l12.m1.2.2.2.2.2.1" xref="alg1.l12.m1.2.2.2.2.2.1.cmml">​</mo><mrow id="alg1.l12.m1.2.2.2.2.2.3.2" xref="alg1.l12.m1.2.2.2.2.2.cmml"><mo stretchy="false" id="alg1.l12.m1.2.2.2.2.2.3.2.1" xref="alg1.l12.m1.2.2.2.2.2.3.1.cmml">(</mo><mo stretchy="false" id="alg1.l12.m1.2.2.2.2.2.3.2.2" xref="alg1.l12.m1.2.2.2.2.2.3.1.cmml">)</mo></mrow></mrow><mo id="alg1.l12.m1.2.2.2.2.1" xref="alg1.l12.m1.2.2.2.2.1.cmml">+</mo><mi id="alg1.l12.m1.2.2.2.2.3" xref="alg1.l12.m1.2.2.2.2.3.cmml">P</mi></mrow><mo lspace="0em" rspace="0.167em" id="alg1.l12.m1.3.3.3.5" xref="alg1.l12.m1.3.3.4a.cmml">.</mo><mrow id="alg1.l12.m1.3.3.3.3" xref="alg1.l12.m1.3.3.3.3.cmml"><mtext id="alg1.l12.m1.3.3.3.3.2" xref="alg1.l12.m1.3.3.3.3.2a.cmml">available_size</mtext><mo lspace="0em" rspace="0em" id="alg1.l12.m1.3.3.3.3.1" xref="alg1.l12.m1.3.3.3.3.1.cmml">​</mo><mrow id="alg1.l12.m1.3.3.3.3.3.2" xref="alg1.l12.m1.3.3.3.3.cmml"><mo stretchy="false" id="alg1.l12.m1.3.3.3.3.3.2.1" xref="alg1.l12.m1.3.3.3.3.3.1.cmml">(</mo><mo stretchy="false" id="alg1.l12.m1.3.3.3.3.3.2.2" xref="alg1.l12.m1.3.3.3.3.3.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l12.m1.3b"><apply id="alg1.l12.m1.3.3.4.cmml" xref="alg1.l12.m1.3.3.3"><csymbol cd="ambiguous" id="alg1.l12.m1.3.3.4a.cmml" xref="alg1.l12.m1.3.3.3.4">formulae-sequence</csymbol><apply id="alg1.l12.m1.1.1.1.1.cmml" xref="alg1.l12.m1.1.1.1.1"><ci id="alg1.l12.m1.1.1.1.1.1.cmml" xref="alg1.l12.m1.1.1.1.1.1">←</ci><apply id="alg1.l12.m1.1.1.1.1.2.cmml" xref="alg1.l12.m1.1.1.1.1.2"><times id="alg1.l12.m1.1.1.1.1.2.1.cmml" xref="alg1.l12.m1.1.1.1.1.2.1"></times><ci id="alg1.l12.m1.1.1.1.1.2.2.cmml" xref="alg1.l12.m1.1.1.1.1.2.2">𝑎</ci><ci id="alg1.l12.m1.1.1.1.1.2.3.cmml" xref="alg1.l12.m1.1.1.1.1.2.3">𝑣</ci><ci id="alg1.l12.m1.1.1.1.1.2.4.cmml" xref="alg1.l12.m1.1.1.1.1.2.4">𝑎</ci><ci id="alg1.l12.m1.1.1.1.1.2.5.cmml" xref="alg1.l12.m1.1.1.1.1.2.5">𝑖</ci><ci id="alg1.l12.m1.1.1.1.1.2.6.cmml" xref="alg1.l12.m1.1.1.1.1.2.6">𝑙</ci><ci id="alg1.l12.m1.1.1.1.1.2.7.cmml" xref="alg1.l12.m1.1.1.1.1.2.7">𝑎</ci><ci id="alg1.l12.m1.1.1.1.1.2.8.cmml" xref="alg1.l12.m1.1.1.1.1.2.8">𝑏</ci><ci id="alg1.l12.m1.1.1.1.1.2.9.cmml" xref="alg1.l12.m1.1.1.1.1.2.9">𝑙</ci><ci id="alg1.l12.m1.1.1.1.1.2.10.cmml" xref="alg1.l12.m1.1.1.1.1.2.10">𝑒</ci><ci id="alg1.l12.m1.1.1.1.1.2.11.cmml" xref="alg1.l12.m1.1.1.1.1.2.11">_</ci><ci id="alg1.l12.m1.1.1.1.1.2.12.cmml" xref="alg1.l12.m1.1.1.1.1.2.12">𝑠</ci><ci id="alg1.l12.m1.1.1.1.1.2.13.cmml" xref="alg1.l12.m1.1.1.1.1.2.13">𝑖</ci><ci id="alg1.l12.m1.1.1.1.1.2.14.cmml" xref="alg1.l12.m1.1.1.1.1.2.14">𝑧</ci><ci id="alg1.l12.m1.1.1.1.1.2.15.cmml" xref="alg1.l12.m1.1.1.1.1.2.15">𝑒</ci></apply><ci id="alg1.l12.m1.1.1.1.1.3.cmml" xref="alg1.l12.m1.1.1.1.1.3">𝑇</ci></apply><apply id="alg1.l12.m1.2.2.2.2.cmml" xref="alg1.l12.m1.2.2.2.2"><plus id="alg1.l12.m1.2.2.2.2.1.cmml" xref="alg1.l12.m1.2.2.2.2.1"></plus><apply id="alg1.l12.m1.2.2.2.2.2.cmml" xref="alg1.l12.m1.2.2.2.2.2"><times id="alg1.l12.m1.2.2.2.2.2.1.cmml" xref="alg1.l12.m1.2.2.2.2.2.1"></times><ci id="alg1.l12.m1.2.2.2.2.2.2a.cmml" xref="alg1.l12.m1.2.2.2.2.2.2"><mtext id="alg1.l12.m1.2.2.2.2.2.2.cmml" xref="alg1.l12.m1.2.2.2.2.2.2">evictable_size</mtext></ci><list id="alg1.l12.m1.2.2.2.2.2.3.1.cmml" xref="alg1.l12.m1.2.2.2.2.2.3.2.1"></list></apply><ci id="alg1.l12.m1.2.2.2.2.3.cmml" xref="alg1.l12.m1.2.2.2.2.3">𝑃</ci></apply><apply id="alg1.l12.m1.3.3.3.3.cmml" xref="alg1.l12.m1.3.3.3.3"><times id="alg1.l12.m1.3.3.3.3.1.cmml" xref="alg1.l12.m1.3.3.3.3.1"></times><ci id="alg1.l12.m1.3.3.3.3.2a.cmml" xref="alg1.l12.m1.3.3.3.3.2"><mtext id="alg1.l12.m1.3.3.3.3.2.cmml" xref="alg1.l12.m1.3.3.3.3.2">available_size</mtext></ci><list id="alg1.l12.m1.3.3.3.3.3.1.cmml" xref="alg1.l12.m1.3.3.3.3.3.2.1"></list></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l12.m1.3c">available\_size\leftarrow T.\text{evictable\_size}()+P.\text{available\_size}()</annotation></semantics></math>

</div>
<div id="alg1.l13" class="ltx_listingline">
<math id="alg1.l13.m1.1" class="ltx_Math" alttext="current\_size\leftarrow 0" display="inline"><semantics id="alg1.l13.m1.1a"><mrow id="alg1.l13.m1.1.1" xref="alg1.l13.m1.1.1.cmml"><mrow id="alg1.l13.m1.1.1.2" xref="alg1.l13.m1.1.1.2.cmml"><mi id="alg1.l13.m1.1.1.2.2" xref="alg1.l13.m1.1.1.2.2.cmml">c</mi><mo lspace="0em" rspace="0em" id="alg1.l13.m1.1.1.2.1" xref="alg1.l13.m1.1.1.2.1.cmml">​</mo><mi id="alg1.l13.m1.1.1.2.3" xref="alg1.l13.m1.1.1.2.3.cmml">u</mi><mo lspace="0em" rspace="0em" id="alg1.l13.m1.1.1.2.1a" xref="alg1.l13.m1.1.1.2.1.cmml">​</mo><mi id="alg1.l13.m1.1.1.2.4" xref="alg1.l13.m1.1.1.2.4.cmml">r</mi><mo lspace="0em" rspace="0em" id="alg1.l13.m1.1.1.2.1b" xref="alg1.l13.m1.1.1.2.1.cmml">​</mo><mi id="alg1.l13.m1.1.1.2.5" xref="alg1.l13.m1.1.1.2.5.cmml">r</mi><mo lspace="0em" rspace="0em" id="alg1.l13.m1.1.1.2.1c" xref="alg1.l13.m1.1.1.2.1.cmml">​</mo><mi id="alg1.l13.m1.1.1.2.6" xref="alg1.l13.m1.1.1.2.6.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l13.m1.1.1.2.1d" xref="alg1.l13.m1.1.1.2.1.cmml">​</mo><mi id="alg1.l13.m1.1.1.2.7" xref="alg1.l13.m1.1.1.2.7.cmml">n</mi><mo lspace="0em" rspace="0em" id="alg1.l13.m1.1.1.2.1e" xref="alg1.l13.m1.1.1.2.1.cmml">​</mo><mi id="alg1.l13.m1.1.1.2.8" xref="alg1.l13.m1.1.1.2.8.cmml">t</mi><mo lspace="0em" rspace="0em" id="alg1.l13.m1.1.1.2.1f" xref="alg1.l13.m1.1.1.2.1.cmml">​</mo><mi mathvariant="normal" id="alg1.l13.m1.1.1.2.9" xref="alg1.l13.m1.1.1.2.9.cmml">_</mi><mo lspace="0em" rspace="0em" id="alg1.l13.m1.1.1.2.1g" xref="alg1.l13.m1.1.1.2.1.cmml">​</mo><mi id="alg1.l13.m1.1.1.2.10" xref="alg1.l13.m1.1.1.2.10.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg1.l13.m1.1.1.2.1h" xref="alg1.l13.m1.1.1.2.1.cmml">​</mo><mi id="alg1.l13.m1.1.1.2.11" xref="alg1.l13.m1.1.1.2.11.cmml">i</mi><mo lspace="0em" rspace="0em" id="alg1.l13.m1.1.1.2.1i" xref="alg1.l13.m1.1.1.2.1.cmml">​</mo><mi id="alg1.l13.m1.1.1.2.12" xref="alg1.l13.m1.1.1.2.12.cmml">z</mi><mo lspace="0em" rspace="0em" id="alg1.l13.m1.1.1.2.1j" xref="alg1.l13.m1.1.1.2.1.cmml">​</mo><mi id="alg1.l13.m1.1.1.2.13" xref="alg1.l13.m1.1.1.2.13.cmml">e</mi></mrow><mo stretchy="false" id="alg1.l13.m1.1.1.1" xref="alg1.l13.m1.1.1.1.cmml">←</mo><mn id="alg1.l13.m1.1.1.3" xref="alg1.l13.m1.1.1.3.cmml">0</mn></mrow><annotation-xml encoding="MathML-Content" id="alg1.l13.m1.1b"><apply id="alg1.l13.m1.1.1.cmml" xref="alg1.l13.m1.1.1"><ci id="alg1.l13.m1.1.1.1.cmml" xref="alg1.l13.m1.1.1.1">←</ci><apply id="alg1.l13.m1.1.1.2.cmml" xref="alg1.l13.m1.1.1.2"><times id="alg1.l13.m1.1.1.2.1.cmml" xref="alg1.l13.m1.1.1.2.1"></times><ci id="alg1.l13.m1.1.1.2.2.cmml" xref="alg1.l13.m1.1.1.2.2">𝑐</ci><ci id="alg1.l13.m1.1.1.2.3.cmml" xref="alg1.l13.m1.1.1.2.3">𝑢</ci><ci id="alg1.l13.m1.1.1.2.4.cmml" xref="alg1.l13.m1.1.1.2.4">𝑟</ci><ci id="alg1.l13.m1.1.1.2.5.cmml" xref="alg1.l13.m1.1.1.2.5">𝑟</ci><ci id="alg1.l13.m1.1.1.2.6.cmml" xref="alg1.l13.m1.1.1.2.6">𝑒</ci><ci id="alg1.l13.m1.1.1.2.7.cmml" xref="alg1.l13.m1.1.1.2.7">𝑛</ci><ci id="alg1.l13.m1.1.1.2.8.cmml" xref="alg1.l13.m1.1.1.2.8">𝑡</ci><ci id="alg1.l13.m1.1.1.2.9.cmml" xref="alg1.l13.m1.1.1.2.9">_</ci><ci id="alg1.l13.m1.1.1.2.10.cmml" xref="alg1.l13.m1.1.1.2.10">𝑠</ci><ci id="alg1.l13.m1.1.1.2.11.cmml" xref="alg1.l13.m1.1.1.2.11">𝑖</ci><ci id="alg1.l13.m1.1.1.2.12.cmml" xref="alg1.l13.m1.1.1.2.12">𝑧</ci><ci id="alg1.l13.m1.1.1.2.13.cmml" xref="alg1.l13.m1.1.1.2.13">𝑒</ci></apply><cn type="integer" id="alg1.l13.m1.1.1.3.cmml" xref="alg1.l13.m1.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l13.m1.1c">current\_size\leftarrow 0</annotation></semantics></math>

</div>
<div id="alg1.l14" class="ltx_listingline">
<math id="alg1.l14.m1.1" class="ltx_Math" alttext="new\_batch\leftarrow[]" display="inline"><semantics id="alg1.l14.m1.1a"><mrow id="alg1.l14.m1.1.1" xref="alg1.l14.m1.1.1.cmml"><mrow id="alg1.l14.m1.1.1.2" xref="alg1.l14.m1.1.1.2.cmml"><mi id="alg1.l14.m1.1.1.2.2" xref="alg1.l14.m1.1.1.2.2.cmml">n</mi><mo lspace="0em" rspace="0em" id="alg1.l14.m1.1.1.2.1" xref="alg1.l14.m1.1.1.2.1.cmml">​</mo><mi id="alg1.l14.m1.1.1.2.3" xref="alg1.l14.m1.1.1.2.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l14.m1.1.1.2.1a" xref="alg1.l14.m1.1.1.2.1.cmml">​</mo><mi id="alg1.l14.m1.1.1.2.4" xref="alg1.l14.m1.1.1.2.4.cmml">w</mi><mo lspace="0em" rspace="0em" id="alg1.l14.m1.1.1.2.1b" xref="alg1.l14.m1.1.1.2.1.cmml">​</mo><mi mathvariant="normal" id="alg1.l14.m1.1.1.2.5" xref="alg1.l14.m1.1.1.2.5.cmml">_</mi><mo lspace="0em" rspace="0em" id="alg1.l14.m1.1.1.2.1c" xref="alg1.l14.m1.1.1.2.1.cmml">​</mo><mi id="alg1.l14.m1.1.1.2.6" xref="alg1.l14.m1.1.1.2.6.cmml">b</mi><mo lspace="0em" rspace="0em" id="alg1.l14.m1.1.1.2.1d" xref="alg1.l14.m1.1.1.2.1.cmml">​</mo><mi id="alg1.l14.m1.1.1.2.7" xref="alg1.l14.m1.1.1.2.7.cmml">a</mi><mo lspace="0em" rspace="0em" id="alg1.l14.m1.1.1.2.1e" xref="alg1.l14.m1.1.1.2.1.cmml">​</mo><mi id="alg1.l14.m1.1.1.2.8" xref="alg1.l14.m1.1.1.2.8.cmml">t</mi><mo lspace="0em" rspace="0em" id="alg1.l14.m1.1.1.2.1f" xref="alg1.l14.m1.1.1.2.1.cmml">​</mo><mi id="alg1.l14.m1.1.1.2.9" xref="alg1.l14.m1.1.1.2.9.cmml">c</mi><mo lspace="0em" rspace="0em" id="alg1.l14.m1.1.1.2.1g" xref="alg1.l14.m1.1.1.2.1.cmml">​</mo><mi id="alg1.l14.m1.1.1.2.10" xref="alg1.l14.m1.1.1.2.10.cmml">h</mi></mrow><mo stretchy="false" id="alg1.l14.m1.1.1.1" xref="alg1.l14.m1.1.1.1.cmml">←</mo><mrow id="alg1.l14.m1.1.1.3.2" xref="alg1.l14.m1.1.1.cmml"><mo stretchy="false" id="alg1.l14.m1.1.1.3.2.1" xref="alg1.l14.m1.1.1.3.1.cmml">[</mo><mo stretchy="false" id="alg1.l14.m1.1.1.3.2.2" xref="alg1.l14.m1.1.1.3.1.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l14.m1.1b"><apply id="alg1.l14.m1.1.1.cmml" xref="alg1.l14.m1.1.1"><ci id="alg1.l14.m1.1.1.1.cmml" xref="alg1.l14.m1.1.1.1">←</ci><apply id="alg1.l14.m1.1.1.2.cmml" xref="alg1.l14.m1.1.1.2"><times id="alg1.l14.m1.1.1.2.1.cmml" xref="alg1.l14.m1.1.1.2.1"></times><ci id="alg1.l14.m1.1.1.2.2.cmml" xref="alg1.l14.m1.1.1.2.2">𝑛</ci><ci id="alg1.l14.m1.1.1.2.3.cmml" xref="alg1.l14.m1.1.1.2.3">𝑒</ci><ci id="alg1.l14.m1.1.1.2.4.cmml" xref="alg1.l14.m1.1.1.2.4">𝑤</ci><ci id="alg1.l14.m1.1.1.2.5.cmml" xref="alg1.l14.m1.1.1.2.5">_</ci><ci id="alg1.l14.m1.1.1.2.6.cmml" xref="alg1.l14.m1.1.1.2.6">𝑏</ci><ci id="alg1.l14.m1.1.1.2.7.cmml" xref="alg1.l14.m1.1.1.2.7">𝑎</ci><ci id="alg1.l14.m1.1.1.2.8.cmml" xref="alg1.l14.m1.1.1.2.8">𝑡</ci><ci id="alg1.l14.m1.1.1.2.9.cmml" xref="alg1.l14.m1.1.1.2.9">𝑐</ci><ci id="alg1.l14.m1.1.1.2.10.cmml" xref="alg1.l14.m1.1.1.2.10">ℎ</ci></apply><list id="alg1.l14.m1.1.1.3.1.cmml" xref="alg1.l14.m1.1.1.3.2.1"></list></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l14.m1.1c">new\_batch\leftarrow[]</annotation></semantics></math>

</div>
<div id="alg1.l15" class="ltx_listingline">
<span id="alg1.l15.1" class="ltx_text ltx_font_bold">for</span> <math id="alg1.l15.m1.1" class="ltx_Math" alttext="req\in requests" display="inline"><semantics id="alg1.l15.m1.1a"><mrow id="alg1.l15.m1.1.1" xref="alg1.l15.m1.1.1.cmml"><mrow id="alg1.l15.m1.1.1.2" xref="alg1.l15.m1.1.1.2.cmml"><mi id="alg1.l15.m1.1.1.2.2" xref="alg1.l15.m1.1.1.2.2.cmml">r</mi><mo lspace="0em" rspace="0em" id="alg1.l15.m1.1.1.2.1" xref="alg1.l15.m1.1.1.2.1.cmml">​</mo><mi id="alg1.l15.m1.1.1.2.3" xref="alg1.l15.m1.1.1.2.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l15.m1.1.1.2.1a" xref="alg1.l15.m1.1.1.2.1.cmml">​</mo><mi id="alg1.l15.m1.1.1.2.4" xref="alg1.l15.m1.1.1.2.4.cmml">q</mi></mrow><mo id="alg1.l15.m1.1.1.1" xref="alg1.l15.m1.1.1.1.cmml">∈</mo><mrow id="alg1.l15.m1.1.1.3" xref="alg1.l15.m1.1.1.3.cmml"><mi id="alg1.l15.m1.1.1.3.2" xref="alg1.l15.m1.1.1.3.2.cmml">r</mi><mo lspace="0em" rspace="0em" id="alg1.l15.m1.1.1.3.1" xref="alg1.l15.m1.1.1.3.1.cmml">​</mo><mi id="alg1.l15.m1.1.1.3.3" xref="alg1.l15.m1.1.1.3.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l15.m1.1.1.3.1a" xref="alg1.l15.m1.1.1.3.1.cmml">​</mo><mi id="alg1.l15.m1.1.1.3.4" xref="alg1.l15.m1.1.1.3.4.cmml">q</mi><mo lspace="0em" rspace="0em" id="alg1.l15.m1.1.1.3.1b" xref="alg1.l15.m1.1.1.3.1.cmml">​</mo><mi id="alg1.l15.m1.1.1.3.5" xref="alg1.l15.m1.1.1.3.5.cmml">u</mi><mo lspace="0em" rspace="0em" id="alg1.l15.m1.1.1.3.1c" xref="alg1.l15.m1.1.1.3.1.cmml">​</mo><mi id="alg1.l15.m1.1.1.3.6" xref="alg1.l15.m1.1.1.3.6.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l15.m1.1.1.3.1d" xref="alg1.l15.m1.1.1.3.1.cmml">​</mo><mi id="alg1.l15.m1.1.1.3.7" xref="alg1.l15.m1.1.1.3.7.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg1.l15.m1.1.1.3.1e" xref="alg1.l15.m1.1.1.3.1.cmml">​</mo><mi id="alg1.l15.m1.1.1.3.8" xref="alg1.l15.m1.1.1.3.8.cmml">t</mi><mo lspace="0em" rspace="0em" id="alg1.l15.m1.1.1.3.1f" xref="alg1.l15.m1.1.1.3.1.cmml">​</mo><mi id="alg1.l15.m1.1.1.3.9" xref="alg1.l15.m1.1.1.3.9.cmml">s</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l15.m1.1b"><apply id="alg1.l15.m1.1.1.cmml" xref="alg1.l15.m1.1.1"><in id="alg1.l15.m1.1.1.1.cmml" xref="alg1.l15.m1.1.1.1"></in><apply id="alg1.l15.m1.1.1.2.cmml" xref="alg1.l15.m1.1.1.2"><times id="alg1.l15.m1.1.1.2.1.cmml" xref="alg1.l15.m1.1.1.2.1"></times><ci id="alg1.l15.m1.1.1.2.2.cmml" xref="alg1.l15.m1.1.1.2.2">𝑟</ci><ci id="alg1.l15.m1.1.1.2.3.cmml" xref="alg1.l15.m1.1.1.2.3">𝑒</ci><ci id="alg1.l15.m1.1.1.2.4.cmml" xref="alg1.l15.m1.1.1.2.4">𝑞</ci></apply><apply id="alg1.l15.m1.1.1.3.cmml" xref="alg1.l15.m1.1.1.3"><times id="alg1.l15.m1.1.1.3.1.cmml" xref="alg1.l15.m1.1.1.3.1"></times><ci id="alg1.l15.m1.1.1.3.2.cmml" xref="alg1.l15.m1.1.1.3.2">𝑟</ci><ci id="alg1.l15.m1.1.1.3.3.cmml" xref="alg1.l15.m1.1.1.3.3">𝑒</ci><ci id="alg1.l15.m1.1.1.3.4.cmml" xref="alg1.l15.m1.1.1.3.4">𝑞</ci><ci id="alg1.l15.m1.1.1.3.5.cmml" xref="alg1.l15.m1.1.1.3.5">𝑢</ci><ci id="alg1.l15.m1.1.1.3.6.cmml" xref="alg1.l15.m1.1.1.3.6">𝑒</ci><ci id="alg1.l15.m1.1.1.3.7.cmml" xref="alg1.l15.m1.1.1.3.7">𝑠</ci><ci id="alg1.l15.m1.1.1.3.8.cmml" xref="alg1.l15.m1.1.1.3.8">𝑡</ci><ci id="alg1.l15.m1.1.1.3.9.cmml" xref="alg1.l15.m1.1.1.3.9">𝑠</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l15.m1.1c">req\in requests</annotation></semantics></math> <span id="alg1.l15.2" class="ltx_text ltx_font_bold">do</span>

</div>
<div id="alg1.l16" class="ltx_listingline">     <span id="alg1.l16.1" class="ltx_text ltx_font_bold">if</span> <math id="alg1.l16.m1.2" class="ltx_Math" alttext="req.\text{size}()+current\_size&lt;available\_size" display="inline"><semantics id="alg1.l16.m1.2a"><mrow id="alg1.l16.m1.2.2.2" xref="alg1.l16.m1.2.2.3.cmml"><mrow id="alg1.l16.m1.1.1.1.1" xref="alg1.l16.m1.1.1.1.1.cmml"><mi id="alg1.l16.m1.1.1.1.1.2" xref="alg1.l16.m1.1.1.1.1.2.cmml">r</mi><mo lspace="0em" rspace="0em" id="alg1.l16.m1.1.1.1.1.1" xref="alg1.l16.m1.1.1.1.1.1.cmml">​</mo><mi id="alg1.l16.m1.1.1.1.1.3" xref="alg1.l16.m1.1.1.1.1.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l16.m1.1.1.1.1.1a" xref="alg1.l16.m1.1.1.1.1.1.cmml">​</mo><mi id="alg1.l16.m1.1.1.1.1.4" xref="alg1.l16.m1.1.1.1.1.4.cmml">q</mi></mrow><mo lspace="0em" rspace="0.167em" id="alg1.l16.m1.2.2.2.3" xref="alg1.l16.m1.2.2.3a.cmml">.</mo><mrow id="alg1.l16.m1.2.2.2.2" xref="alg1.l16.m1.2.2.2.2.cmml"><mrow id="alg1.l16.m1.2.2.2.2.2" xref="alg1.l16.m1.2.2.2.2.2.cmml"><mrow id="alg1.l16.m1.2.2.2.2.2.2" xref="alg1.l16.m1.2.2.2.2.2.2.cmml"><mtext id="alg1.l16.m1.2.2.2.2.2.2.2" xref="alg1.l16.m1.2.2.2.2.2.2.2a.cmml">size</mtext><mo lspace="0em" rspace="0em" id="alg1.l16.m1.2.2.2.2.2.2.1" xref="alg1.l16.m1.2.2.2.2.2.2.1.cmml">​</mo><mrow id="alg1.l16.m1.2.2.2.2.2.2.3.2" xref="alg1.l16.m1.2.2.2.2.2.2.cmml"><mo stretchy="false" id="alg1.l16.m1.2.2.2.2.2.2.3.2.1" xref="alg1.l16.m1.2.2.2.2.2.2.3.1.cmml">(</mo><mo stretchy="false" id="alg1.l16.m1.2.2.2.2.2.2.3.2.2" xref="alg1.l16.m1.2.2.2.2.2.2.3.1.cmml">)</mo></mrow></mrow><mo id="alg1.l16.m1.2.2.2.2.2.1" xref="alg1.l16.m1.2.2.2.2.2.1.cmml">+</mo><mrow id="alg1.l16.m1.2.2.2.2.2.3" xref="alg1.l16.m1.2.2.2.2.2.3.cmml"><mi id="alg1.l16.m1.2.2.2.2.2.3.2" xref="alg1.l16.m1.2.2.2.2.2.3.2.cmml">c</mi><mo lspace="0em" rspace="0em" id="alg1.l16.m1.2.2.2.2.2.3.1" xref="alg1.l16.m1.2.2.2.2.2.3.1.cmml">​</mo><mi id="alg1.l16.m1.2.2.2.2.2.3.3" xref="alg1.l16.m1.2.2.2.2.2.3.3.cmml">u</mi><mo lspace="0em" rspace="0em" id="alg1.l16.m1.2.2.2.2.2.3.1a" xref="alg1.l16.m1.2.2.2.2.2.3.1.cmml">​</mo><mi id="alg1.l16.m1.2.2.2.2.2.3.4" xref="alg1.l16.m1.2.2.2.2.2.3.4.cmml">r</mi><mo lspace="0em" rspace="0em" id="alg1.l16.m1.2.2.2.2.2.3.1b" xref="alg1.l16.m1.2.2.2.2.2.3.1.cmml">​</mo><mi id="alg1.l16.m1.2.2.2.2.2.3.5" xref="alg1.l16.m1.2.2.2.2.2.3.5.cmml">r</mi><mo lspace="0em" rspace="0em" id="alg1.l16.m1.2.2.2.2.2.3.1c" xref="alg1.l16.m1.2.2.2.2.2.3.1.cmml">​</mo><mi id="alg1.l16.m1.2.2.2.2.2.3.6" xref="alg1.l16.m1.2.2.2.2.2.3.6.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l16.m1.2.2.2.2.2.3.1d" xref="alg1.l16.m1.2.2.2.2.2.3.1.cmml">​</mo><mi id="alg1.l16.m1.2.2.2.2.2.3.7" xref="alg1.l16.m1.2.2.2.2.2.3.7.cmml">n</mi><mo lspace="0em" rspace="0em" id="alg1.l16.m1.2.2.2.2.2.3.1e" xref="alg1.l16.m1.2.2.2.2.2.3.1.cmml">​</mo><mi id="alg1.l16.m1.2.2.2.2.2.3.8" xref="alg1.l16.m1.2.2.2.2.2.3.8.cmml">t</mi><mo lspace="0em" rspace="0em" id="alg1.l16.m1.2.2.2.2.2.3.1f" xref="alg1.l16.m1.2.2.2.2.2.3.1.cmml">​</mo><mi mathvariant="normal" id="alg1.l16.m1.2.2.2.2.2.3.9" xref="alg1.l16.m1.2.2.2.2.2.3.9.cmml">_</mi><mo lspace="0em" rspace="0em" id="alg1.l16.m1.2.2.2.2.2.3.1g" xref="alg1.l16.m1.2.2.2.2.2.3.1.cmml">​</mo><mi id="alg1.l16.m1.2.2.2.2.2.3.10" xref="alg1.l16.m1.2.2.2.2.2.3.10.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg1.l16.m1.2.2.2.2.2.3.1h" xref="alg1.l16.m1.2.2.2.2.2.3.1.cmml">​</mo><mi id="alg1.l16.m1.2.2.2.2.2.3.11" xref="alg1.l16.m1.2.2.2.2.2.3.11.cmml">i</mi><mo lspace="0em" rspace="0em" id="alg1.l16.m1.2.2.2.2.2.3.1i" xref="alg1.l16.m1.2.2.2.2.2.3.1.cmml">​</mo><mi id="alg1.l16.m1.2.2.2.2.2.3.12" xref="alg1.l16.m1.2.2.2.2.2.3.12.cmml">z</mi><mo lspace="0em" rspace="0em" id="alg1.l16.m1.2.2.2.2.2.3.1j" xref="alg1.l16.m1.2.2.2.2.2.3.1.cmml">​</mo><mi id="alg1.l16.m1.2.2.2.2.2.3.13" xref="alg1.l16.m1.2.2.2.2.2.3.13.cmml">e</mi></mrow></mrow><mo id="alg1.l16.m1.2.2.2.2.1" xref="alg1.l16.m1.2.2.2.2.1.cmml">&lt;</mo><mrow id="alg1.l16.m1.2.2.2.2.3" xref="alg1.l16.m1.2.2.2.2.3.cmml"><mi id="alg1.l16.m1.2.2.2.2.3.2" xref="alg1.l16.m1.2.2.2.2.3.2.cmml">a</mi><mo lspace="0em" rspace="0em" id="alg1.l16.m1.2.2.2.2.3.1" xref="alg1.l16.m1.2.2.2.2.3.1.cmml">​</mo><mi id="alg1.l16.m1.2.2.2.2.3.3" xref="alg1.l16.m1.2.2.2.2.3.3.cmml">v</mi><mo lspace="0em" rspace="0em" id="alg1.l16.m1.2.2.2.2.3.1a" xref="alg1.l16.m1.2.2.2.2.3.1.cmml">​</mo><mi id="alg1.l16.m1.2.2.2.2.3.4" xref="alg1.l16.m1.2.2.2.2.3.4.cmml">a</mi><mo lspace="0em" rspace="0em" id="alg1.l16.m1.2.2.2.2.3.1b" xref="alg1.l16.m1.2.2.2.2.3.1.cmml">​</mo><mi id="alg1.l16.m1.2.2.2.2.3.5" xref="alg1.l16.m1.2.2.2.2.3.5.cmml">i</mi><mo lspace="0em" rspace="0em" id="alg1.l16.m1.2.2.2.2.3.1c" xref="alg1.l16.m1.2.2.2.2.3.1.cmml">​</mo><mi id="alg1.l16.m1.2.2.2.2.3.6" xref="alg1.l16.m1.2.2.2.2.3.6.cmml">l</mi><mo lspace="0em" rspace="0em" id="alg1.l16.m1.2.2.2.2.3.1d" xref="alg1.l16.m1.2.2.2.2.3.1.cmml">​</mo><mi id="alg1.l16.m1.2.2.2.2.3.7" xref="alg1.l16.m1.2.2.2.2.3.7.cmml">a</mi><mo lspace="0em" rspace="0em" id="alg1.l16.m1.2.2.2.2.3.1e" xref="alg1.l16.m1.2.2.2.2.3.1.cmml">​</mo><mi id="alg1.l16.m1.2.2.2.2.3.8" xref="alg1.l16.m1.2.2.2.2.3.8.cmml">b</mi><mo lspace="0em" rspace="0em" id="alg1.l16.m1.2.2.2.2.3.1f" xref="alg1.l16.m1.2.2.2.2.3.1.cmml">​</mo><mi id="alg1.l16.m1.2.2.2.2.3.9" xref="alg1.l16.m1.2.2.2.2.3.9.cmml">l</mi><mo lspace="0em" rspace="0em" id="alg1.l16.m1.2.2.2.2.3.1g" xref="alg1.l16.m1.2.2.2.2.3.1.cmml">​</mo><mi id="alg1.l16.m1.2.2.2.2.3.10" xref="alg1.l16.m1.2.2.2.2.3.10.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l16.m1.2.2.2.2.3.1h" xref="alg1.l16.m1.2.2.2.2.3.1.cmml">​</mo><mi mathvariant="normal" id="alg1.l16.m1.2.2.2.2.3.11" xref="alg1.l16.m1.2.2.2.2.3.11.cmml">_</mi><mo lspace="0em" rspace="0em" id="alg1.l16.m1.2.2.2.2.3.1i" xref="alg1.l16.m1.2.2.2.2.3.1.cmml">​</mo><mi id="alg1.l16.m1.2.2.2.2.3.12" xref="alg1.l16.m1.2.2.2.2.3.12.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg1.l16.m1.2.2.2.2.3.1j" xref="alg1.l16.m1.2.2.2.2.3.1.cmml">​</mo><mi id="alg1.l16.m1.2.2.2.2.3.13" xref="alg1.l16.m1.2.2.2.2.3.13.cmml">i</mi><mo lspace="0em" rspace="0em" id="alg1.l16.m1.2.2.2.2.3.1k" xref="alg1.l16.m1.2.2.2.2.3.1.cmml">​</mo><mi id="alg1.l16.m1.2.2.2.2.3.14" xref="alg1.l16.m1.2.2.2.2.3.14.cmml">z</mi><mo lspace="0em" rspace="0em" id="alg1.l16.m1.2.2.2.2.3.1l" xref="alg1.l16.m1.2.2.2.2.3.1.cmml">​</mo><mi id="alg1.l16.m1.2.2.2.2.3.15" xref="alg1.l16.m1.2.2.2.2.3.15.cmml">e</mi></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l16.m1.2b"><apply id="alg1.l16.m1.2.2.3.cmml" xref="alg1.l16.m1.2.2.2"><csymbol cd="ambiguous" id="alg1.l16.m1.2.2.3a.cmml" xref="alg1.l16.m1.2.2.2.3">formulae-sequence</csymbol><apply id="alg1.l16.m1.1.1.1.1.cmml" xref="alg1.l16.m1.1.1.1.1"><times id="alg1.l16.m1.1.1.1.1.1.cmml" xref="alg1.l16.m1.1.1.1.1.1"></times><ci id="alg1.l16.m1.1.1.1.1.2.cmml" xref="alg1.l16.m1.1.1.1.1.2">𝑟</ci><ci id="alg1.l16.m1.1.1.1.1.3.cmml" xref="alg1.l16.m1.1.1.1.1.3">𝑒</ci><ci id="alg1.l16.m1.1.1.1.1.4.cmml" xref="alg1.l16.m1.1.1.1.1.4">𝑞</ci></apply><apply id="alg1.l16.m1.2.2.2.2.cmml" xref="alg1.l16.m1.2.2.2.2"><lt id="alg1.l16.m1.2.2.2.2.1.cmml" xref="alg1.l16.m1.2.2.2.2.1"></lt><apply id="alg1.l16.m1.2.2.2.2.2.cmml" xref="alg1.l16.m1.2.2.2.2.2"><plus id="alg1.l16.m1.2.2.2.2.2.1.cmml" xref="alg1.l16.m1.2.2.2.2.2.1"></plus><apply id="alg1.l16.m1.2.2.2.2.2.2.cmml" xref="alg1.l16.m1.2.2.2.2.2.2"><times id="alg1.l16.m1.2.2.2.2.2.2.1.cmml" xref="alg1.l16.m1.2.2.2.2.2.2.1"></times><ci id="alg1.l16.m1.2.2.2.2.2.2.2a.cmml" xref="alg1.l16.m1.2.2.2.2.2.2.2"><mtext id="alg1.l16.m1.2.2.2.2.2.2.2.cmml" xref="alg1.l16.m1.2.2.2.2.2.2.2">size</mtext></ci><list id="alg1.l16.m1.2.2.2.2.2.2.3.1.cmml" xref="alg1.l16.m1.2.2.2.2.2.2.3.2.1"></list></apply><apply id="alg1.l16.m1.2.2.2.2.2.3.cmml" xref="alg1.l16.m1.2.2.2.2.2.3"><times id="alg1.l16.m1.2.2.2.2.2.3.1.cmml" xref="alg1.l16.m1.2.2.2.2.2.3.1"></times><ci id="alg1.l16.m1.2.2.2.2.2.3.2.cmml" xref="alg1.l16.m1.2.2.2.2.2.3.2">𝑐</ci><ci id="alg1.l16.m1.2.2.2.2.2.3.3.cmml" xref="alg1.l16.m1.2.2.2.2.2.3.3">𝑢</ci><ci id="alg1.l16.m1.2.2.2.2.2.3.4.cmml" xref="alg1.l16.m1.2.2.2.2.2.3.4">𝑟</ci><ci id="alg1.l16.m1.2.2.2.2.2.3.5.cmml" xref="alg1.l16.m1.2.2.2.2.2.3.5">𝑟</ci><ci id="alg1.l16.m1.2.2.2.2.2.3.6.cmml" xref="alg1.l16.m1.2.2.2.2.2.3.6">𝑒</ci><ci id="alg1.l16.m1.2.2.2.2.2.3.7.cmml" xref="alg1.l16.m1.2.2.2.2.2.3.7">𝑛</ci><ci id="alg1.l16.m1.2.2.2.2.2.3.8.cmml" xref="alg1.l16.m1.2.2.2.2.2.3.8">𝑡</ci><ci id="alg1.l16.m1.2.2.2.2.2.3.9.cmml" xref="alg1.l16.m1.2.2.2.2.2.3.9">_</ci><ci id="alg1.l16.m1.2.2.2.2.2.3.10.cmml" xref="alg1.l16.m1.2.2.2.2.2.3.10">𝑠</ci><ci id="alg1.l16.m1.2.2.2.2.2.3.11.cmml" xref="alg1.l16.m1.2.2.2.2.2.3.11">𝑖</ci><ci id="alg1.l16.m1.2.2.2.2.2.3.12.cmml" xref="alg1.l16.m1.2.2.2.2.2.3.12">𝑧</ci><ci id="alg1.l16.m1.2.2.2.2.2.3.13.cmml" xref="alg1.l16.m1.2.2.2.2.2.3.13">𝑒</ci></apply></apply><apply id="alg1.l16.m1.2.2.2.2.3.cmml" xref="alg1.l16.m1.2.2.2.2.3"><times id="alg1.l16.m1.2.2.2.2.3.1.cmml" xref="alg1.l16.m1.2.2.2.2.3.1"></times><ci id="alg1.l16.m1.2.2.2.2.3.2.cmml" xref="alg1.l16.m1.2.2.2.2.3.2">𝑎</ci><ci id="alg1.l16.m1.2.2.2.2.3.3.cmml" xref="alg1.l16.m1.2.2.2.2.3.3">𝑣</ci><ci id="alg1.l16.m1.2.2.2.2.3.4.cmml" xref="alg1.l16.m1.2.2.2.2.3.4">𝑎</ci><ci id="alg1.l16.m1.2.2.2.2.3.5.cmml" xref="alg1.l16.m1.2.2.2.2.3.5">𝑖</ci><ci id="alg1.l16.m1.2.2.2.2.3.6.cmml" xref="alg1.l16.m1.2.2.2.2.3.6">𝑙</ci><ci id="alg1.l16.m1.2.2.2.2.3.7.cmml" xref="alg1.l16.m1.2.2.2.2.3.7">𝑎</ci><ci id="alg1.l16.m1.2.2.2.2.3.8.cmml" xref="alg1.l16.m1.2.2.2.2.3.8">𝑏</ci><ci id="alg1.l16.m1.2.2.2.2.3.9.cmml" xref="alg1.l16.m1.2.2.2.2.3.9">𝑙</ci><ci id="alg1.l16.m1.2.2.2.2.3.10.cmml" xref="alg1.l16.m1.2.2.2.2.3.10">𝑒</ci><ci id="alg1.l16.m1.2.2.2.2.3.11.cmml" xref="alg1.l16.m1.2.2.2.2.3.11">_</ci><ci id="alg1.l16.m1.2.2.2.2.3.12.cmml" xref="alg1.l16.m1.2.2.2.2.3.12">𝑠</ci><ci id="alg1.l16.m1.2.2.2.2.3.13.cmml" xref="alg1.l16.m1.2.2.2.2.3.13">𝑖</ci><ci id="alg1.l16.m1.2.2.2.2.3.14.cmml" xref="alg1.l16.m1.2.2.2.2.3.14">𝑧</ci><ci id="alg1.l16.m1.2.2.2.2.3.15.cmml" xref="alg1.l16.m1.2.2.2.2.3.15">𝑒</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l16.m1.2c">req.\text{size}()+current\_size&lt;available\_size</annotation></semantics></math> <span id="alg1.l16.2" class="ltx_text ltx_font_bold">then</span>

</div>
<div id="alg1.l17" class="ltx_listingline">         <math id="alg1.l17.m1.2" class="ltx_Math" alttext="new\_batch.\text{append}(req)" display="inline"><semantics id="alg1.l17.m1.2a"><mrow id="alg1.l17.m1.2.2.2" xref="alg1.l17.m1.2.2.3.cmml"><mrow id="alg1.l17.m1.1.1.1.1" xref="alg1.l17.m1.1.1.1.1.cmml"><mi id="alg1.l17.m1.1.1.1.1.2" xref="alg1.l17.m1.1.1.1.1.2.cmml">n</mi><mo lspace="0em" rspace="0em" id="alg1.l17.m1.1.1.1.1.1" xref="alg1.l17.m1.1.1.1.1.1.cmml">​</mo><mi id="alg1.l17.m1.1.1.1.1.3" xref="alg1.l17.m1.1.1.1.1.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l17.m1.1.1.1.1.1a" xref="alg1.l17.m1.1.1.1.1.1.cmml">​</mo><mi id="alg1.l17.m1.1.1.1.1.4" xref="alg1.l17.m1.1.1.1.1.4.cmml">w</mi><mo lspace="0em" rspace="0em" id="alg1.l17.m1.1.1.1.1.1b" xref="alg1.l17.m1.1.1.1.1.1.cmml">​</mo><mi mathvariant="normal" id="alg1.l17.m1.1.1.1.1.5" xref="alg1.l17.m1.1.1.1.1.5.cmml">_</mi><mo lspace="0em" rspace="0em" id="alg1.l17.m1.1.1.1.1.1c" xref="alg1.l17.m1.1.1.1.1.1.cmml">​</mo><mi id="alg1.l17.m1.1.1.1.1.6" xref="alg1.l17.m1.1.1.1.1.6.cmml">b</mi><mo lspace="0em" rspace="0em" id="alg1.l17.m1.1.1.1.1.1d" xref="alg1.l17.m1.1.1.1.1.1.cmml">​</mo><mi id="alg1.l17.m1.1.1.1.1.7" xref="alg1.l17.m1.1.1.1.1.7.cmml">a</mi><mo lspace="0em" rspace="0em" id="alg1.l17.m1.1.1.1.1.1e" xref="alg1.l17.m1.1.1.1.1.1.cmml">​</mo><mi id="alg1.l17.m1.1.1.1.1.8" xref="alg1.l17.m1.1.1.1.1.8.cmml">t</mi><mo lspace="0em" rspace="0em" id="alg1.l17.m1.1.1.1.1.1f" xref="alg1.l17.m1.1.1.1.1.1.cmml">​</mo><mi id="alg1.l17.m1.1.1.1.1.9" xref="alg1.l17.m1.1.1.1.1.9.cmml">c</mi><mo lspace="0em" rspace="0em" id="alg1.l17.m1.1.1.1.1.1g" xref="alg1.l17.m1.1.1.1.1.1.cmml">​</mo><mi id="alg1.l17.m1.1.1.1.1.10" xref="alg1.l17.m1.1.1.1.1.10.cmml">h</mi></mrow><mo lspace="0em" rspace="0.167em" id="alg1.l17.m1.2.2.2.3" xref="alg1.l17.m1.2.2.3a.cmml">.</mo><mrow id="alg1.l17.m1.2.2.2.2" xref="alg1.l17.m1.2.2.2.2.cmml"><mtext id="alg1.l17.m1.2.2.2.2.3" xref="alg1.l17.m1.2.2.2.2.3a.cmml">append</mtext><mo lspace="0em" rspace="0em" id="alg1.l17.m1.2.2.2.2.2" xref="alg1.l17.m1.2.2.2.2.2.cmml">​</mo><mrow id="alg1.l17.m1.2.2.2.2.1.1" xref="alg1.l17.m1.2.2.2.2.1.1.1.cmml"><mo stretchy="false" id="alg1.l17.m1.2.2.2.2.1.1.2" xref="alg1.l17.m1.2.2.2.2.1.1.1.cmml">(</mo><mrow id="alg1.l17.m1.2.2.2.2.1.1.1" xref="alg1.l17.m1.2.2.2.2.1.1.1.cmml"><mi id="alg1.l17.m1.2.2.2.2.1.1.1.2" xref="alg1.l17.m1.2.2.2.2.1.1.1.2.cmml">r</mi><mo lspace="0em" rspace="0em" id="alg1.l17.m1.2.2.2.2.1.1.1.1" xref="alg1.l17.m1.2.2.2.2.1.1.1.1.cmml">​</mo><mi id="alg1.l17.m1.2.2.2.2.1.1.1.3" xref="alg1.l17.m1.2.2.2.2.1.1.1.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l17.m1.2.2.2.2.1.1.1.1a" xref="alg1.l17.m1.2.2.2.2.1.1.1.1.cmml">​</mo><mi id="alg1.l17.m1.2.2.2.2.1.1.1.4" xref="alg1.l17.m1.2.2.2.2.1.1.1.4.cmml">q</mi></mrow><mo stretchy="false" id="alg1.l17.m1.2.2.2.2.1.1.3" xref="alg1.l17.m1.2.2.2.2.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l17.m1.2b"><apply id="alg1.l17.m1.2.2.3.cmml" xref="alg1.l17.m1.2.2.2"><csymbol cd="ambiguous" id="alg1.l17.m1.2.2.3a.cmml" xref="alg1.l17.m1.2.2.2.3">formulae-sequence</csymbol><apply id="alg1.l17.m1.1.1.1.1.cmml" xref="alg1.l17.m1.1.1.1.1"><times id="alg1.l17.m1.1.1.1.1.1.cmml" xref="alg1.l17.m1.1.1.1.1.1"></times><ci id="alg1.l17.m1.1.1.1.1.2.cmml" xref="alg1.l17.m1.1.1.1.1.2">𝑛</ci><ci id="alg1.l17.m1.1.1.1.1.3.cmml" xref="alg1.l17.m1.1.1.1.1.3">𝑒</ci><ci id="alg1.l17.m1.1.1.1.1.4.cmml" xref="alg1.l17.m1.1.1.1.1.4">𝑤</ci><ci id="alg1.l17.m1.1.1.1.1.5.cmml" xref="alg1.l17.m1.1.1.1.1.5">_</ci><ci id="alg1.l17.m1.1.1.1.1.6.cmml" xref="alg1.l17.m1.1.1.1.1.6">𝑏</ci><ci id="alg1.l17.m1.1.1.1.1.7.cmml" xref="alg1.l17.m1.1.1.1.1.7">𝑎</ci><ci id="alg1.l17.m1.1.1.1.1.8.cmml" xref="alg1.l17.m1.1.1.1.1.8">𝑡</ci><ci id="alg1.l17.m1.1.1.1.1.9.cmml" xref="alg1.l17.m1.1.1.1.1.9">𝑐</ci><ci id="alg1.l17.m1.1.1.1.1.10.cmml" xref="alg1.l17.m1.1.1.1.1.10">ℎ</ci></apply><apply id="alg1.l17.m1.2.2.2.2.cmml" xref="alg1.l17.m1.2.2.2.2"><times id="alg1.l17.m1.2.2.2.2.2.cmml" xref="alg1.l17.m1.2.2.2.2.2"></times><ci id="alg1.l17.m1.2.2.2.2.3a.cmml" xref="alg1.l17.m1.2.2.2.2.3"><mtext id="alg1.l17.m1.2.2.2.2.3.cmml" xref="alg1.l17.m1.2.2.2.2.3">append</mtext></ci><apply id="alg1.l17.m1.2.2.2.2.1.1.1.cmml" xref="alg1.l17.m1.2.2.2.2.1.1"><times id="alg1.l17.m1.2.2.2.2.1.1.1.1.cmml" xref="alg1.l17.m1.2.2.2.2.1.1.1.1"></times><ci id="alg1.l17.m1.2.2.2.2.1.1.1.2.cmml" xref="alg1.l17.m1.2.2.2.2.1.1.1.2">𝑟</ci><ci id="alg1.l17.m1.2.2.2.2.1.1.1.3.cmml" xref="alg1.l17.m1.2.2.2.2.1.1.1.3">𝑒</ci><ci id="alg1.l17.m1.2.2.2.2.1.1.1.4.cmml" xref="alg1.l17.m1.2.2.2.2.1.1.1.4">𝑞</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l17.m1.2c">new\_batch.\text{append}(req)</annotation></semantics></math>

</div>
<div id="alg1.l18" class="ltx_listingline">         <math id="alg1.l18.m1.1" class="ltx_math_unparsed" alttext="delta\leftarrow T.\text{increase\_ref\_counter}(req.prefix\_node)" display="inline"><semantics id="alg1.l18.m1.1a"><mrow id="alg1.l18.m1.1b"><mi id="alg1.l18.m1.1.2">d</mi><mi id="alg1.l18.m1.1.3">e</mi><mi id="alg1.l18.m1.1.4">l</mi><mi id="alg1.l18.m1.1.5">t</mi><mi id="alg1.l18.m1.1.6">a</mi><mo stretchy="false" id="alg1.l18.m1.1.7">←</mo><mi id="alg1.l18.m1.1.8">T</mi><mo lspace="0em" rspace="0.167em" id="alg1.l18.m1.1.9">.</mo><mtext id="alg1.l18.m1.1.1">increase_ref_counter</mtext><mrow id="alg1.l18.m1.1.10"><mo stretchy="false" id="alg1.l18.m1.1.10.1">(</mo><mi id="alg1.l18.m1.1.10.2">r</mi><mi id="alg1.l18.m1.1.10.3">e</mi><mi id="alg1.l18.m1.1.10.4">q</mi><mo lspace="0em" rspace="0.167em" id="alg1.l18.m1.1.10.5">.</mo><mi id="alg1.l18.m1.1.10.6">p</mi><mi id="alg1.l18.m1.1.10.7">r</mi><mi id="alg1.l18.m1.1.10.8">e</mi><mi id="alg1.l18.m1.1.10.9">f</mi><mi id="alg1.l18.m1.1.10.10">i</mi><mi id="alg1.l18.m1.1.10.11">x</mi><mi mathvariant="normal" id="alg1.l18.m1.1.10.12">_</mi><mi id="alg1.l18.m1.1.10.13">n</mi><mi id="alg1.l18.m1.1.10.14">o</mi><mi id="alg1.l18.m1.1.10.15">d</mi><mi id="alg1.l18.m1.1.10.16">e</mi><mo stretchy="false" id="alg1.l18.m1.1.10.17">)</mo></mrow></mrow><annotation encoding="application/x-tex" id="alg1.l18.m1.1c">delta\leftarrow T.\text{increase\_ref\_counter}(req.prefix\_node)</annotation></semantics></math>

</div>
<div id="alg1.l19" class="ltx_listingline">         <math id="alg1.l19.m1.1" class="ltx_Math" alttext="available\_size\leftarrow available\_size+delta" display="inline"><semantics id="alg1.l19.m1.1a"><mrow id="alg1.l19.m1.1.1" xref="alg1.l19.m1.1.1.cmml"><mrow id="alg1.l19.m1.1.1.2" xref="alg1.l19.m1.1.1.2.cmml"><mi id="alg1.l19.m1.1.1.2.2" xref="alg1.l19.m1.1.1.2.2.cmml">a</mi><mo lspace="0em" rspace="0em" id="alg1.l19.m1.1.1.2.1" xref="alg1.l19.m1.1.1.2.1.cmml">​</mo><mi id="alg1.l19.m1.1.1.2.3" xref="alg1.l19.m1.1.1.2.3.cmml">v</mi><mo lspace="0em" rspace="0em" id="alg1.l19.m1.1.1.2.1a" xref="alg1.l19.m1.1.1.2.1.cmml">​</mo><mi id="alg1.l19.m1.1.1.2.4" xref="alg1.l19.m1.1.1.2.4.cmml">a</mi><mo lspace="0em" rspace="0em" id="alg1.l19.m1.1.1.2.1b" xref="alg1.l19.m1.1.1.2.1.cmml">​</mo><mi id="alg1.l19.m1.1.1.2.5" xref="alg1.l19.m1.1.1.2.5.cmml">i</mi><mo lspace="0em" rspace="0em" id="alg1.l19.m1.1.1.2.1c" xref="alg1.l19.m1.1.1.2.1.cmml">​</mo><mi id="alg1.l19.m1.1.1.2.6" xref="alg1.l19.m1.1.1.2.6.cmml">l</mi><mo lspace="0em" rspace="0em" id="alg1.l19.m1.1.1.2.1d" xref="alg1.l19.m1.1.1.2.1.cmml">​</mo><mi id="alg1.l19.m1.1.1.2.7" xref="alg1.l19.m1.1.1.2.7.cmml">a</mi><mo lspace="0em" rspace="0em" id="alg1.l19.m1.1.1.2.1e" xref="alg1.l19.m1.1.1.2.1.cmml">​</mo><mi id="alg1.l19.m1.1.1.2.8" xref="alg1.l19.m1.1.1.2.8.cmml">b</mi><mo lspace="0em" rspace="0em" id="alg1.l19.m1.1.1.2.1f" xref="alg1.l19.m1.1.1.2.1.cmml">​</mo><mi id="alg1.l19.m1.1.1.2.9" xref="alg1.l19.m1.1.1.2.9.cmml">l</mi><mo lspace="0em" rspace="0em" id="alg1.l19.m1.1.1.2.1g" xref="alg1.l19.m1.1.1.2.1.cmml">​</mo><mi id="alg1.l19.m1.1.1.2.10" xref="alg1.l19.m1.1.1.2.10.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l19.m1.1.1.2.1h" xref="alg1.l19.m1.1.1.2.1.cmml">​</mo><mi mathvariant="normal" id="alg1.l19.m1.1.1.2.11" xref="alg1.l19.m1.1.1.2.11.cmml">_</mi><mo lspace="0em" rspace="0em" id="alg1.l19.m1.1.1.2.1i" xref="alg1.l19.m1.1.1.2.1.cmml">​</mo><mi id="alg1.l19.m1.1.1.2.12" xref="alg1.l19.m1.1.1.2.12.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg1.l19.m1.1.1.2.1j" xref="alg1.l19.m1.1.1.2.1.cmml">​</mo><mi id="alg1.l19.m1.1.1.2.13" xref="alg1.l19.m1.1.1.2.13.cmml">i</mi><mo lspace="0em" rspace="0em" id="alg1.l19.m1.1.1.2.1k" xref="alg1.l19.m1.1.1.2.1.cmml">​</mo><mi id="alg1.l19.m1.1.1.2.14" xref="alg1.l19.m1.1.1.2.14.cmml">z</mi><mo lspace="0em" rspace="0em" id="alg1.l19.m1.1.1.2.1l" xref="alg1.l19.m1.1.1.2.1.cmml">​</mo><mi id="alg1.l19.m1.1.1.2.15" xref="alg1.l19.m1.1.1.2.15.cmml">e</mi></mrow><mo stretchy="false" id="alg1.l19.m1.1.1.1" xref="alg1.l19.m1.1.1.1.cmml">←</mo><mrow id="alg1.l19.m1.1.1.3" xref="alg1.l19.m1.1.1.3.cmml"><mrow id="alg1.l19.m1.1.1.3.2" xref="alg1.l19.m1.1.1.3.2.cmml"><mi id="alg1.l19.m1.1.1.3.2.2" xref="alg1.l19.m1.1.1.3.2.2.cmml">a</mi><mo lspace="0em" rspace="0em" id="alg1.l19.m1.1.1.3.2.1" xref="alg1.l19.m1.1.1.3.2.1.cmml">​</mo><mi id="alg1.l19.m1.1.1.3.2.3" xref="alg1.l19.m1.1.1.3.2.3.cmml">v</mi><mo lspace="0em" rspace="0em" id="alg1.l19.m1.1.1.3.2.1a" xref="alg1.l19.m1.1.1.3.2.1.cmml">​</mo><mi id="alg1.l19.m1.1.1.3.2.4" xref="alg1.l19.m1.1.1.3.2.4.cmml">a</mi><mo lspace="0em" rspace="0em" id="alg1.l19.m1.1.1.3.2.1b" xref="alg1.l19.m1.1.1.3.2.1.cmml">​</mo><mi id="alg1.l19.m1.1.1.3.2.5" xref="alg1.l19.m1.1.1.3.2.5.cmml">i</mi><mo lspace="0em" rspace="0em" id="alg1.l19.m1.1.1.3.2.1c" xref="alg1.l19.m1.1.1.3.2.1.cmml">​</mo><mi id="alg1.l19.m1.1.1.3.2.6" xref="alg1.l19.m1.1.1.3.2.6.cmml">l</mi><mo lspace="0em" rspace="0em" id="alg1.l19.m1.1.1.3.2.1d" xref="alg1.l19.m1.1.1.3.2.1.cmml">​</mo><mi id="alg1.l19.m1.1.1.3.2.7" xref="alg1.l19.m1.1.1.3.2.7.cmml">a</mi><mo lspace="0em" rspace="0em" id="alg1.l19.m1.1.1.3.2.1e" xref="alg1.l19.m1.1.1.3.2.1.cmml">​</mo><mi id="alg1.l19.m1.1.1.3.2.8" xref="alg1.l19.m1.1.1.3.2.8.cmml">b</mi><mo lspace="0em" rspace="0em" id="alg1.l19.m1.1.1.3.2.1f" xref="alg1.l19.m1.1.1.3.2.1.cmml">​</mo><mi id="alg1.l19.m1.1.1.3.2.9" xref="alg1.l19.m1.1.1.3.2.9.cmml">l</mi><mo lspace="0em" rspace="0em" id="alg1.l19.m1.1.1.3.2.1g" xref="alg1.l19.m1.1.1.3.2.1.cmml">​</mo><mi id="alg1.l19.m1.1.1.3.2.10" xref="alg1.l19.m1.1.1.3.2.10.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l19.m1.1.1.3.2.1h" xref="alg1.l19.m1.1.1.3.2.1.cmml">​</mo><mi mathvariant="normal" id="alg1.l19.m1.1.1.3.2.11" xref="alg1.l19.m1.1.1.3.2.11.cmml">_</mi><mo lspace="0em" rspace="0em" id="alg1.l19.m1.1.1.3.2.1i" xref="alg1.l19.m1.1.1.3.2.1.cmml">​</mo><mi id="alg1.l19.m1.1.1.3.2.12" xref="alg1.l19.m1.1.1.3.2.12.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg1.l19.m1.1.1.3.2.1j" xref="alg1.l19.m1.1.1.3.2.1.cmml">​</mo><mi id="alg1.l19.m1.1.1.3.2.13" xref="alg1.l19.m1.1.1.3.2.13.cmml">i</mi><mo lspace="0em" rspace="0em" id="alg1.l19.m1.1.1.3.2.1k" xref="alg1.l19.m1.1.1.3.2.1.cmml">​</mo><mi id="alg1.l19.m1.1.1.3.2.14" xref="alg1.l19.m1.1.1.3.2.14.cmml">z</mi><mo lspace="0em" rspace="0em" id="alg1.l19.m1.1.1.3.2.1l" xref="alg1.l19.m1.1.1.3.2.1.cmml">​</mo><mi id="alg1.l19.m1.1.1.3.2.15" xref="alg1.l19.m1.1.1.3.2.15.cmml">e</mi></mrow><mo id="alg1.l19.m1.1.1.3.1" xref="alg1.l19.m1.1.1.3.1.cmml">+</mo><mrow id="alg1.l19.m1.1.1.3.3" xref="alg1.l19.m1.1.1.3.3.cmml"><mi id="alg1.l19.m1.1.1.3.3.2" xref="alg1.l19.m1.1.1.3.3.2.cmml">d</mi><mo lspace="0em" rspace="0em" id="alg1.l19.m1.1.1.3.3.1" xref="alg1.l19.m1.1.1.3.3.1.cmml">​</mo><mi id="alg1.l19.m1.1.1.3.3.3" xref="alg1.l19.m1.1.1.3.3.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l19.m1.1.1.3.3.1a" xref="alg1.l19.m1.1.1.3.3.1.cmml">​</mo><mi id="alg1.l19.m1.1.1.3.3.4" xref="alg1.l19.m1.1.1.3.3.4.cmml">l</mi><mo lspace="0em" rspace="0em" id="alg1.l19.m1.1.1.3.3.1b" xref="alg1.l19.m1.1.1.3.3.1.cmml">​</mo><mi id="alg1.l19.m1.1.1.3.3.5" xref="alg1.l19.m1.1.1.3.3.5.cmml">t</mi><mo lspace="0em" rspace="0em" id="alg1.l19.m1.1.1.3.3.1c" xref="alg1.l19.m1.1.1.3.3.1.cmml">​</mo><mi id="alg1.l19.m1.1.1.3.3.6" xref="alg1.l19.m1.1.1.3.3.6.cmml">a</mi></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l19.m1.1b"><apply id="alg1.l19.m1.1.1.cmml" xref="alg1.l19.m1.1.1"><ci id="alg1.l19.m1.1.1.1.cmml" xref="alg1.l19.m1.1.1.1">←</ci><apply id="alg1.l19.m1.1.1.2.cmml" xref="alg1.l19.m1.1.1.2"><times id="alg1.l19.m1.1.1.2.1.cmml" xref="alg1.l19.m1.1.1.2.1"></times><ci id="alg1.l19.m1.1.1.2.2.cmml" xref="alg1.l19.m1.1.1.2.2">𝑎</ci><ci id="alg1.l19.m1.1.1.2.3.cmml" xref="alg1.l19.m1.1.1.2.3">𝑣</ci><ci id="alg1.l19.m1.1.1.2.4.cmml" xref="alg1.l19.m1.1.1.2.4">𝑎</ci><ci id="alg1.l19.m1.1.1.2.5.cmml" xref="alg1.l19.m1.1.1.2.5">𝑖</ci><ci id="alg1.l19.m1.1.1.2.6.cmml" xref="alg1.l19.m1.1.1.2.6">𝑙</ci><ci id="alg1.l19.m1.1.1.2.7.cmml" xref="alg1.l19.m1.1.1.2.7">𝑎</ci><ci id="alg1.l19.m1.1.1.2.8.cmml" xref="alg1.l19.m1.1.1.2.8">𝑏</ci><ci id="alg1.l19.m1.1.1.2.9.cmml" xref="alg1.l19.m1.1.1.2.9">𝑙</ci><ci id="alg1.l19.m1.1.1.2.10.cmml" xref="alg1.l19.m1.1.1.2.10">𝑒</ci><ci id="alg1.l19.m1.1.1.2.11.cmml" xref="alg1.l19.m1.1.1.2.11">_</ci><ci id="alg1.l19.m1.1.1.2.12.cmml" xref="alg1.l19.m1.1.1.2.12">𝑠</ci><ci id="alg1.l19.m1.1.1.2.13.cmml" xref="alg1.l19.m1.1.1.2.13">𝑖</ci><ci id="alg1.l19.m1.1.1.2.14.cmml" xref="alg1.l19.m1.1.1.2.14">𝑧</ci><ci id="alg1.l19.m1.1.1.2.15.cmml" xref="alg1.l19.m1.1.1.2.15">𝑒</ci></apply><apply id="alg1.l19.m1.1.1.3.cmml" xref="alg1.l19.m1.1.1.3"><plus id="alg1.l19.m1.1.1.3.1.cmml" xref="alg1.l19.m1.1.1.3.1"></plus><apply id="alg1.l19.m1.1.1.3.2.cmml" xref="alg1.l19.m1.1.1.3.2"><times id="alg1.l19.m1.1.1.3.2.1.cmml" xref="alg1.l19.m1.1.1.3.2.1"></times><ci id="alg1.l19.m1.1.1.3.2.2.cmml" xref="alg1.l19.m1.1.1.3.2.2">𝑎</ci><ci id="alg1.l19.m1.1.1.3.2.3.cmml" xref="alg1.l19.m1.1.1.3.2.3">𝑣</ci><ci id="alg1.l19.m1.1.1.3.2.4.cmml" xref="alg1.l19.m1.1.1.3.2.4">𝑎</ci><ci id="alg1.l19.m1.1.1.3.2.5.cmml" xref="alg1.l19.m1.1.1.3.2.5">𝑖</ci><ci id="alg1.l19.m1.1.1.3.2.6.cmml" xref="alg1.l19.m1.1.1.3.2.6">𝑙</ci><ci id="alg1.l19.m1.1.1.3.2.7.cmml" xref="alg1.l19.m1.1.1.3.2.7">𝑎</ci><ci id="alg1.l19.m1.1.1.3.2.8.cmml" xref="alg1.l19.m1.1.1.3.2.8">𝑏</ci><ci id="alg1.l19.m1.1.1.3.2.9.cmml" xref="alg1.l19.m1.1.1.3.2.9">𝑙</ci><ci id="alg1.l19.m1.1.1.3.2.10.cmml" xref="alg1.l19.m1.1.1.3.2.10">𝑒</ci><ci id="alg1.l19.m1.1.1.3.2.11.cmml" xref="alg1.l19.m1.1.1.3.2.11">_</ci><ci id="alg1.l19.m1.1.1.3.2.12.cmml" xref="alg1.l19.m1.1.1.3.2.12">𝑠</ci><ci id="alg1.l19.m1.1.1.3.2.13.cmml" xref="alg1.l19.m1.1.1.3.2.13">𝑖</ci><ci id="alg1.l19.m1.1.1.3.2.14.cmml" xref="alg1.l19.m1.1.1.3.2.14">𝑧</ci><ci id="alg1.l19.m1.1.1.3.2.15.cmml" xref="alg1.l19.m1.1.1.3.2.15">𝑒</ci></apply><apply id="alg1.l19.m1.1.1.3.3.cmml" xref="alg1.l19.m1.1.1.3.3"><times id="alg1.l19.m1.1.1.3.3.1.cmml" xref="alg1.l19.m1.1.1.3.3.1"></times><ci id="alg1.l19.m1.1.1.3.3.2.cmml" xref="alg1.l19.m1.1.1.3.3.2">𝑑</ci><ci id="alg1.l19.m1.1.1.3.3.3.cmml" xref="alg1.l19.m1.1.1.3.3.3">𝑒</ci><ci id="alg1.l19.m1.1.1.3.3.4.cmml" xref="alg1.l19.m1.1.1.3.3.4">𝑙</ci><ci id="alg1.l19.m1.1.1.3.3.5.cmml" xref="alg1.l19.m1.1.1.3.3.5">𝑡</ci><ci id="alg1.l19.m1.1.1.3.3.6.cmml" xref="alg1.l19.m1.1.1.3.3.6">𝑎</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l19.m1.1c">available\_size\leftarrow available\_size+delta</annotation></semantics></math>
     
</div>
<div id="alg1.l20" class="ltx_listingline">
<math id="alg1.l20.m1.2" class="ltx_Math" alttext="Q.\text{remove\_requests}(new\_batch)" display="inline"><semantics id="alg1.l20.m1.2a"><mrow id="alg1.l20.m1.2.2.1" xref="alg1.l20.m1.2.2.2.cmml"><mi id="alg1.l20.m1.1.1" xref="alg1.l20.m1.1.1.cmml">Q</mi><mo lspace="0em" rspace="0.167em" id="alg1.l20.m1.2.2.1.2" xref="alg1.l20.m1.2.2.2a.cmml">.</mo><mrow id="alg1.l20.m1.2.2.1.1" xref="alg1.l20.m1.2.2.1.1.cmml"><mtext id="alg1.l20.m1.2.2.1.1.3" xref="alg1.l20.m1.2.2.1.1.3a.cmml">remove_requests</mtext><mo lspace="0em" rspace="0em" id="alg1.l20.m1.2.2.1.1.2" xref="alg1.l20.m1.2.2.1.1.2.cmml">​</mo><mrow id="alg1.l20.m1.2.2.1.1.1.1" xref="alg1.l20.m1.2.2.1.1.1.1.1.cmml"><mo stretchy="false" id="alg1.l20.m1.2.2.1.1.1.1.2" xref="alg1.l20.m1.2.2.1.1.1.1.1.cmml">(</mo><mrow id="alg1.l20.m1.2.2.1.1.1.1.1" xref="alg1.l20.m1.2.2.1.1.1.1.1.cmml"><mi id="alg1.l20.m1.2.2.1.1.1.1.1.2" xref="alg1.l20.m1.2.2.1.1.1.1.1.2.cmml">n</mi><mo lspace="0em" rspace="0em" id="alg1.l20.m1.2.2.1.1.1.1.1.1" xref="alg1.l20.m1.2.2.1.1.1.1.1.1.cmml">​</mo><mi id="alg1.l20.m1.2.2.1.1.1.1.1.3" xref="alg1.l20.m1.2.2.1.1.1.1.1.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l20.m1.2.2.1.1.1.1.1.1a" xref="alg1.l20.m1.2.2.1.1.1.1.1.1.cmml">​</mo><mi id="alg1.l20.m1.2.2.1.1.1.1.1.4" xref="alg1.l20.m1.2.2.1.1.1.1.1.4.cmml">w</mi><mo lspace="0em" rspace="0em" id="alg1.l20.m1.2.2.1.1.1.1.1.1b" xref="alg1.l20.m1.2.2.1.1.1.1.1.1.cmml">​</mo><mi mathvariant="normal" id="alg1.l20.m1.2.2.1.1.1.1.1.5" xref="alg1.l20.m1.2.2.1.1.1.1.1.5.cmml">_</mi><mo lspace="0em" rspace="0em" id="alg1.l20.m1.2.2.1.1.1.1.1.1c" xref="alg1.l20.m1.2.2.1.1.1.1.1.1.cmml">​</mo><mi id="alg1.l20.m1.2.2.1.1.1.1.1.6" xref="alg1.l20.m1.2.2.1.1.1.1.1.6.cmml">b</mi><mo lspace="0em" rspace="0em" id="alg1.l20.m1.2.2.1.1.1.1.1.1d" xref="alg1.l20.m1.2.2.1.1.1.1.1.1.cmml">​</mo><mi id="alg1.l20.m1.2.2.1.1.1.1.1.7" xref="alg1.l20.m1.2.2.1.1.1.1.1.7.cmml">a</mi><mo lspace="0em" rspace="0em" id="alg1.l20.m1.2.2.1.1.1.1.1.1e" xref="alg1.l20.m1.2.2.1.1.1.1.1.1.cmml">​</mo><mi id="alg1.l20.m1.2.2.1.1.1.1.1.8" xref="alg1.l20.m1.2.2.1.1.1.1.1.8.cmml">t</mi><mo lspace="0em" rspace="0em" id="alg1.l20.m1.2.2.1.1.1.1.1.1f" xref="alg1.l20.m1.2.2.1.1.1.1.1.1.cmml">​</mo><mi id="alg1.l20.m1.2.2.1.1.1.1.1.9" xref="alg1.l20.m1.2.2.1.1.1.1.1.9.cmml">c</mi><mo lspace="0em" rspace="0em" id="alg1.l20.m1.2.2.1.1.1.1.1.1g" xref="alg1.l20.m1.2.2.1.1.1.1.1.1.cmml">​</mo><mi id="alg1.l20.m1.2.2.1.1.1.1.1.10" xref="alg1.l20.m1.2.2.1.1.1.1.1.10.cmml">h</mi></mrow><mo stretchy="false" id="alg1.l20.m1.2.2.1.1.1.1.3" xref="alg1.l20.m1.2.2.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l20.m1.2b"><apply id="alg1.l20.m1.2.2.2.cmml" xref="alg1.l20.m1.2.2.1"><csymbol cd="ambiguous" id="alg1.l20.m1.2.2.2a.cmml" xref="alg1.l20.m1.2.2.1.2">formulae-sequence</csymbol><ci id="alg1.l20.m1.1.1.cmml" xref="alg1.l20.m1.1.1">𝑄</ci><apply id="alg1.l20.m1.2.2.1.1.cmml" xref="alg1.l20.m1.2.2.1.1"><times id="alg1.l20.m1.2.2.1.1.2.cmml" xref="alg1.l20.m1.2.2.1.1.2"></times><ci id="alg1.l20.m1.2.2.1.1.3a.cmml" xref="alg1.l20.m1.2.2.1.1.3"><mtext id="alg1.l20.m1.2.2.1.1.3.cmml" xref="alg1.l20.m1.2.2.1.1.3">remove_requests</mtext></ci><apply id="alg1.l20.m1.2.2.1.1.1.1.1.cmml" xref="alg1.l20.m1.2.2.1.1.1.1"><times id="alg1.l20.m1.2.2.1.1.1.1.1.1.cmml" xref="alg1.l20.m1.2.2.1.1.1.1.1.1"></times><ci id="alg1.l20.m1.2.2.1.1.1.1.1.2.cmml" xref="alg1.l20.m1.2.2.1.1.1.1.1.2">𝑛</ci><ci id="alg1.l20.m1.2.2.1.1.1.1.1.3.cmml" xref="alg1.l20.m1.2.2.1.1.1.1.1.3">𝑒</ci><ci id="alg1.l20.m1.2.2.1.1.1.1.1.4.cmml" xref="alg1.l20.m1.2.2.1.1.1.1.1.4">𝑤</ci><ci id="alg1.l20.m1.2.2.1.1.1.1.1.5.cmml" xref="alg1.l20.m1.2.2.1.1.1.1.1.5">_</ci><ci id="alg1.l20.m1.2.2.1.1.1.1.1.6.cmml" xref="alg1.l20.m1.2.2.1.1.1.1.1.6">𝑏</ci><ci id="alg1.l20.m1.2.2.1.1.1.1.1.7.cmml" xref="alg1.l20.m1.2.2.1.1.1.1.1.7">𝑎</ci><ci id="alg1.l20.m1.2.2.1.1.1.1.1.8.cmml" xref="alg1.l20.m1.2.2.1.1.1.1.1.8">𝑡</ci><ci id="alg1.l20.m1.2.2.1.1.1.1.1.9.cmml" xref="alg1.l20.m1.2.2.1.1.1.1.1.9">𝑐</ci><ci id="alg1.l20.m1.2.2.1.1.1.1.1.10.cmml" xref="alg1.l20.m1.2.2.1.1.1.1.1.10">ℎ</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l20.m1.2c">Q.\text{remove\_requests}(new\_batch)</annotation></semantics></math>

</div>
<div id="alg1.l21" class="ltx_listingline">
<span id="alg1.l21.1" class="ltx_text">// Insert requests into the current running batch</span>

</div>
<div id="alg1.l22" class="ltx_listingline">
<math id="alg1.l22.m1.2" class="ltx_Math" alttext="B.\text{merge}(new\_batch)" display="inline"><semantics id="alg1.l22.m1.2a"><mrow id="alg1.l22.m1.2.2.1" xref="alg1.l22.m1.2.2.2.cmml"><mi id="alg1.l22.m1.1.1" xref="alg1.l22.m1.1.1.cmml">B</mi><mo lspace="0em" rspace="0.167em" id="alg1.l22.m1.2.2.1.2" xref="alg1.l22.m1.2.2.2a.cmml">.</mo><mrow id="alg1.l22.m1.2.2.1.1" xref="alg1.l22.m1.2.2.1.1.cmml"><mtext id="alg1.l22.m1.2.2.1.1.3" xref="alg1.l22.m1.2.2.1.1.3a.cmml">merge</mtext><mo lspace="0em" rspace="0em" id="alg1.l22.m1.2.2.1.1.2" xref="alg1.l22.m1.2.2.1.1.2.cmml">​</mo><mrow id="alg1.l22.m1.2.2.1.1.1.1" xref="alg1.l22.m1.2.2.1.1.1.1.1.cmml"><mo stretchy="false" id="alg1.l22.m1.2.2.1.1.1.1.2" xref="alg1.l22.m1.2.2.1.1.1.1.1.cmml">(</mo><mrow id="alg1.l22.m1.2.2.1.1.1.1.1" xref="alg1.l22.m1.2.2.1.1.1.1.1.cmml"><mi id="alg1.l22.m1.2.2.1.1.1.1.1.2" xref="alg1.l22.m1.2.2.1.1.1.1.1.2.cmml">n</mi><mo lspace="0em" rspace="0em" id="alg1.l22.m1.2.2.1.1.1.1.1.1" xref="alg1.l22.m1.2.2.1.1.1.1.1.1.cmml">​</mo><mi id="alg1.l22.m1.2.2.1.1.1.1.1.3" xref="alg1.l22.m1.2.2.1.1.1.1.1.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l22.m1.2.2.1.1.1.1.1.1a" xref="alg1.l22.m1.2.2.1.1.1.1.1.1.cmml">​</mo><mi id="alg1.l22.m1.2.2.1.1.1.1.1.4" xref="alg1.l22.m1.2.2.1.1.1.1.1.4.cmml">w</mi><mo lspace="0em" rspace="0em" id="alg1.l22.m1.2.2.1.1.1.1.1.1b" xref="alg1.l22.m1.2.2.1.1.1.1.1.1.cmml">​</mo><mi mathvariant="normal" id="alg1.l22.m1.2.2.1.1.1.1.1.5" xref="alg1.l22.m1.2.2.1.1.1.1.1.5.cmml">_</mi><mo lspace="0em" rspace="0em" id="alg1.l22.m1.2.2.1.1.1.1.1.1c" xref="alg1.l22.m1.2.2.1.1.1.1.1.1.cmml">​</mo><mi id="alg1.l22.m1.2.2.1.1.1.1.1.6" xref="alg1.l22.m1.2.2.1.1.1.1.1.6.cmml">b</mi><mo lspace="0em" rspace="0em" id="alg1.l22.m1.2.2.1.1.1.1.1.1d" xref="alg1.l22.m1.2.2.1.1.1.1.1.1.cmml">​</mo><mi id="alg1.l22.m1.2.2.1.1.1.1.1.7" xref="alg1.l22.m1.2.2.1.1.1.1.1.7.cmml">a</mi><mo lspace="0em" rspace="0em" id="alg1.l22.m1.2.2.1.1.1.1.1.1e" xref="alg1.l22.m1.2.2.1.1.1.1.1.1.cmml">​</mo><mi id="alg1.l22.m1.2.2.1.1.1.1.1.8" xref="alg1.l22.m1.2.2.1.1.1.1.1.8.cmml">t</mi><mo lspace="0em" rspace="0em" id="alg1.l22.m1.2.2.1.1.1.1.1.1f" xref="alg1.l22.m1.2.2.1.1.1.1.1.1.cmml">​</mo><mi id="alg1.l22.m1.2.2.1.1.1.1.1.9" xref="alg1.l22.m1.2.2.1.1.1.1.1.9.cmml">c</mi><mo lspace="0em" rspace="0em" id="alg1.l22.m1.2.2.1.1.1.1.1.1g" xref="alg1.l22.m1.2.2.1.1.1.1.1.1.cmml">​</mo><mi id="alg1.l22.m1.2.2.1.1.1.1.1.10" xref="alg1.l22.m1.2.2.1.1.1.1.1.10.cmml">h</mi></mrow><mo stretchy="false" id="alg1.l22.m1.2.2.1.1.1.1.3" xref="alg1.l22.m1.2.2.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l22.m1.2b"><apply id="alg1.l22.m1.2.2.2.cmml" xref="alg1.l22.m1.2.2.1"><csymbol cd="ambiguous" id="alg1.l22.m1.2.2.2a.cmml" xref="alg1.l22.m1.2.2.1.2">formulae-sequence</csymbol><ci id="alg1.l22.m1.1.1.cmml" xref="alg1.l22.m1.1.1">𝐵</ci><apply id="alg1.l22.m1.2.2.1.1.cmml" xref="alg1.l22.m1.2.2.1.1"><times id="alg1.l22.m1.2.2.1.1.2.cmml" xref="alg1.l22.m1.2.2.1.1.2"></times><ci id="alg1.l22.m1.2.2.1.1.3a.cmml" xref="alg1.l22.m1.2.2.1.1.3"><mtext id="alg1.l22.m1.2.2.1.1.3.cmml" xref="alg1.l22.m1.2.2.1.1.3">merge</mtext></ci><apply id="alg1.l22.m1.2.2.1.1.1.1.1.cmml" xref="alg1.l22.m1.2.2.1.1.1.1"><times id="alg1.l22.m1.2.2.1.1.1.1.1.1.cmml" xref="alg1.l22.m1.2.2.1.1.1.1.1.1"></times><ci id="alg1.l22.m1.2.2.1.1.1.1.1.2.cmml" xref="alg1.l22.m1.2.2.1.1.1.1.1.2">𝑛</ci><ci id="alg1.l22.m1.2.2.1.1.1.1.1.3.cmml" xref="alg1.l22.m1.2.2.1.1.1.1.1.3">𝑒</ci><ci id="alg1.l22.m1.2.2.1.1.1.1.1.4.cmml" xref="alg1.l22.m1.2.2.1.1.1.1.1.4">𝑤</ci><ci id="alg1.l22.m1.2.2.1.1.1.1.1.5.cmml" xref="alg1.l22.m1.2.2.1.1.1.1.1.5">_</ci><ci id="alg1.l22.m1.2.2.1.1.1.1.1.6.cmml" xref="alg1.l22.m1.2.2.1.1.1.1.1.6">𝑏</ci><ci id="alg1.l22.m1.2.2.1.1.1.1.1.7.cmml" xref="alg1.l22.m1.2.2.1.1.1.1.1.7">𝑎</ci><ci id="alg1.l22.m1.2.2.1.1.1.1.1.8.cmml" xref="alg1.l22.m1.2.2.1.1.1.1.1.8">𝑡</ci><ci id="alg1.l22.m1.2.2.1.1.1.1.1.9.cmml" xref="alg1.l22.m1.2.2.1.1.1.1.1.9">𝑐</ci><ci id="alg1.l22.m1.2.2.1.1.1.1.1.10.cmml" xref="alg1.l22.m1.2.2.1.1.1.1.1.10">ℎ</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l22.m1.2c">B.\text{merge}(new\_batch)</annotation></semantics></math>

</div>
<div id="alg1.l23" class="ltx_listingline">
<span id="alg1.l23.1" class="ltx_text">// Allocate new memory and do eviction if necessary</span>

</div>
<div id="alg1.l24" class="ltx_listingline">
<math id="alg1.l24.m1.2" class="ltx_Math" alttext="needed\_size\leftarrow B.\text{needed\_size}()" display="inline"><semantics id="alg1.l24.m1.2a"><mrow id="alg1.l24.m1.2.2.2" xref="alg1.l24.m1.2.2.3.cmml"><mrow id="alg1.l24.m1.1.1.1.1" xref="alg1.l24.m1.1.1.1.1.cmml"><mrow id="alg1.l24.m1.1.1.1.1.2" xref="alg1.l24.m1.1.1.1.1.2.cmml"><mi id="alg1.l24.m1.1.1.1.1.2.2" xref="alg1.l24.m1.1.1.1.1.2.2.cmml">n</mi><mo lspace="0em" rspace="0em" id="alg1.l24.m1.1.1.1.1.2.1" xref="alg1.l24.m1.1.1.1.1.2.1.cmml">​</mo><mi id="alg1.l24.m1.1.1.1.1.2.3" xref="alg1.l24.m1.1.1.1.1.2.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l24.m1.1.1.1.1.2.1a" xref="alg1.l24.m1.1.1.1.1.2.1.cmml">​</mo><mi id="alg1.l24.m1.1.1.1.1.2.4" xref="alg1.l24.m1.1.1.1.1.2.4.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l24.m1.1.1.1.1.2.1b" xref="alg1.l24.m1.1.1.1.1.2.1.cmml">​</mo><mi id="alg1.l24.m1.1.1.1.1.2.5" xref="alg1.l24.m1.1.1.1.1.2.5.cmml">d</mi><mo lspace="0em" rspace="0em" id="alg1.l24.m1.1.1.1.1.2.1c" xref="alg1.l24.m1.1.1.1.1.2.1.cmml">​</mo><mi id="alg1.l24.m1.1.1.1.1.2.6" xref="alg1.l24.m1.1.1.1.1.2.6.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l24.m1.1.1.1.1.2.1d" xref="alg1.l24.m1.1.1.1.1.2.1.cmml">​</mo><mi id="alg1.l24.m1.1.1.1.1.2.7" xref="alg1.l24.m1.1.1.1.1.2.7.cmml">d</mi><mo lspace="0em" rspace="0em" id="alg1.l24.m1.1.1.1.1.2.1e" xref="alg1.l24.m1.1.1.1.1.2.1.cmml">​</mo><mi mathvariant="normal" id="alg1.l24.m1.1.1.1.1.2.8" xref="alg1.l24.m1.1.1.1.1.2.8.cmml">_</mi><mo lspace="0em" rspace="0em" id="alg1.l24.m1.1.1.1.1.2.1f" xref="alg1.l24.m1.1.1.1.1.2.1.cmml">​</mo><mi id="alg1.l24.m1.1.1.1.1.2.9" xref="alg1.l24.m1.1.1.1.1.2.9.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg1.l24.m1.1.1.1.1.2.1g" xref="alg1.l24.m1.1.1.1.1.2.1.cmml">​</mo><mi id="alg1.l24.m1.1.1.1.1.2.10" xref="alg1.l24.m1.1.1.1.1.2.10.cmml">i</mi><mo lspace="0em" rspace="0em" id="alg1.l24.m1.1.1.1.1.2.1h" xref="alg1.l24.m1.1.1.1.1.2.1.cmml">​</mo><mi id="alg1.l24.m1.1.1.1.1.2.11" xref="alg1.l24.m1.1.1.1.1.2.11.cmml">z</mi><mo lspace="0em" rspace="0em" id="alg1.l24.m1.1.1.1.1.2.1i" xref="alg1.l24.m1.1.1.1.1.2.1.cmml">​</mo><mi id="alg1.l24.m1.1.1.1.1.2.12" xref="alg1.l24.m1.1.1.1.1.2.12.cmml">e</mi></mrow><mo stretchy="false" id="alg1.l24.m1.1.1.1.1.1" xref="alg1.l24.m1.1.1.1.1.1.cmml">←</mo><mi id="alg1.l24.m1.1.1.1.1.3" xref="alg1.l24.m1.1.1.1.1.3.cmml">B</mi></mrow><mo lspace="0em" rspace="0.167em" id="alg1.l24.m1.2.2.2.3" xref="alg1.l24.m1.2.2.3a.cmml">.</mo><mrow id="alg1.l24.m1.2.2.2.2" xref="alg1.l24.m1.2.2.2.2.cmml"><mtext id="alg1.l24.m1.2.2.2.2.2" xref="alg1.l24.m1.2.2.2.2.2a.cmml">needed_size</mtext><mo lspace="0em" rspace="0em" id="alg1.l24.m1.2.2.2.2.1" xref="alg1.l24.m1.2.2.2.2.1.cmml">​</mo><mrow id="alg1.l24.m1.2.2.2.2.3.2" xref="alg1.l24.m1.2.2.2.2.cmml"><mo stretchy="false" id="alg1.l24.m1.2.2.2.2.3.2.1" xref="alg1.l24.m1.2.2.2.2.3.1.cmml">(</mo><mo stretchy="false" id="alg1.l24.m1.2.2.2.2.3.2.2" xref="alg1.l24.m1.2.2.2.2.3.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l24.m1.2b"><apply id="alg1.l24.m1.2.2.3.cmml" xref="alg1.l24.m1.2.2.2"><csymbol cd="ambiguous" id="alg1.l24.m1.2.2.3a.cmml" xref="alg1.l24.m1.2.2.2.3">formulae-sequence</csymbol><apply id="alg1.l24.m1.1.1.1.1.cmml" xref="alg1.l24.m1.1.1.1.1"><ci id="alg1.l24.m1.1.1.1.1.1.cmml" xref="alg1.l24.m1.1.1.1.1.1">←</ci><apply id="alg1.l24.m1.1.1.1.1.2.cmml" xref="alg1.l24.m1.1.1.1.1.2"><times id="alg1.l24.m1.1.1.1.1.2.1.cmml" xref="alg1.l24.m1.1.1.1.1.2.1"></times><ci id="alg1.l24.m1.1.1.1.1.2.2.cmml" xref="alg1.l24.m1.1.1.1.1.2.2">𝑛</ci><ci id="alg1.l24.m1.1.1.1.1.2.3.cmml" xref="alg1.l24.m1.1.1.1.1.2.3">𝑒</ci><ci id="alg1.l24.m1.1.1.1.1.2.4.cmml" xref="alg1.l24.m1.1.1.1.1.2.4">𝑒</ci><ci id="alg1.l24.m1.1.1.1.1.2.5.cmml" xref="alg1.l24.m1.1.1.1.1.2.5">𝑑</ci><ci id="alg1.l24.m1.1.1.1.1.2.6.cmml" xref="alg1.l24.m1.1.1.1.1.2.6">𝑒</ci><ci id="alg1.l24.m1.1.1.1.1.2.7.cmml" xref="alg1.l24.m1.1.1.1.1.2.7">𝑑</ci><ci id="alg1.l24.m1.1.1.1.1.2.8.cmml" xref="alg1.l24.m1.1.1.1.1.2.8">_</ci><ci id="alg1.l24.m1.1.1.1.1.2.9.cmml" xref="alg1.l24.m1.1.1.1.1.2.9">𝑠</ci><ci id="alg1.l24.m1.1.1.1.1.2.10.cmml" xref="alg1.l24.m1.1.1.1.1.2.10">𝑖</ci><ci id="alg1.l24.m1.1.1.1.1.2.11.cmml" xref="alg1.l24.m1.1.1.1.1.2.11">𝑧</ci><ci id="alg1.l24.m1.1.1.1.1.2.12.cmml" xref="alg1.l24.m1.1.1.1.1.2.12">𝑒</ci></apply><ci id="alg1.l24.m1.1.1.1.1.3.cmml" xref="alg1.l24.m1.1.1.1.1.3">𝐵</ci></apply><apply id="alg1.l24.m1.2.2.2.2.cmml" xref="alg1.l24.m1.2.2.2.2"><times id="alg1.l24.m1.2.2.2.2.1.cmml" xref="alg1.l24.m1.2.2.2.2.1"></times><ci id="alg1.l24.m1.2.2.2.2.2a.cmml" xref="alg1.l24.m1.2.2.2.2.2"><mtext id="alg1.l24.m1.2.2.2.2.2.cmml" xref="alg1.l24.m1.2.2.2.2.2">needed_size</mtext></ci><list id="alg1.l24.m1.2.2.2.2.3.1.cmml" xref="alg1.l24.m1.2.2.2.2.3.2.1"></list></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l24.m1.2c">needed\_size\leftarrow B.\text{needed\_size}()</annotation></semantics></math>

</div>
<div id="alg1.l25" class="ltx_listingline">
<math id="alg1.l25.m1.2" class="ltx_Math" alttext="success,buffer\leftarrow P.\text{alloc}(needed\_size)" display="inline"><semantics id="alg1.l25.m1.2a"><mrow id="alg1.l25.m1.2.2.2" xref="alg1.l25.m1.2.2.3.cmml"><mrow id="alg1.l25.m1.1.1.1.1" xref="alg1.l25.m1.1.1.1.1.cmml"><mrow id="alg1.l25.m1.1.1.1.1.2.2" xref="alg1.l25.m1.1.1.1.1.2.3.cmml"><mrow id="alg1.l25.m1.1.1.1.1.1.1.1" xref="alg1.l25.m1.1.1.1.1.1.1.1.cmml"><mi id="alg1.l25.m1.1.1.1.1.1.1.1.2" xref="alg1.l25.m1.1.1.1.1.1.1.1.2.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg1.l25.m1.1.1.1.1.1.1.1.1" xref="alg1.l25.m1.1.1.1.1.1.1.1.1.cmml">​</mo><mi id="alg1.l25.m1.1.1.1.1.1.1.1.3" xref="alg1.l25.m1.1.1.1.1.1.1.1.3.cmml">u</mi><mo lspace="0em" rspace="0em" id="alg1.l25.m1.1.1.1.1.1.1.1.1a" xref="alg1.l25.m1.1.1.1.1.1.1.1.1.cmml">​</mo><mi id="alg1.l25.m1.1.1.1.1.1.1.1.4" xref="alg1.l25.m1.1.1.1.1.1.1.1.4.cmml">c</mi><mo lspace="0em" rspace="0em" id="alg1.l25.m1.1.1.1.1.1.1.1.1b" xref="alg1.l25.m1.1.1.1.1.1.1.1.1.cmml">​</mo><mi id="alg1.l25.m1.1.1.1.1.1.1.1.5" xref="alg1.l25.m1.1.1.1.1.1.1.1.5.cmml">c</mi><mo lspace="0em" rspace="0em" id="alg1.l25.m1.1.1.1.1.1.1.1.1c" xref="alg1.l25.m1.1.1.1.1.1.1.1.1.cmml">​</mo><mi id="alg1.l25.m1.1.1.1.1.1.1.1.6" xref="alg1.l25.m1.1.1.1.1.1.1.1.6.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l25.m1.1.1.1.1.1.1.1.1d" xref="alg1.l25.m1.1.1.1.1.1.1.1.1.cmml">​</mo><mi id="alg1.l25.m1.1.1.1.1.1.1.1.7" xref="alg1.l25.m1.1.1.1.1.1.1.1.7.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg1.l25.m1.1.1.1.1.1.1.1.1e" xref="alg1.l25.m1.1.1.1.1.1.1.1.1.cmml">​</mo><mi id="alg1.l25.m1.1.1.1.1.1.1.1.8" xref="alg1.l25.m1.1.1.1.1.1.1.1.8.cmml">s</mi></mrow><mo id="alg1.l25.m1.1.1.1.1.2.2.3" xref="alg1.l25.m1.1.1.1.1.2.3.cmml">,</mo><mrow id="alg1.l25.m1.1.1.1.1.2.2.2" xref="alg1.l25.m1.1.1.1.1.2.2.2.cmml"><mi id="alg1.l25.m1.1.1.1.1.2.2.2.2" xref="alg1.l25.m1.1.1.1.1.2.2.2.2.cmml">b</mi><mo lspace="0em" rspace="0em" id="alg1.l25.m1.1.1.1.1.2.2.2.1" xref="alg1.l25.m1.1.1.1.1.2.2.2.1.cmml">​</mo><mi id="alg1.l25.m1.1.1.1.1.2.2.2.3" xref="alg1.l25.m1.1.1.1.1.2.2.2.3.cmml">u</mi><mo lspace="0em" rspace="0em" id="alg1.l25.m1.1.1.1.1.2.2.2.1a" xref="alg1.l25.m1.1.1.1.1.2.2.2.1.cmml">​</mo><mi id="alg1.l25.m1.1.1.1.1.2.2.2.4" xref="alg1.l25.m1.1.1.1.1.2.2.2.4.cmml">f</mi><mo lspace="0em" rspace="0em" id="alg1.l25.m1.1.1.1.1.2.2.2.1b" xref="alg1.l25.m1.1.1.1.1.2.2.2.1.cmml">​</mo><mi id="alg1.l25.m1.1.1.1.1.2.2.2.5" xref="alg1.l25.m1.1.1.1.1.2.2.2.5.cmml">f</mi><mo lspace="0em" rspace="0em" id="alg1.l25.m1.1.1.1.1.2.2.2.1c" xref="alg1.l25.m1.1.1.1.1.2.2.2.1.cmml">​</mo><mi id="alg1.l25.m1.1.1.1.1.2.2.2.6" xref="alg1.l25.m1.1.1.1.1.2.2.2.6.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l25.m1.1.1.1.1.2.2.2.1d" xref="alg1.l25.m1.1.1.1.1.2.2.2.1.cmml">​</mo><mi id="alg1.l25.m1.1.1.1.1.2.2.2.7" xref="alg1.l25.m1.1.1.1.1.2.2.2.7.cmml">r</mi></mrow></mrow><mo stretchy="false" id="alg1.l25.m1.1.1.1.1.3" xref="alg1.l25.m1.1.1.1.1.3.cmml">←</mo><mi id="alg1.l25.m1.1.1.1.1.4" xref="alg1.l25.m1.1.1.1.1.4.cmml">P</mi></mrow><mo lspace="0em" rspace="0.167em" id="alg1.l25.m1.2.2.2.3" xref="alg1.l25.m1.2.2.3a.cmml">.</mo><mrow id="alg1.l25.m1.2.2.2.2" xref="alg1.l25.m1.2.2.2.2.cmml"><mtext id="alg1.l25.m1.2.2.2.2.3" xref="alg1.l25.m1.2.2.2.2.3a.cmml">alloc</mtext><mo lspace="0em" rspace="0em" id="alg1.l25.m1.2.2.2.2.2" xref="alg1.l25.m1.2.2.2.2.2.cmml">​</mo><mrow id="alg1.l25.m1.2.2.2.2.1.1" xref="alg1.l25.m1.2.2.2.2.1.1.1.cmml"><mo stretchy="false" id="alg1.l25.m1.2.2.2.2.1.1.2" xref="alg1.l25.m1.2.2.2.2.1.1.1.cmml">(</mo><mrow id="alg1.l25.m1.2.2.2.2.1.1.1" xref="alg1.l25.m1.2.2.2.2.1.1.1.cmml"><mi id="alg1.l25.m1.2.2.2.2.1.1.1.2" xref="alg1.l25.m1.2.2.2.2.1.1.1.2.cmml">n</mi><mo lspace="0em" rspace="0em" id="alg1.l25.m1.2.2.2.2.1.1.1.1" xref="alg1.l25.m1.2.2.2.2.1.1.1.1.cmml">​</mo><mi id="alg1.l25.m1.2.2.2.2.1.1.1.3" xref="alg1.l25.m1.2.2.2.2.1.1.1.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l25.m1.2.2.2.2.1.1.1.1a" xref="alg1.l25.m1.2.2.2.2.1.1.1.1.cmml">​</mo><mi id="alg1.l25.m1.2.2.2.2.1.1.1.4" xref="alg1.l25.m1.2.2.2.2.1.1.1.4.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l25.m1.2.2.2.2.1.1.1.1b" xref="alg1.l25.m1.2.2.2.2.1.1.1.1.cmml">​</mo><mi id="alg1.l25.m1.2.2.2.2.1.1.1.5" xref="alg1.l25.m1.2.2.2.2.1.1.1.5.cmml">d</mi><mo lspace="0em" rspace="0em" id="alg1.l25.m1.2.2.2.2.1.1.1.1c" xref="alg1.l25.m1.2.2.2.2.1.1.1.1.cmml">​</mo><mi id="alg1.l25.m1.2.2.2.2.1.1.1.6" xref="alg1.l25.m1.2.2.2.2.1.1.1.6.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l25.m1.2.2.2.2.1.1.1.1d" xref="alg1.l25.m1.2.2.2.2.1.1.1.1.cmml">​</mo><mi id="alg1.l25.m1.2.2.2.2.1.1.1.7" xref="alg1.l25.m1.2.2.2.2.1.1.1.7.cmml">d</mi><mo lspace="0em" rspace="0em" id="alg1.l25.m1.2.2.2.2.1.1.1.1e" xref="alg1.l25.m1.2.2.2.2.1.1.1.1.cmml">​</mo><mi mathvariant="normal" id="alg1.l25.m1.2.2.2.2.1.1.1.8" xref="alg1.l25.m1.2.2.2.2.1.1.1.8.cmml">_</mi><mo lspace="0em" rspace="0em" id="alg1.l25.m1.2.2.2.2.1.1.1.1f" xref="alg1.l25.m1.2.2.2.2.1.1.1.1.cmml">​</mo><mi id="alg1.l25.m1.2.2.2.2.1.1.1.9" xref="alg1.l25.m1.2.2.2.2.1.1.1.9.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg1.l25.m1.2.2.2.2.1.1.1.1g" xref="alg1.l25.m1.2.2.2.2.1.1.1.1.cmml">​</mo><mi id="alg1.l25.m1.2.2.2.2.1.1.1.10" xref="alg1.l25.m1.2.2.2.2.1.1.1.10.cmml">i</mi><mo lspace="0em" rspace="0em" id="alg1.l25.m1.2.2.2.2.1.1.1.1h" xref="alg1.l25.m1.2.2.2.2.1.1.1.1.cmml">​</mo><mi id="alg1.l25.m1.2.2.2.2.1.1.1.11" xref="alg1.l25.m1.2.2.2.2.1.1.1.11.cmml">z</mi><mo lspace="0em" rspace="0em" id="alg1.l25.m1.2.2.2.2.1.1.1.1i" xref="alg1.l25.m1.2.2.2.2.1.1.1.1.cmml">​</mo><mi id="alg1.l25.m1.2.2.2.2.1.1.1.12" xref="alg1.l25.m1.2.2.2.2.1.1.1.12.cmml">e</mi></mrow><mo stretchy="false" id="alg1.l25.m1.2.2.2.2.1.1.3" xref="alg1.l25.m1.2.2.2.2.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l25.m1.2b"><apply id="alg1.l25.m1.2.2.3.cmml" xref="alg1.l25.m1.2.2.2"><csymbol cd="ambiguous" id="alg1.l25.m1.2.2.3a.cmml" xref="alg1.l25.m1.2.2.2.3">formulae-sequence</csymbol><apply id="alg1.l25.m1.1.1.1.1.cmml" xref="alg1.l25.m1.1.1.1.1"><ci id="alg1.l25.m1.1.1.1.1.3.cmml" xref="alg1.l25.m1.1.1.1.1.3">←</ci><list id="alg1.l25.m1.1.1.1.1.2.3.cmml" xref="alg1.l25.m1.1.1.1.1.2.2"><apply id="alg1.l25.m1.1.1.1.1.1.1.1.cmml" xref="alg1.l25.m1.1.1.1.1.1.1.1"><times id="alg1.l25.m1.1.1.1.1.1.1.1.1.cmml" xref="alg1.l25.m1.1.1.1.1.1.1.1.1"></times><ci id="alg1.l25.m1.1.1.1.1.1.1.1.2.cmml" xref="alg1.l25.m1.1.1.1.1.1.1.1.2">𝑠</ci><ci id="alg1.l25.m1.1.1.1.1.1.1.1.3.cmml" xref="alg1.l25.m1.1.1.1.1.1.1.1.3">𝑢</ci><ci id="alg1.l25.m1.1.1.1.1.1.1.1.4.cmml" xref="alg1.l25.m1.1.1.1.1.1.1.1.4">𝑐</ci><ci id="alg1.l25.m1.1.1.1.1.1.1.1.5.cmml" xref="alg1.l25.m1.1.1.1.1.1.1.1.5">𝑐</ci><ci id="alg1.l25.m1.1.1.1.1.1.1.1.6.cmml" xref="alg1.l25.m1.1.1.1.1.1.1.1.6">𝑒</ci><ci id="alg1.l25.m1.1.1.1.1.1.1.1.7.cmml" xref="alg1.l25.m1.1.1.1.1.1.1.1.7">𝑠</ci><ci id="alg1.l25.m1.1.1.1.1.1.1.1.8.cmml" xref="alg1.l25.m1.1.1.1.1.1.1.1.8">𝑠</ci></apply><apply id="alg1.l25.m1.1.1.1.1.2.2.2.cmml" xref="alg1.l25.m1.1.1.1.1.2.2.2"><times id="alg1.l25.m1.1.1.1.1.2.2.2.1.cmml" xref="alg1.l25.m1.1.1.1.1.2.2.2.1"></times><ci id="alg1.l25.m1.1.1.1.1.2.2.2.2.cmml" xref="alg1.l25.m1.1.1.1.1.2.2.2.2">𝑏</ci><ci id="alg1.l25.m1.1.1.1.1.2.2.2.3.cmml" xref="alg1.l25.m1.1.1.1.1.2.2.2.3">𝑢</ci><ci id="alg1.l25.m1.1.1.1.1.2.2.2.4.cmml" xref="alg1.l25.m1.1.1.1.1.2.2.2.4">𝑓</ci><ci id="alg1.l25.m1.1.1.1.1.2.2.2.5.cmml" xref="alg1.l25.m1.1.1.1.1.2.2.2.5">𝑓</ci><ci id="alg1.l25.m1.1.1.1.1.2.2.2.6.cmml" xref="alg1.l25.m1.1.1.1.1.2.2.2.6">𝑒</ci><ci id="alg1.l25.m1.1.1.1.1.2.2.2.7.cmml" xref="alg1.l25.m1.1.1.1.1.2.2.2.7">𝑟</ci></apply></list><ci id="alg1.l25.m1.1.1.1.1.4.cmml" xref="alg1.l25.m1.1.1.1.1.4">𝑃</ci></apply><apply id="alg1.l25.m1.2.2.2.2.cmml" xref="alg1.l25.m1.2.2.2.2"><times id="alg1.l25.m1.2.2.2.2.2.cmml" xref="alg1.l25.m1.2.2.2.2.2"></times><ci id="alg1.l25.m1.2.2.2.2.3a.cmml" xref="alg1.l25.m1.2.2.2.2.3"><mtext id="alg1.l25.m1.2.2.2.2.3.cmml" xref="alg1.l25.m1.2.2.2.2.3">alloc</mtext></ci><apply id="alg1.l25.m1.2.2.2.2.1.1.1.cmml" xref="alg1.l25.m1.2.2.2.2.1.1"><times id="alg1.l25.m1.2.2.2.2.1.1.1.1.cmml" xref="alg1.l25.m1.2.2.2.2.1.1.1.1"></times><ci id="alg1.l25.m1.2.2.2.2.1.1.1.2.cmml" xref="alg1.l25.m1.2.2.2.2.1.1.1.2">𝑛</ci><ci id="alg1.l25.m1.2.2.2.2.1.1.1.3.cmml" xref="alg1.l25.m1.2.2.2.2.1.1.1.3">𝑒</ci><ci id="alg1.l25.m1.2.2.2.2.1.1.1.4.cmml" xref="alg1.l25.m1.2.2.2.2.1.1.1.4">𝑒</ci><ci id="alg1.l25.m1.2.2.2.2.1.1.1.5.cmml" xref="alg1.l25.m1.2.2.2.2.1.1.1.5">𝑑</ci><ci id="alg1.l25.m1.2.2.2.2.1.1.1.6.cmml" xref="alg1.l25.m1.2.2.2.2.1.1.1.6">𝑒</ci><ci id="alg1.l25.m1.2.2.2.2.1.1.1.7.cmml" xref="alg1.l25.m1.2.2.2.2.1.1.1.7">𝑑</ci><ci id="alg1.l25.m1.2.2.2.2.1.1.1.8.cmml" xref="alg1.l25.m1.2.2.2.2.1.1.1.8">_</ci><ci id="alg1.l25.m1.2.2.2.2.1.1.1.9.cmml" xref="alg1.l25.m1.2.2.2.2.1.1.1.9">𝑠</ci><ci id="alg1.l25.m1.2.2.2.2.1.1.1.10.cmml" xref="alg1.l25.m1.2.2.2.2.1.1.1.10">𝑖</ci><ci id="alg1.l25.m1.2.2.2.2.1.1.1.11.cmml" xref="alg1.l25.m1.2.2.2.2.1.1.1.11">𝑧</ci><ci id="alg1.l25.m1.2.2.2.2.1.1.1.12.cmml" xref="alg1.l25.m1.2.2.2.2.1.1.1.12">𝑒</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l25.m1.2c">success,buffer\leftarrow P.\text{alloc}(needed\_size)</annotation></semantics></math>

</div>
<div id="alg1.l26" class="ltx_listingline">
<span id="alg1.l26.1" class="ltx_text ltx_font_bold">if</span> not <math id="alg1.l26.m1.1" class="ltx_Math" alttext="success" display="inline"><semantics id="alg1.l26.m1.1a"><mrow id="alg1.l26.m1.1.1" xref="alg1.l26.m1.1.1.cmml"><mi id="alg1.l26.m1.1.1.2" xref="alg1.l26.m1.1.1.2.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg1.l26.m1.1.1.1" xref="alg1.l26.m1.1.1.1.cmml">​</mo><mi id="alg1.l26.m1.1.1.3" xref="alg1.l26.m1.1.1.3.cmml">u</mi><mo lspace="0em" rspace="0em" id="alg1.l26.m1.1.1.1a" xref="alg1.l26.m1.1.1.1.cmml">​</mo><mi id="alg1.l26.m1.1.1.4" xref="alg1.l26.m1.1.1.4.cmml">c</mi><mo lspace="0em" rspace="0em" id="alg1.l26.m1.1.1.1b" xref="alg1.l26.m1.1.1.1.cmml">​</mo><mi id="alg1.l26.m1.1.1.5" xref="alg1.l26.m1.1.1.5.cmml">c</mi><mo lspace="0em" rspace="0em" id="alg1.l26.m1.1.1.1c" xref="alg1.l26.m1.1.1.1.cmml">​</mo><mi id="alg1.l26.m1.1.1.6" xref="alg1.l26.m1.1.1.6.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l26.m1.1.1.1d" xref="alg1.l26.m1.1.1.1.cmml">​</mo><mi id="alg1.l26.m1.1.1.7" xref="alg1.l26.m1.1.1.7.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg1.l26.m1.1.1.1e" xref="alg1.l26.m1.1.1.1.cmml">​</mo><mi id="alg1.l26.m1.1.1.8" xref="alg1.l26.m1.1.1.8.cmml">s</mi></mrow><annotation-xml encoding="MathML-Content" id="alg1.l26.m1.1b"><apply id="alg1.l26.m1.1.1.cmml" xref="alg1.l26.m1.1.1"><times id="alg1.l26.m1.1.1.1.cmml" xref="alg1.l26.m1.1.1.1"></times><ci id="alg1.l26.m1.1.1.2.cmml" xref="alg1.l26.m1.1.1.2">𝑠</ci><ci id="alg1.l26.m1.1.1.3.cmml" xref="alg1.l26.m1.1.1.3">𝑢</ci><ci id="alg1.l26.m1.1.1.4.cmml" xref="alg1.l26.m1.1.1.4">𝑐</ci><ci id="alg1.l26.m1.1.1.5.cmml" xref="alg1.l26.m1.1.1.5">𝑐</ci><ci id="alg1.l26.m1.1.1.6.cmml" xref="alg1.l26.m1.1.1.6">𝑒</ci><ci id="alg1.l26.m1.1.1.7.cmml" xref="alg1.l26.m1.1.1.7">𝑠</ci><ci id="alg1.l26.m1.1.1.8.cmml" xref="alg1.l26.m1.1.1.8">𝑠</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l26.m1.1c">success</annotation></semantics></math> <span id="alg1.l26.2" class="ltx_text ltx_font_bold">then</span>

</div>
<div id="alg1.l27" class="ltx_listingline">     <math id="alg1.l27.m1.2" class="ltx_Math" alttext="T.\text{evict}(needed\_size)" display="inline"><semantics id="alg1.l27.m1.2a"><mrow id="alg1.l27.m1.2.2.1" xref="alg1.l27.m1.2.2.2.cmml"><mi id="alg1.l27.m1.1.1" xref="alg1.l27.m1.1.1.cmml">T</mi><mo lspace="0em" rspace="0.167em" id="alg1.l27.m1.2.2.1.2" xref="alg1.l27.m1.2.2.2a.cmml">.</mo><mrow id="alg1.l27.m1.2.2.1.1" xref="alg1.l27.m1.2.2.1.1.cmml"><mtext id="alg1.l27.m1.2.2.1.1.3" xref="alg1.l27.m1.2.2.1.1.3a.cmml">evict</mtext><mo lspace="0em" rspace="0em" id="alg1.l27.m1.2.2.1.1.2" xref="alg1.l27.m1.2.2.1.1.2.cmml">​</mo><mrow id="alg1.l27.m1.2.2.1.1.1.1" xref="alg1.l27.m1.2.2.1.1.1.1.1.cmml"><mo stretchy="false" id="alg1.l27.m1.2.2.1.1.1.1.2" xref="alg1.l27.m1.2.2.1.1.1.1.1.cmml">(</mo><mrow id="alg1.l27.m1.2.2.1.1.1.1.1" xref="alg1.l27.m1.2.2.1.1.1.1.1.cmml"><mi id="alg1.l27.m1.2.2.1.1.1.1.1.2" xref="alg1.l27.m1.2.2.1.1.1.1.1.2.cmml">n</mi><mo lspace="0em" rspace="0em" id="alg1.l27.m1.2.2.1.1.1.1.1.1" xref="alg1.l27.m1.2.2.1.1.1.1.1.1.cmml">​</mo><mi id="alg1.l27.m1.2.2.1.1.1.1.1.3" xref="alg1.l27.m1.2.2.1.1.1.1.1.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l27.m1.2.2.1.1.1.1.1.1a" xref="alg1.l27.m1.2.2.1.1.1.1.1.1.cmml">​</mo><mi id="alg1.l27.m1.2.2.1.1.1.1.1.4" xref="alg1.l27.m1.2.2.1.1.1.1.1.4.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l27.m1.2.2.1.1.1.1.1.1b" xref="alg1.l27.m1.2.2.1.1.1.1.1.1.cmml">​</mo><mi id="alg1.l27.m1.2.2.1.1.1.1.1.5" xref="alg1.l27.m1.2.2.1.1.1.1.1.5.cmml">d</mi><mo lspace="0em" rspace="0em" id="alg1.l27.m1.2.2.1.1.1.1.1.1c" xref="alg1.l27.m1.2.2.1.1.1.1.1.1.cmml">​</mo><mi id="alg1.l27.m1.2.2.1.1.1.1.1.6" xref="alg1.l27.m1.2.2.1.1.1.1.1.6.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l27.m1.2.2.1.1.1.1.1.1d" xref="alg1.l27.m1.2.2.1.1.1.1.1.1.cmml">​</mo><mi id="alg1.l27.m1.2.2.1.1.1.1.1.7" xref="alg1.l27.m1.2.2.1.1.1.1.1.7.cmml">d</mi><mo lspace="0em" rspace="0em" id="alg1.l27.m1.2.2.1.1.1.1.1.1e" xref="alg1.l27.m1.2.2.1.1.1.1.1.1.cmml">​</mo><mi mathvariant="normal" id="alg1.l27.m1.2.2.1.1.1.1.1.8" xref="alg1.l27.m1.2.2.1.1.1.1.1.8.cmml">_</mi><mo lspace="0em" rspace="0em" id="alg1.l27.m1.2.2.1.1.1.1.1.1f" xref="alg1.l27.m1.2.2.1.1.1.1.1.1.cmml">​</mo><mi id="alg1.l27.m1.2.2.1.1.1.1.1.9" xref="alg1.l27.m1.2.2.1.1.1.1.1.9.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg1.l27.m1.2.2.1.1.1.1.1.1g" xref="alg1.l27.m1.2.2.1.1.1.1.1.1.cmml">​</mo><mi id="alg1.l27.m1.2.2.1.1.1.1.1.10" xref="alg1.l27.m1.2.2.1.1.1.1.1.10.cmml">i</mi><mo lspace="0em" rspace="0em" id="alg1.l27.m1.2.2.1.1.1.1.1.1h" xref="alg1.l27.m1.2.2.1.1.1.1.1.1.cmml">​</mo><mi id="alg1.l27.m1.2.2.1.1.1.1.1.11" xref="alg1.l27.m1.2.2.1.1.1.1.1.11.cmml">z</mi><mo lspace="0em" rspace="0em" id="alg1.l27.m1.2.2.1.1.1.1.1.1i" xref="alg1.l27.m1.2.2.1.1.1.1.1.1.cmml">​</mo><mi id="alg1.l27.m1.2.2.1.1.1.1.1.12" xref="alg1.l27.m1.2.2.1.1.1.1.1.12.cmml">e</mi></mrow><mo stretchy="false" id="alg1.l27.m1.2.2.1.1.1.1.3" xref="alg1.l27.m1.2.2.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l27.m1.2b"><apply id="alg1.l27.m1.2.2.2.cmml" xref="alg1.l27.m1.2.2.1"><csymbol cd="ambiguous" id="alg1.l27.m1.2.2.2a.cmml" xref="alg1.l27.m1.2.2.1.2">formulae-sequence</csymbol><ci id="alg1.l27.m1.1.1.cmml" xref="alg1.l27.m1.1.1">𝑇</ci><apply id="alg1.l27.m1.2.2.1.1.cmml" xref="alg1.l27.m1.2.2.1.1"><times id="alg1.l27.m1.2.2.1.1.2.cmml" xref="alg1.l27.m1.2.2.1.1.2"></times><ci id="alg1.l27.m1.2.2.1.1.3a.cmml" xref="alg1.l27.m1.2.2.1.1.3"><mtext id="alg1.l27.m1.2.2.1.1.3.cmml" xref="alg1.l27.m1.2.2.1.1.3">evict</mtext></ci><apply id="alg1.l27.m1.2.2.1.1.1.1.1.cmml" xref="alg1.l27.m1.2.2.1.1.1.1"><times id="alg1.l27.m1.2.2.1.1.1.1.1.1.cmml" xref="alg1.l27.m1.2.2.1.1.1.1.1.1"></times><ci id="alg1.l27.m1.2.2.1.1.1.1.1.2.cmml" xref="alg1.l27.m1.2.2.1.1.1.1.1.2">𝑛</ci><ci id="alg1.l27.m1.2.2.1.1.1.1.1.3.cmml" xref="alg1.l27.m1.2.2.1.1.1.1.1.3">𝑒</ci><ci id="alg1.l27.m1.2.2.1.1.1.1.1.4.cmml" xref="alg1.l27.m1.2.2.1.1.1.1.1.4">𝑒</ci><ci id="alg1.l27.m1.2.2.1.1.1.1.1.5.cmml" xref="alg1.l27.m1.2.2.1.1.1.1.1.5">𝑑</ci><ci id="alg1.l27.m1.2.2.1.1.1.1.1.6.cmml" xref="alg1.l27.m1.2.2.1.1.1.1.1.6">𝑒</ci><ci id="alg1.l27.m1.2.2.1.1.1.1.1.7.cmml" xref="alg1.l27.m1.2.2.1.1.1.1.1.7">𝑑</ci><ci id="alg1.l27.m1.2.2.1.1.1.1.1.8.cmml" xref="alg1.l27.m1.2.2.1.1.1.1.1.8">_</ci><ci id="alg1.l27.m1.2.2.1.1.1.1.1.9.cmml" xref="alg1.l27.m1.2.2.1.1.1.1.1.9">𝑠</ci><ci id="alg1.l27.m1.2.2.1.1.1.1.1.10.cmml" xref="alg1.l27.m1.2.2.1.1.1.1.1.10">𝑖</ci><ci id="alg1.l27.m1.2.2.1.1.1.1.1.11.cmml" xref="alg1.l27.m1.2.2.1.1.1.1.1.11">𝑧</ci><ci id="alg1.l27.m1.2.2.1.1.1.1.1.12.cmml" xref="alg1.l27.m1.2.2.1.1.1.1.1.12">𝑒</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l27.m1.2c">T.\text{evict}(needed\_size)</annotation></semantics></math>

</div>
<div id="alg1.l28" class="ltx_listingline">     <math id="alg1.l28.m1.2" class="ltx_Math" alttext="success,buffer\leftarrow P.\text{alloc}(needed\_size)" display="inline"><semantics id="alg1.l28.m1.2a"><mrow id="alg1.l28.m1.2.2.2" xref="alg1.l28.m1.2.2.3.cmml"><mrow id="alg1.l28.m1.1.1.1.1" xref="alg1.l28.m1.1.1.1.1.cmml"><mrow id="alg1.l28.m1.1.1.1.1.2.2" xref="alg1.l28.m1.1.1.1.1.2.3.cmml"><mrow id="alg1.l28.m1.1.1.1.1.1.1.1" xref="alg1.l28.m1.1.1.1.1.1.1.1.cmml"><mi id="alg1.l28.m1.1.1.1.1.1.1.1.2" xref="alg1.l28.m1.1.1.1.1.1.1.1.2.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg1.l28.m1.1.1.1.1.1.1.1.1" xref="alg1.l28.m1.1.1.1.1.1.1.1.1.cmml">​</mo><mi id="alg1.l28.m1.1.1.1.1.1.1.1.3" xref="alg1.l28.m1.1.1.1.1.1.1.1.3.cmml">u</mi><mo lspace="0em" rspace="0em" id="alg1.l28.m1.1.1.1.1.1.1.1.1a" xref="alg1.l28.m1.1.1.1.1.1.1.1.1.cmml">​</mo><mi id="alg1.l28.m1.1.1.1.1.1.1.1.4" xref="alg1.l28.m1.1.1.1.1.1.1.1.4.cmml">c</mi><mo lspace="0em" rspace="0em" id="alg1.l28.m1.1.1.1.1.1.1.1.1b" xref="alg1.l28.m1.1.1.1.1.1.1.1.1.cmml">​</mo><mi id="alg1.l28.m1.1.1.1.1.1.1.1.5" xref="alg1.l28.m1.1.1.1.1.1.1.1.5.cmml">c</mi><mo lspace="0em" rspace="0em" id="alg1.l28.m1.1.1.1.1.1.1.1.1c" xref="alg1.l28.m1.1.1.1.1.1.1.1.1.cmml">​</mo><mi id="alg1.l28.m1.1.1.1.1.1.1.1.6" xref="alg1.l28.m1.1.1.1.1.1.1.1.6.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l28.m1.1.1.1.1.1.1.1.1d" xref="alg1.l28.m1.1.1.1.1.1.1.1.1.cmml">​</mo><mi id="alg1.l28.m1.1.1.1.1.1.1.1.7" xref="alg1.l28.m1.1.1.1.1.1.1.1.7.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg1.l28.m1.1.1.1.1.1.1.1.1e" xref="alg1.l28.m1.1.1.1.1.1.1.1.1.cmml">​</mo><mi id="alg1.l28.m1.1.1.1.1.1.1.1.8" xref="alg1.l28.m1.1.1.1.1.1.1.1.8.cmml">s</mi></mrow><mo id="alg1.l28.m1.1.1.1.1.2.2.3" xref="alg1.l28.m1.1.1.1.1.2.3.cmml">,</mo><mrow id="alg1.l28.m1.1.1.1.1.2.2.2" xref="alg1.l28.m1.1.1.1.1.2.2.2.cmml"><mi id="alg1.l28.m1.1.1.1.1.2.2.2.2" xref="alg1.l28.m1.1.1.1.1.2.2.2.2.cmml">b</mi><mo lspace="0em" rspace="0em" id="alg1.l28.m1.1.1.1.1.2.2.2.1" xref="alg1.l28.m1.1.1.1.1.2.2.2.1.cmml">​</mo><mi id="alg1.l28.m1.1.1.1.1.2.2.2.3" xref="alg1.l28.m1.1.1.1.1.2.2.2.3.cmml">u</mi><mo lspace="0em" rspace="0em" id="alg1.l28.m1.1.1.1.1.2.2.2.1a" xref="alg1.l28.m1.1.1.1.1.2.2.2.1.cmml">​</mo><mi id="alg1.l28.m1.1.1.1.1.2.2.2.4" xref="alg1.l28.m1.1.1.1.1.2.2.2.4.cmml">f</mi><mo lspace="0em" rspace="0em" id="alg1.l28.m1.1.1.1.1.2.2.2.1b" xref="alg1.l28.m1.1.1.1.1.2.2.2.1.cmml">​</mo><mi id="alg1.l28.m1.1.1.1.1.2.2.2.5" xref="alg1.l28.m1.1.1.1.1.2.2.2.5.cmml">f</mi><mo lspace="0em" rspace="0em" id="alg1.l28.m1.1.1.1.1.2.2.2.1c" xref="alg1.l28.m1.1.1.1.1.2.2.2.1.cmml">​</mo><mi id="alg1.l28.m1.1.1.1.1.2.2.2.6" xref="alg1.l28.m1.1.1.1.1.2.2.2.6.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l28.m1.1.1.1.1.2.2.2.1d" xref="alg1.l28.m1.1.1.1.1.2.2.2.1.cmml">​</mo><mi id="alg1.l28.m1.1.1.1.1.2.2.2.7" xref="alg1.l28.m1.1.1.1.1.2.2.2.7.cmml">r</mi></mrow></mrow><mo stretchy="false" id="alg1.l28.m1.1.1.1.1.3" xref="alg1.l28.m1.1.1.1.1.3.cmml">←</mo><mi id="alg1.l28.m1.1.1.1.1.4" xref="alg1.l28.m1.1.1.1.1.4.cmml">P</mi></mrow><mo lspace="0em" rspace="0.167em" id="alg1.l28.m1.2.2.2.3" xref="alg1.l28.m1.2.2.3a.cmml">.</mo><mrow id="alg1.l28.m1.2.2.2.2" xref="alg1.l28.m1.2.2.2.2.cmml"><mtext id="alg1.l28.m1.2.2.2.2.3" xref="alg1.l28.m1.2.2.2.2.3a.cmml">alloc</mtext><mo lspace="0em" rspace="0em" id="alg1.l28.m1.2.2.2.2.2" xref="alg1.l28.m1.2.2.2.2.2.cmml">​</mo><mrow id="alg1.l28.m1.2.2.2.2.1.1" xref="alg1.l28.m1.2.2.2.2.1.1.1.cmml"><mo stretchy="false" id="alg1.l28.m1.2.2.2.2.1.1.2" xref="alg1.l28.m1.2.2.2.2.1.1.1.cmml">(</mo><mrow id="alg1.l28.m1.2.2.2.2.1.1.1" xref="alg1.l28.m1.2.2.2.2.1.1.1.cmml"><mi id="alg1.l28.m1.2.2.2.2.1.1.1.2" xref="alg1.l28.m1.2.2.2.2.1.1.1.2.cmml">n</mi><mo lspace="0em" rspace="0em" id="alg1.l28.m1.2.2.2.2.1.1.1.1" xref="alg1.l28.m1.2.2.2.2.1.1.1.1.cmml">​</mo><mi id="alg1.l28.m1.2.2.2.2.1.1.1.3" xref="alg1.l28.m1.2.2.2.2.1.1.1.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l28.m1.2.2.2.2.1.1.1.1a" xref="alg1.l28.m1.2.2.2.2.1.1.1.1.cmml">​</mo><mi id="alg1.l28.m1.2.2.2.2.1.1.1.4" xref="alg1.l28.m1.2.2.2.2.1.1.1.4.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l28.m1.2.2.2.2.1.1.1.1b" xref="alg1.l28.m1.2.2.2.2.1.1.1.1.cmml">​</mo><mi id="alg1.l28.m1.2.2.2.2.1.1.1.5" xref="alg1.l28.m1.2.2.2.2.1.1.1.5.cmml">d</mi><mo lspace="0em" rspace="0em" id="alg1.l28.m1.2.2.2.2.1.1.1.1c" xref="alg1.l28.m1.2.2.2.2.1.1.1.1.cmml">​</mo><mi id="alg1.l28.m1.2.2.2.2.1.1.1.6" xref="alg1.l28.m1.2.2.2.2.1.1.1.6.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l28.m1.2.2.2.2.1.1.1.1d" xref="alg1.l28.m1.2.2.2.2.1.1.1.1.cmml">​</mo><mi id="alg1.l28.m1.2.2.2.2.1.1.1.7" xref="alg1.l28.m1.2.2.2.2.1.1.1.7.cmml">d</mi><mo lspace="0em" rspace="0em" id="alg1.l28.m1.2.2.2.2.1.1.1.1e" xref="alg1.l28.m1.2.2.2.2.1.1.1.1.cmml">​</mo><mi mathvariant="normal" id="alg1.l28.m1.2.2.2.2.1.1.1.8" xref="alg1.l28.m1.2.2.2.2.1.1.1.8.cmml">_</mi><mo lspace="0em" rspace="0em" id="alg1.l28.m1.2.2.2.2.1.1.1.1f" xref="alg1.l28.m1.2.2.2.2.1.1.1.1.cmml">​</mo><mi id="alg1.l28.m1.2.2.2.2.1.1.1.9" xref="alg1.l28.m1.2.2.2.2.1.1.1.9.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg1.l28.m1.2.2.2.2.1.1.1.1g" xref="alg1.l28.m1.2.2.2.2.1.1.1.1.cmml">​</mo><mi id="alg1.l28.m1.2.2.2.2.1.1.1.10" xref="alg1.l28.m1.2.2.2.2.1.1.1.10.cmml">i</mi><mo lspace="0em" rspace="0em" id="alg1.l28.m1.2.2.2.2.1.1.1.1h" xref="alg1.l28.m1.2.2.2.2.1.1.1.1.cmml">​</mo><mi id="alg1.l28.m1.2.2.2.2.1.1.1.11" xref="alg1.l28.m1.2.2.2.2.1.1.1.11.cmml">z</mi><mo lspace="0em" rspace="0em" id="alg1.l28.m1.2.2.2.2.1.1.1.1i" xref="alg1.l28.m1.2.2.2.2.1.1.1.1.cmml">​</mo><mi id="alg1.l28.m1.2.2.2.2.1.1.1.12" xref="alg1.l28.m1.2.2.2.2.1.1.1.12.cmml">e</mi></mrow><mo stretchy="false" id="alg1.l28.m1.2.2.2.2.1.1.3" xref="alg1.l28.m1.2.2.2.2.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l28.m1.2b"><apply id="alg1.l28.m1.2.2.3.cmml" xref="alg1.l28.m1.2.2.2"><csymbol cd="ambiguous" id="alg1.l28.m1.2.2.3a.cmml" xref="alg1.l28.m1.2.2.2.3">formulae-sequence</csymbol><apply id="alg1.l28.m1.1.1.1.1.cmml" xref="alg1.l28.m1.1.1.1.1"><ci id="alg1.l28.m1.1.1.1.1.3.cmml" xref="alg1.l28.m1.1.1.1.1.3">←</ci><list id="alg1.l28.m1.1.1.1.1.2.3.cmml" xref="alg1.l28.m1.1.1.1.1.2.2"><apply id="alg1.l28.m1.1.1.1.1.1.1.1.cmml" xref="alg1.l28.m1.1.1.1.1.1.1.1"><times id="alg1.l28.m1.1.1.1.1.1.1.1.1.cmml" xref="alg1.l28.m1.1.1.1.1.1.1.1.1"></times><ci id="alg1.l28.m1.1.1.1.1.1.1.1.2.cmml" xref="alg1.l28.m1.1.1.1.1.1.1.1.2">𝑠</ci><ci id="alg1.l28.m1.1.1.1.1.1.1.1.3.cmml" xref="alg1.l28.m1.1.1.1.1.1.1.1.3">𝑢</ci><ci id="alg1.l28.m1.1.1.1.1.1.1.1.4.cmml" xref="alg1.l28.m1.1.1.1.1.1.1.1.4">𝑐</ci><ci id="alg1.l28.m1.1.1.1.1.1.1.1.5.cmml" xref="alg1.l28.m1.1.1.1.1.1.1.1.5">𝑐</ci><ci id="alg1.l28.m1.1.1.1.1.1.1.1.6.cmml" xref="alg1.l28.m1.1.1.1.1.1.1.1.6">𝑒</ci><ci id="alg1.l28.m1.1.1.1.1.1.1.1.7.cmml" xref="alg1.l28.m1.1.1.1.1.1.1.1.7">𝑠</ci><ci id="alg1.l28.m1.1.1.1.1.1.1.1.8.cmml" xref="alg1.l28.m1.1.1.1.1.1.1.1.8">𝑠</ci></apply><apply id="alg1.l28.m1.1.1.1.1.2.2.2.cmml" xref="alg1.l28.m1.1.1.1.1.2.2.2"><times id="alg1.l28.m1.1.1.1.1.2.2.2.1.cmml" xref="alg1.l28.m1.1.1.1.1.2.2.2.1"></times><ci id="alg1.l28.m1.1.1.1.1.2.2.2.2.cmml" xref="alg1.l28.m1.1.1.1.1.2.2.2.2">𝑏</ci><ci id="alg1.l28.m1.1.1.1.1.2.2.2.3.cmml" xref="alg1.l28.m1.1.1.1.1.2.2.2.3">𝑢</ci><ci id="alg1.l28.m1.1.1.1.1.2.2.2.4.cmml" xref="alg1.l28.m1.1.1.1.1.2.2.2.4">𝑓</ci><ci id="alg1.l28.m1.1.1.1.1.2.2.2.5.cmml" xref="alg1.l28.m1.1.1.1.1.2.2.2.5">𝑓</ci><ci id="alg1.l28.m1.1.1.1.1.2.2.2.6.cmml" xref="alg1.l28.m1.1.1.1.1.2.2.2.6">𝑒</ci><ci id="alg1.l28.m1.1.1.1.1.2.2.2.7.cmml" xref="alg1.l28.m1.1.1.1.1.2.2.2.7">𝑟</ci></apply></list><ci id="alg1.l28.m1.1.1.1.1.4.cmml" xref="alg1.l28.m1.1.1.1.1.4">𝑃</ci></apply><apply id="alg1.l28.m1.2.2.2.2.cmml" xref="alg1.l28.m1.2.2.2.2"><times id="alg1.l28.m1.2.2.2.2.2.cmml" xref="alg1.l28.m1.2.2.2.2.2"></times><ci id="alg1.l28.m1.2.2.2.2.3a.cmml" xref="alg1.l28.m1.2.2.2.2.3"><mtext id="alg1.l28.m1.2.2.2.2.3.cmml" xref="alg1.l28.m1.2.2.2.2.3">alloc</mtext></ci><apply id="alg1.l28.m1.2.2.2.2.1.1.1.cmml" xref="alg1.l28.m1.2.2.2.2.1.1"><times id="alg1.l28.m1.2.2.2.2.1.1.1.1.cmml" xref="alg1.l28.m1.2.2.2.2.1.1.1.1"></times><ci id="alg1.l28.m1.2.2.2.2.1.1.1.2.cmml" xref="alg1.l28.m1.2.2.2.2.1.1.1.2">𝑛</ci><ci id="alg1.l28.m1.2.2.2.2.1.1.1.3.cmml" xref="alg1.l28.m1.2.2.2.2.1.1.1.3">𝑒</ci><ci id="alg1.l28.m1.2.2.2.2.1.1.1.4.cmml" xref="alg1.l28.m1.2.2.2.2.1.1.1.4">𝑒</ci><ci id="alg1.l28.m1.2.2.2.2.1.1.1.5.cmml" xref="alg1.l28.m1.2.2.2.2.1.1.1.5">𝑑</ci><ci id="alg1.l28.m1.2.2.2.2.1.1.1.6.cmml" xref="alg1.l28.m1.2.2.2.2.1.1.1.6">𝑒</ci><ci id="alg1.l28.m1.2.2.2.2.1.1.1.7.cmml" xref="alg1.l28.m1.2.2.2.2.1.1.1.7">𝑑</ci><ci id="alg1.l28.m1.2.2.2.2.1.1.1.8.cmml" xref="alg1.l28.m1.2.2.2.2.1.1.1.8">_</ci><ci id="alg1.l28.m1.2.2.2.2.1.1.1.9.cmml" xref="alg1.l28.m1.2.2.2.2.1.1.1.9">𝑠</ci><ci id="alg1.l28.m1.2.2.2.2.1.1.1.10.cmml" xref="alg1.l28.m1.2.2.2.2.1.1.1.10">𝑖</ci><ci id="alg1.l28.m1.2.2.2.2.1.1.1.11.cmml" xref="alg1.l28.m1.2.2.2.2.1.1.1.11">𝑧</ci><ci id="alg1.l28.m1.2.2.2.2.1.1.1.12.cmml" xref="alg1.l28.m1.2.2.2.2.1.1.1.12">𝑒</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l28.m1.2c">success,buffer\leftarrow P.\text{alloc}(needed\_size)</annotation></semantics></math>

</div>
<div id="alg1.l29" class="ltx_listingline">
<math id="alg1.l29.m1.2" class="ltx_Math" alttext="B.\text{run}(buffer)" display="inline"><semantics id="alg1.l29.m1.2a"><mrow id="alg1.l29.m1.2.2.1" xref="alg1.l29.m1.2.2.2.cmml"><mi id="alg1.l29.m1.1.1" xref="alg1.l29.m1.1.1.cmml">B</mi><mo lspace="0em" rspace="0.167em" id="alg1.l29.m1.2.2.1.2" xref="alg1.l29.m1.2.2.2a.cmml">.</mo><mrow id="alg1.l29.m1.2.2.1.1" xref="alg1.l29.m1.2.2.1.1.cmml"><mtext id="alg1.l29.m1.2.2.1.1.3" xref="alg1.l29.m1.2.2.1.1.3a.cmml">run</mtext><mo lspace="0em" rspace="0em" id="alg1.l29.m1.2.2.1.1.2" xref="alg1.l29.m1.2.2.1.1.2.cmml">​</mo><mrow id="alg1.l29.m1.2.2.1.1.1.1" xref="alg1.l29.m1.2.2.1.1.1.1.1.cmml"><mo stretchy="false" id="alg1.l29.m1.2.2.1.1.1.1.2" xref="alg1.l29.m1.2.2.1.1.1.1.1.cmml">(</mo><mrow id="alg1.l29.m1.2.2.1.1.1.1.1" xref="alg1.l29.m1.2.2.1.1.1.1.1.cmml"><mi id="alg1.l29.m1.2.2.1.1.1.1.1.2" xref="alg1.l29.m1.2.2.1.1.1.1.1.2.cmml">b</mi><mo lspace="0em" rspace="0em" id="alg1.l29.m1.2.2.1.1.1.1.1.1" xref="alg1.l29.m1.2.2.1.1.1.1.1.1.cmml">​</mo><mi id="alg1.l29.m1.2.2.1.1.1.1.1.3" xref="alg1.l29.m1.2.2.1.1.1.1.1.3.cmml">u</mi><mo lspace="0em" rspace="0em" id="alg1.l29.m1.2.2.1.1.1.1.1.1a" xref="alg1.l29.m1.2.2.1.1.1.1.1.1.cmml">​</mo><mi id="alg1.l29.m1.2.2.1.1.1.1.1.4" xref="alg1.l29.m1.2.2.1.1.1.1.1.4.cmml">f</mi><mo lspace="0em" rspace="0em" id="alg1.l29.m1.2.2.1.1.1.1.1.1b" xref="alg1.l29.m1.2.2.1.1.1.1.1.1.cmml">​</mo><mi id="alg1.l29.m1.2.2.1.1.1.1.1.5" xref="alg1.l29.m1.2.2.1.1.1.1.1.5.cmml">f</mi><mo lspace="0em" rspace="0em" id="alg1.l29.m1.2.2.1.1.1.1.1.1c" xref="alg1.l29.m1.2.2.1.1.1.1.1.1.cmml">​</mo><mi id="alg1.l29.m1.2.2.1.1.1.1.1.6" xref="alg1.l29.m1.2.2.1.1.1.1.1.6.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l29.m1.2.2.1.1.1.1.1.1d" xref="alg1.l29.m1.2.2.1.1.1.1.1.1.cmml">​</mo><mi id="alg1.l29.m1.2.2.1.1.1.1.1.7" xref="alg1.l29.m1.2.2.1.1.1.1.1.7.cmml">r</mi></mrow><mo stretchy="false" id="alg1.l29.m1.2.2.1.1.1.1.3" xref="alg1.l29.m1.2.2.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l29.m1.2b"><apply id="alg1.l29.m1.2.2.2.cmml" xref="alg1.l29.m1.2.2.1"><csymbol cd="ambiguous" id="alg1.l29.m1.2.2.2a.cmml" xref="alg1.l29.m1.2.2.1.2">formulae-sequence</csymbol><ci id="alg1.l29.m1.1.1.cmml" xref="alg1.l29.m1.1.1">𝐵</ci><apply id="alg1.l29.m1.2.2.1.1.cmml" xref="alg1.l29.m1.2.2.1.1"><times id="alg1.l29.m1.2.2.1.1.2.cmml" xref="alg1.l29.m1.2.2.1.1.2"></times><ci id="alg1.l29.m1.2.2.1.1.3a.cmml" xref="alg1.l29.m1.2.2.1.1.3"><mtext id="alg1.l29.m1.2.2.1.1.3.cmml" xref="alg1.l29.m1.2.2.1.1.3">run</mtext></ci><apply id="alg1.l29.m1.2.2.1.1.1.1.1.cmml" xref="alg1.l29.m1.2.2.1.1.1.1"><times id="alg1.l29.m1.2.2.1.1.1.1.1.1.cmml" xref="alg1.l29.m1.2.2.1.1.1.1.1.1"></times><ci id="alg1.l29.m1.2.2.1.1.1.1.1.2.cmml" xref="alg1.l29.m1.2.2.1.1.1.1.1.2">𝑏</ci><ci id="alg1.l29.m1.2.2.1.1.1.1.1.3.cmml" xref="alg1.l29.m1.2.2.1.1.1.1.1.3">𝑢</ci><ci id="alg1.l29.m1.2.2.1.1.1.1.1.4.cmml" xref="alg1.l29.m1.2.2.1.1.1.1.1.4">𝑓</ci><ci id="alg1.l29.m1.2.2.1.1.1.1.1.5.cmml" xref="alg1.l29.m1.2.2.1.1.1.1.1.5">𝑓</ci><ci id="alg1.l29.m1.2.2.1.1.1.1.1.6.cmml" xref="alg1.l29.m1.2.2.1.1.1.1.1.6">𝑒</ci><ci id="alg1.l29.m1.2.2.1.1.1.1.1.7.cmml" xref="alg1.l29.m1.2.2.1.1.1.1.1.7">𝑟</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l29.m1.2c">B.\text{run}(buffer)</annotation></semantics></math>

</div>
<div id="alg1.l30" class="ltx_listingline">
<span id="alg1.l30.1" class="ltx_text">// Process finished requests</span>

</div>
<div id="alg1.l31" class="ltx_listingline">
<math id="alg1.l31.m1.2" class="ltx_Math" alttext="finished\_requests\leftarrow B.\text{drop\_finished\_requests}()" display="inline"><semantics id="alg1.l31.m1.2a"><mrow id="alg1.l31.m1.2.2.2" xref="alg1.l31.m1.2.2.3.cmml"><mrow id="alg1.l31.m1.1.1.1.1" xref="alg1.l31.m1.1.1.1.1.cmml"><mrow id="alg1.l31.m1.1.1.1.1.2" xref="alg1.l31.m1.1.1.1.1.2.cmml"><mi id="alg1.l31.m1.1.1.1.1.2.2" xref="alg1.l31.m1.1.1.1.1.2.2.cmml">f</mi><mo lspace="0em" rspace="0em" id="alg1.l31.m1.1.1.1.1.2.1" xref="alg1.l31.m1.1.1.1.1.2.1.cmml">​</mo><mi id="alg1.l31.m1.1.1.1.1.2.3" xref="alg1.l31.m1.1.1.1.1.2.3.cmml">i</mi><mo lspace="0em" rspace="0em" id="alg1.l31.m1.1.1.1.1.2.1a" xref="alg1.l31.m1.1.1.1.1.2.1.cmml">​</mo><mi id="alg1.l31.m1.1.1.1.1.2.4" xref="alg1.l31.m1.1.1.1.1.2.4.cmml">n</mi><mo lspace="0em" rspace="0em" id="alg1.l31.m1.1.1.1.1.2.1b" xref="alg1.l31.m1.1.1.1.1.2.1.cmml">​</mo><mi id="alg1.l31.m1.1.1.1.1.2.5" xref="alg1.l31.m1.1.1.1.1.2.5.cmml">i</mi><mo lspace="0em" rspace="0em" id="alg1.l31.m1.1.1.1.1.2.1c" xref="alg1.l31.m1.1.1.1.1.2.1.cmml">​</mo><mi id="alg1.l31.m1.1.1.1.1.2.6" xref="alg1.l31.m1.1.1.1.1.2.6.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg1.l31.m1.1.1.1.1.2.1d" xref="alg1.l31.m1.1.1.1.1.2.1.cmml">​</mo><mi id="alg1.l31.m1.1.1.1.1.2.7" xref="alg1.l31.m1.1.1.1.1.2.7.cmml">h</mi><mo lspace="0em" rspace="0em" id="alg1.l31.m1.1.1.1.1.2.1e" xref="alg1.l31.m1.1.1.1.1.2.1.cmml">​</mo><mi id="alg1.l31.m1.1.1.1.1.2.8" xref="alg1.l31.m1.1.1.1.1.2.8.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l31.m1.1.1.1.1.2.1f" xref="alg1.l31.m1.1.1.1.1.2.1.cmml">​</mo><mi id="alg1.l31.m1.1.1.1.1.2.9" xref="alg1.l31.m1.1.1.1.1.2.9.cmml">d</mi><mo lspace="0em" rspace="0em" id="alg1.l31.m1.1.1.1.1.2.1g" xref="alg1.l31.m1.1.1.1.1.2.1.cmml">​</mo><mi mathvariant="normal" id="alg1.l31.m1.1.1.1.1.2.10" xref="alg1.l31.m1.1.1.1.1.2.10.cmml">_</mi><mo lspace="0em" rspace="0em" id="alg1.l31.m1.1.1.1.1.2.1h" xref="alg1.l31.m1.1.1.1.1.2.1.cmml">​</mo><mi id="alg1.l31.m1.1.1.1.1.2.11" xref="alg1.l31.m1.1.1.1.1.2.11.cmml">r</mi><mo lspace="0em" rspace="0em" id="alg1.l31.m1.1.1.1.1.2.1i" xref="alg1.l31.m1.1.1.1.1.2.1.cmml">​</mo><mi id="alg1.l31.m1.1.1.1.1.2.12" xref="alg1.l31.m1.1.1.1.1.2.12.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l31.m1.1.1.1.1.2.1j" xref="alg1.l31.m1.1.1.1.1.2.1.cmml">​</mo><mi id="alg1.l31.m1.1.1.1.1.2.13" xref="alg1.l31.m1.1.1.1.1.2.13.cmml">q</mi><mo lspace="0em" rspace="0em" id="alg1.l31.m1.1.1.1.1.2.1k" xref="alg1.l31.m1.1.1.1.1.2.1.cmml">​</mo><mi id="alg1.l31.m1.1.1.1.1.2.14" xref="alg1.l31.m1.1.1.1.1.2.14.cmml">u</mi><mo lspace="0em" rspace="0em" id="alg1.l31.m1.1.1.1.1.2.1l" xref="alg1.l31.m1.1.1.1.1.2.1.cmml">​</mo><mi id="alg1.l31.m1.1.1.1.1.2.15" xref="alg1.l31.m1.1.1.1.1.2.15.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l31.m1.1.1.1.1.2.1m" xref="alg1.l31.m1.1.1.1.1.2.1.cmml">​</mo><mi id="alg1.l31.m1.1.1.1.1.2.16" xref="alg1.l31.m1.1.1.1.1.2.16.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg1.l31.m1.1.1.1.1.2.1n" xref="alg1.l31.m1.1.1.1.1.2.1.cmml">​</mo><mi id="alg1.l31.m1.1.1.1.1.2.17" xref="alg1.l31.m1.1.1.1.1.2.17.cmml">t</mi><mo lspace="0em" rspace="0em" id="alg1.l31.m1.1.1.1.1.2.1o" xref="alg1.l31.m1.1.1.1.1.2.1.cmml">​</mo><mi id="alg1.l31.m1.1.1.1.1.2.18" xref="alg1.l31.m1.1.1.1.1.2.18.cmml">s</mi></mrow><mo stretchy="false" id="alg1.l31.m1.1.1.1.1.1" xref="alg1.l31.m1.1.1.1.1.1.cmml">←</mo><mi id="alg1.l31.m1.1.1.1.1.3" xref="alg1.l31.m1.1.1.1.1.3.cmml">B</mi></mrow><mo lspace="0em" rspace="0.167em" id="alg1.l31.m1.2.2.2.3" xref="alg1.l31.m1.2.2.3a.cmml">.</mo><mrow id="alg1.l31.m1.2.2.2.2" xref="alg1.l31.m1.2.2.2.2.cmml"><mtext id="alg1.l31.m1.2.2.2.2.2" xref="alg1.l31.m1.2.2.2.2.2a.cmml">drop_finished_requests</mtext><mo lspace="0em" rspace="0em" id="alg1.l31.m1.2.2.2.2.1" xref="alg1.l31.m1.2.2.2.2.1.cmml">​</mo><mrow id="alg1.l31.m1.2.2.2.2.3.2" xref="alg1.l31.m1.2.2.2.2.cmml"><mo stretchy="false" id="alg1.l31.m1.2.2.2.2.3.2.1" xref="alg1.l31.m1.2.2.2.2.3.1.cmml">(</mo><mo stretchy="false" id="alg1.l31.m1.2.2.2.2.3.2.2" xref="alg1.l31.m1.2.2.2.2.3.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l31.m1.2b"><apply id="alg1.l31.m1.2.2.3.cmml" xref="alg1.l31.m1.2.2.2"><csymbol cd="ambiguous" id="alg1.l31.m1.2.2.3a.cmml" xref="alg1.l31.m1.2.2.2.3">formulae-sequence</csymbol><apply id="alg1.l31.m1.1.1.1.1.cmml" xref="alg1.l31.m1.1.1.1.1"><ci id="alg1.l31.m1.1.1.1.1.1.cmml" xref="alg1.l31.m1.1.1.1.1.1">←</ci><apply id="alg1.l31.m1.1.1.1.1.2.cmml" xref="alg1.l31.m1.1.1.1.1.2"><times id="alg1.l31.m1.1.1.1.1.2.1.cmml" xref="alg1.l31.m1.1.1.1.1.2.1"></times><ci id="alg1.l31.m1.1.1.1.1.2.2.cmml" xref="alg1.l31.m1.1.1.1.1.2.2">𝑓</ci><ci id="alg1.l31.m1.1.1.1.1.2.3.cmml" xref="alg1.l31.m1.1.1.1.1.2.3">𝑖</ci><ci id="alg1.l31.m1.1.1.1.1.2.4.cmml" xref="alg1.l31.m1.1.1.1.1.2.4">𝑛</ci><ci id="alg1.l31.m1.1.1.1.1.2.5.cmml" xref="alg1.l31.m1.1.1.1.1.2.5">𝑖</ci><ci id="alg1.l31.m1.1.1.1.1.2.6.cmml" xref="alg1.l31.m1.1.1.1.1.2.6">𝑠</ci><ci id="alg1.l31.m1.1.1.1.1.2.7.cmml" xref="alg1.l31.m1.1.1.1.1.2.7">ℎ</ci><ci id="alg1.l31.m1.1.1.1.1.2.8.cmml" xref="alg1.l31.m1.1.1.1.1.2.8">𝑒</ci><ci id="alg1.l31.m1.1.1.1.1.2.9.cmml" xref="alg1.l31.m1.1.1.1.1.2.9">𝑑</ci><ci id="alg1.l31.m1.1.1.1.1.2.10.cmml" xref="alg1.l31.m1.1.1.1.1.2.10">_</ci><ci id="alg1.l31.m1.1.1.1.1.2.11.cmml" xref="alg1.l31.m1.1.1.1.1.2.11">𝑟</ci><ci id="alg1.l31.m1.1.1.1.1.2.12.cmml" xref="alg1.l31.m1.1.1.1.1.2.12">𝑒</ci><ci id="alg1.l31.m1.1.1.1.1.2.13.cmml" xref="alg1.l31.m1.1.1.1.1.2.13">𝑞</ci><ci id="alg1.l31.m1.1.1.1.1.2.14.cmml" xref="alg1.l31.m1.1.1.1.1.2.14">𝑢</ci><ci id="alg1.l31.m1.1.1.1.1.2.15.cmml" xref="alg1.l31.m1.1.1.1.1.2.15">𝑒</ci><ci id="alg1.l31.m1.1.1.1.1.2.16.cmml" xref="alg1.l31.m1.1.1.1.1.2.16">𝑠</ci><ci id="alg1.l31.m1.1.1.1.1.2.17.cmml" xref="alg1.l31.m1.1.1.1.1.2.17">𝑡</ci><ci id="alg1.l31.m1.1.1.1.1.2.18.cmml" xref="alg1.l31.m1.1.1.1.1.2.18">𝑠</ci></apply><ci id="alg1.l31.m1.1.1.1.1.3.cmml" xref="alg1.l31.m1.1.1.1.1.3">𝐵</ci></apply><apply id="alg1.l31.m1.2.2.2.2.cmml" xref="alg1.l31.m1.2.2.2.2"><times id="alg1.l31.m1.2.2.2.2.1.cmml" xref="alg1.l31.m1.2.2.2.2.1"></times><ci id="alg1.l31.m1.2.2.2.2.2a.cmml" xref="alg1.l31.m1.2.2.2.2.2"><mtext id="alg1.l31.m1.2.2.2.2.2.cmml" xref="alg1.l31.m1.2.2.2.2.2">drop_finished_requests</mtext></ci><list id="alg1.l31.m1.2.2.2.2.3.1.cmml" xref="alg1.l31.m1.2.2.2.2.3.2.1"></list></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l31.m1.2c">finished\_requests\leftarrow B.\text{drop\_finished\_requests}()</annotation></semantics></math>

</div>
<div id="alg1.l32" class="ltx_listingline">
<span id="alg1.l32.1" class="ltx_text ltx_font_bold">for</span> <math id="alg1.l32.m1.1" class="ltx_Math" alttext="req\in finished\_requests" display="inline"><semantics id="alg1.l32.m1.1a"><mrow id="alg1.l32.m1.1.1" xref="alg1.l32.m1.1.1.cmml"><mrow id="alg1.l32.m1.1.1.2" xref="alg1.l32.m1.1.1.2.cmml"><mi id="alg1.l32.m1.1.1.2.2" xref="alg1.l32.m1.1.1.2.2.cmml">r</mi><mo lspace="0em" rspace="0em" id="alg1.l32.m1.1.1.2.1" xref="alg1.l32.m1.1.1.2.1.cmml">​</mo><mi id="alg1.l32.m1.1.1.2.3" xref="alg1.l32.m1.1.1.2.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l32.m1.1.1.2.1a" xref="alg1.l32.m1.1.1.2.1.cmml">​</mo><mi id="alg1.l32.m1.1.1.2.4" xref="alg1.l32.m1.1.1.2.4.cmml">q</mi></mrow><mo id="alg1.l32.m1.1.1.1" xref="alg1.l32.m1.1.1.1.cmml">∈</mo><mrow id="alg1.l32.m1.1.1.3" xref="alg1.l32.m1.1.1.3.cmml"><mi id="alg1.l32.m1.1.1.3.2" xref="alg1.l32.m1.1.1.3.2.cmml">f</mi><mo lspace="0em" rspace="0em" id="alg1.l32.m1.1.1.3.1" xref="alg1.l32.m1.1.1.3.1.cmml">​</mo><mi id="alg1.l32.m1.1.1.3.3" xref="alg1.l32.m1.1.1.3.3.cmml">i</mi><mo lspace="0em" rspace="0em" id="alg1.l32.m1.1.1.3.1a" xref="alg1.l32.m1.1.1.3.1.cmml">​</mo><mi id="alg1.l32.m1.1.1.3.4" xref="alg1.l32.m1.1.1.3.4.cmml">n</mi><mo lspace="0em" rspace="0em" id="alg1.l32.m1.1.1.3.1b" xref="alg1.l32.m1.1.1.3.1.cmml">​</mo><mi id="alg1.l32.m1.1.1.3.5" xref="alg1.l32.m1.1.1.3.5.cmml">i</mi><mo lspace="0em" rspace="0em" id="alg1.l32.m1.1.1.3.1c" xref="alg1.l32.m1.1.1.3.1.cmml">​</mo><mi id="alg1.l32.m1.1.1.3.6" xref="alg1.l32.m1.1.1.3.6.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg1.l32.m1.1.1.3.1d" xref="alg1.l32.m1.1.1.3.1.cmml">​</mo><mi id="alg1.l32.m1.1.1.3.7" xref="alg1.l32.m1.1.1.3.7.cmml">h</mi><mo lspace="0em" rspace="0em" id="alg1.l32.m1.1.1.3.1e" xref="alg1.l32.m1.1.1.3.1.cmml">​</mo><mi id="alg1.l32.m1.1.1.3.8" xref="alg1.l32.m1.1.1.3.8.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l32.m1.1.1.3.1f" xref="alg1.l32.m1.1.1.3.1.cmml">​</mo><mi id="alg1.l32.m1.1.1.3.9" xref="alg1.l32.m1.1.1.3.9.cmml">d</mi><mo lspace="0em" rspace="0em" id="alg1.l32.m1.1.1.3.1g" xref="alg1.l32.m1.1.1.3.1.cmml">​</mo><mi mathvariant="normal" id="alg1.l32.m1.1.1.3.10" xref="alg1.l32.m1.1.1.3.10.cmml">_</mi><mo lspace="0em" rspace="0em" id="alg1.l32.m1.1.1.3.1h" xref="alg1.l32.m1.1.1.3.1.cmml">​</mo><mi id="alg1.l32.m1.1.1.3.11" xref="alg1.l32.m1.1.1.3.11.cmml">r</mi><mo lspace="0em" rspace="0em" id="alg1.l32.m1.1.1.3.1i" xref="alg1.l32.m1.1.1.3.1.cmml">​</mo><mi id="alg1.l32.m1.1.1.3.12" xref="alg1.l32.m1.1.1.3.12.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l32.m1.1.1.3.1j" xref="alg1.l32.m1.1.1.3.1.cmml">​</mo><mi id="alg1.l32.m1.1.1.3.13" xref="alg1.l32.m1.1.1.3.13.cmml">q</mi><mo lspace="0em" rspace="0em" id="alg1.l32.m1.1.1.3.1k" xref="alg1.l32.m1.1.1.3.1.cmml">​</mo><mi id="alg1.l32.m1.1.1.3.14" xref="alg1.l32.m1.1.1.3.14.cmml">u</mi><mo lspace="0em" rspace="0em" id="alg1.l32.m1.1.1.3.1l" xref="alg1.l32.m1.1.1.3.1.cmml">​</mo><mi id="alg1.l32.m1.1.1.3.15" xref="alg1.l32.m1.1.1.3.15.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l32.m1.1.1.3.1m" xref="alg1.l32.m1.1.1.3.1.cmml">​</mo><mi id="alg1.l32.m1.1.1.3.16" xref="alg1.l32.m1.1.1.3.16.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg1.l32.m1.1.1.3.1n" xref="alg1.l32.m1.1.1.3.1.cmml">​</mo><mi id="alg1.l32.m1.1.1.3.17" xref="alg1.l32.m1.1.1.3.17.cmml">t</mi><mo lspace="0em" rspace="0em" id="alg1.l32.m1.1.1.3.1o" xref="alg1.l32.m1.1.1.3.1.cmml">​</mo><mi id="alg1.l32.m1.1.1.3.18" xref="alg1.l32.m1.1.1.3.18.cmml">s</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l32.m1.1b"><apply id="alg1.l32.m1.1.1.cmml" xref="alg1.l32.m1.1.1"><in id="alg1.l32.m1.1.1.1.cmml" xref="alg1.l32.m1.1.1.1"></in><apply id="alg1.l32.m1.1.1.2.cmml" xref="alg1.l32.m1.1.1.2"><times id="alg1.l32.m1.1.1.2.1.cmml" xref="alg1.l32.m1.1.1.2.1"></times><ci id="alg1.l32.m1.1.1.2.2.cmml" xref="alg1.l32.m1.1.1.2.2">𝑟</ci><ci id="alg1.l32.m1.1.1.2.3.cmml" xref="alg1.l32.m1.1.1.2.3">𝑒</ci><ci id="alg1.l32.m1.1.1.2.4.cmml" xref="alg1.l32.m1.1.1.2.4">𝑞</ci></apply><apply id="alg1.l32.m1.1.1.3.cmml" xref="alg1.l32.m1.1.1.3"><times id="alg1.l32.m1.1.1.3.1.cmml" xref="alg1.l32.m1.1.1.3.1"></times><ci id="alg1.l32.m1.1.1.3.2.cmml" xref="alg1.l32.m1.1.1.3.2">𝑓</ci><ci id="alg1.l32.m1.1.1.3.3.cmml" xref="alg1.l32.m1.1.1.3.3">𝑖</ci><ci id="alg1.l32.m1.1.1.3.4.cmml" xref="alg1.l32.m1.1.1.3.4">𝑛</ci><ci id="alg1.l32.m1.1.1.3.5.cmml" xref="alg1.l32.m1.1.1.3.5">𝑖</ci><ci id="alg1.l32.m1.1.1.3.6.cmml" xref="alg1.l32.m1.1.1.3.6">𝑠</ci><ci id="alg1.l32.m1.1.1.3.7.cmml" xref="alg1.l32.m1.1.1.3.7">ℎ</ci><ci id="alg1.l32.m1.1.1.3.8.cmml" xref="alg1.l32.m1.1.1.3.8">𝑒</ci><ci id="alg1.l32.m1.1.1.3.9.cmml" xref="alg1.l32.m1.1.1.3.9">𝑑</ci><ci id="alg1.l32.m1.1.1.3.10.cmml" xref="alg1.l32.m1.1.1.3.10">_</ci><ci id="alg1.l32.m1.1.1.3.11.cmml" xref="alg1.l32.m1.1.1.3.11">𝑟</ci><ci id="alg1.l32.m1.1.1.3.12.cmml" xref="alg1.l32.m1.1.1.3.12">𝑒</ci><ci id="alg1.l32.m1.1.1.3.13.cmml" xref="alg1.l32.m1.1.1.3.13">𝑞</ci><ci id="alg1.l32.m1.1.1.3.14.cmml" xref="alg1.l32.m1.1.1.3.14">𝑢</ci><ci id="alg1.l32.m1.1.1.3.15.cmml" xref="alg1.l32.m1.1.1.3.15">𝑒</ci><ci id="alg1.l32.m1.1.1.3.16.cmml" xref="alg1.l32.m1.1.1.3.16">𝑠</ci><ci id="alg1.l32.m1.1.1.3.17.cmml" xref="alg1.l32.m1.1.1.3.17">𝑡</ci><ci id="alg1.l32.m1.1.1.3.18.cmml" xref="alg1.l32.m1.1.1.3.18">𝑠</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l32.m1.1c">req\in finished\_requests</annotation></semantics></math> <span id="alg1.l32.2" class="ltx_text ltx_font_bold">do</span>

</div>
<div id="alg1.l33" class="ltx_listingline">     <math id="alg1.l33.m1.2" class="ltx_math_unparsed" alttext="T.\text{decrease\_ref\_counter}(req.prefix\_node)" display="inline"><semantics id="alg1.l33.m1.2a"><mrow id="alg1.l33.m1.2b"><mi id="alg1.l33.m1.1.1">T</mi><mo lspace="0em" rspace="0.167em" id="alg1.l33.m1.2.3">.</mo><mtext id="alg1.l33.m1.2.2">decrease_ref_counter</mtext><mrow id="alg1.l33.m1.2.4"><mo stretchy="false" id="alg1.l33.m1.2.4.1">(</mo><mi id="alg1.l33.m1.2.4.2">r</mi><mi id="alg1.l33.m1.2.4.3">e</mi><mi id="alg1.l33.m1.2.4.4">q</mi><mo lspace="0em" rspace="0.167em" id="alg1.l33.m1.2.4.5">.</mo><mi id="alg1.l33.m1.2.4.6">p</mi><mi id="alg1.l33.m1.2.4.7">r</mi><mi id="alg1.l33.m1.2.4.8">e</mi><mi id="alg1.l33.m1.2.4.9">f</mi><mi id="alg1.l33.m1.2.4.10">i</mi><mi id="alg1.l33.m1.2.4.11">x</mi><mi mathvariant="normal" id="alg1.l33.m1.2.4.12">_</mi><mi id="alg1.l33.m1.2.4.13">n</mi><mi id="alg1.l33.m1.2.4.14">o</mi><mi id="alg1.l33.m1.2.4.15">d</mi><mi id="alg1.l33.m1.2.4.16">e</mi><mo stretchy="false" id="alg1.l33.m1.2.4.17">)</mo></mrow></mrow><annotation encoding="application/x-tex" id="alg1.l33.m1.2c">T.\text{decrease\_ref\_counter}(req.prefix\_node)</annotation></semantics></math>

</div>
<div id="alg1.l34" class="ltx_listingline">     <math id="alg1.l34.m1.2" class="ltx_Math" alttext="T.\text{insert}(req)" display="inline"><semantics id="alg1.l34.m1.2a"><mrow id="alg1.l34.m1.2.2.1" xref="alg1.l34.m1.2.2.2.cmml"><mi id="alg1.l34.m1.1.1" xref="alg1.l34.m1.1.1.cmml">T</mi><mo lspace="0em" rspace="0.167em" id="alg1.l34.m1.2.2.1.2" xref="alg1.l34.m1.2.2.2a.cmml">.</mo><mrow id="alg1.l34.m1.2.2.1.1" xref="alg1.l34.m1.2.2.1.1.cmml"><mtext id="alg1.l34.m1.2.2.1.1.3" xref="alg1.l34.m1.2.2.1.1.3a.cmml">insert</mtext><mo lspace="0em" rspace="0em" id="alg1.l34.m1.2.2.1.1.2" xref="alg1.l34.m1.2.2.1.1.2.cmml">​</mo><mrow id="alg1.l34.m1.2.2.1.1.1.1" xref="alg1.l34.m1.2.2.1.1.1.1.1.cmml"><mo stretchy="false" id="alg1.l34.m1.2.2.1.1.1.1.2" xref="alg1.l34.m1.2.2.1.1.1.1.1.cmml">(</mo><mrow id="alg1.l34.m1.2.2.1.1.1.1.1" xref="alg1.l34.m1.2.2.1.1.1.1.1.cmml"><mi id="alg1.l34.m1.2.2.1.1.1.1.1.2" xref="alg1.l34.m1.2.2.1.1.1.1.1.2.cmml">r</mi><mo lspace="0em" rspace="0em" id="alg1.l34.m1.2.2.1.1.1.1.1.1" xref="alg1.l34.m1.2.2.1.1.1.1.1.1.cmml">​</mo><mi id="alg1.l34.m1.2.2.1.1.1.1.1.3" xref="alg1.l34.m1.2.2.1.1.1.1.1.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l34.m1.2.2.1.1.1.1.1.1a" xref="alg1.l34.m1.2.2.1.1.1.1.1.1.cmml">​</mo><mi id="alg1.l34.m1.2.2.1.1.1.1.1.4" xref="alg1.l34.m1.2.2.1.1.1.1.1.4.cmml">q</mi></mrow><mo stretchy="false" id="alg1.l34.m1.2.2.1.1.1.1.3" xref="alg1.l34.m1.2.2.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l34.m1.2b"><apply id="alg1.l34.m1.2.2.2.cmml" xref="alg1.l34.m1.2.2.1"><csymbol cd="ambiguous" id="alg1.l34.m1.2.2.2a.cmml" xref="alg1.l34.m1.2.2.1.2">formulae-sequence</csymbol><ci id="alg1.l34.m1.1.1.cmml" xref="alg1.l34.m1.1.1">𝑇</ci><apply id="alg1.l34.m1.2.2.1.1.cmml" xref="alg1.l34.m1.2.2.1.1"><times id="alg1.l34.m1.2.2.1.1.2.cmml" xref="alg1.l34.m1.2.2.1.1.2"></times><ci id="alg1.l34.m1.2.2.1.1.3a.cmml" xref="alg1.l34.m1.2.2.1.1.3"><mtext id="alg1.l34.m1.2.2.1.1.3.cmml" xref="alg1.l34.m1.2.2.1.1.3">insert</mtext></ci><apply id="alg1.l34.m1.2.2.1.1.1.1.1.cmml" xref="alg1.l34.m1.2.2.1.1.1.1"><times id="alg1.l34.m1.2.2.1.1.1.1.1.1.cmml" xref="alg1.l34.m1.2.2.1.1.1.1.1.1"></times><ci id="alg1.l34.m1.2.2.1.1.1.1.1.2.cmml" xref="alg1.l34.m1.2.2.1.1.1.1.1.2">𝑟</ci><ci id="alg1.l34.m1.2.2.1.1.1.1.1.3.cmml" xref="alg1.l34.m1.2.2.1.1.1.1.1.3">𝑒</ci><ci id="alg1.l34.m1.2.2.1.1.1.1.1.4.cmml" xref="alg1.l34.m1.2.2.1.1.1.1.1.4">𝑞</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l34.m1.2c">T.\text{insert}(req)</annotation></semantics></math>

</div>
<div id="alg1.l35" class="ltx_listingline">
<span id="alg1.l35.1" class="ltx_text ltx_font_bold">return</span> <math id="alg1.l35.m1.1" class="ltx_Math" alttext="finished\_requests" display="inline"><semantics id="alg1.l35.m1.1a"><mrow id="alg1.l35.m1.1.1" xref="alg1.l35.m1.1.1.cmml"><mi id="alg1.l35.m1.1.1.2" xref="alg1.l35.m1.1.1.2.cmml">f</mi><mo lspace="0em" rspace="0em" id="alg1.l35.m1.1.1.1" xref="alg1.l35.m1.1.1.1.cmml">​</mo><mi id="alg1.l35.m1.1.1.3" xref="alg1.l35.m1.1.1.3.cmml">i</mi><mo lspace="0em" rspace="0em" id="alg1.l35.m1.1.1.1a" xref="alg1.l35.m1.1.1.1.cmml">​</mo><mi id="alg1.l35.m1.1.1.4" xref="alg1.l35.m1.1.1.4.cmml">n</mi><mo lspace="0em" rspace="0em" id="alg1.l35.m1.1.1.1b" xref="alg1.l35.m1.1.1.1.cmml">​</mo><mi id="alg1.l35.m1.1.1.5" xref="alg1.l35.m1.1.1.5.cmml">i</mi><mo lspace="0em" rspace="0em" id="alg1.l35.m1.1.1.1c" xref="alg1.l35.m1.1.1.1.cmml">​</mo><mi id="alg1.l35.m1.1.1.6" xref="alg1.l35.m1.1.1.6.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg1.l35.m1.1.1.1d" xref="alg1.l35.m1.1.1.1.cmml">​</mo><mi id="alg1.l35.m1.1.1.7" xref="alg1.l35.m1.1.1.7.cmml">h</mi><mo lspace="0em" rspace="0em" id="alg1.l35.m1.1.1.1e" xref="alg1.l35.m1.1.1.1.cmml">​</mo><mi id="alg1.l35.m1.1.1.8" xref="alg1.l35.m1.1.1.8.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l35.m1.1.1.1f" xref="alg1.l35.m1.1.1.1.cmml">​</mo><mi id="alg1.l35.m1.1.1.9" xref="alg1.l35.m1.1.1.9.cmml">d</mi><mo lspace="0em" rspace="0em" id="alg1.l35.m1.1.1.1g" xref="alg1.l35.m1.1.1.1.cmml">​</mo><mi mathvariant="normal" id="alg1.l35.m1.1.1.10" xref="alg1.l35.m1.1.1.10.cmml">_</mi><mo lspace="0em" rspace="0em" id="alg1.l35.m1.1.1.1h" xref="alg1.l35.m1.1.1.1.cmml">​</mo><mi id="alg1.l35.m1.1.1.11" xref="alg1.l35.m1.1.1.11.cmml">r</mi><mo lspace="0em" rspace="0em" id="alg1.l35.m1.1.1.1i" xref="alg1.l35.m1.1.1.1.cmml">​</mo><mi id="alg1.l35.m1.1.1.12" xref="alg1.l35.m1.1.1.12.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l35.m1.1.1.1j" xref="alg1.l35.m1.1.1.1.cmml">​</mo><mi id="alg1.l35.m1.1.1.13" xref="alg1.l35.m1.1.1.13.cmml">q</mi><mo lspace="0em" rspace="0em" id="alg1.l35.m1.1.1.1k" xref="alg1.l35.m1.1.1.1.cmml">​</mo><mi id="alg1.l35.m1.1.1.14" xref="alg1.l35.m1.1.1.14.cmml">u</mi><mo lspace="0em" rspace="0em" id="alg1.l35.m1.1.1.1l" xref="alg1.l35.m1.1.1.1.cmml">​</mo><mi id="alg1.l35.m1.1.1.15" xref="alg1.l35.m1.1.1.15.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l35.m1.1.1.1m" xref="alg1.l35.m1.1.1.1.cmml">​</mo><mi id="alg1.l35.m1.1.1.16" xref="alg1.l35.m1.1.1.16.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg1.l35.m1.1.1.1n" xref="alg1.l35.m1.1.1.1.cmml">​</mo><mi id="alg1.l35.m1.1.1.17" xref="alg1.l35.m1.1.1.17.cmml">t</mi><mo lspace="0em" rspace="0em" id="alg1.l35.m1.1.1.1o" xref="alg1.l35.m1.1.1.1.cmml">​</mo><mi id="alg1.l35.m1.1.1.18" xref="alg1.l35.m1.1.1.18.cmml">s</mi></mrow><annotation-xml encoding="MathML-Content" id="alg1.l35.m1.1b"><apply id="alg1.l35.m1.1.1.cmml" xref="alg1.l35.m1.1.1"><times id="alg1.l35.m1.1.1.1.cmml" xref="alg1.l35.m1.1.1.1"></times><ci id="alg1.l35.m1.1.1.2.cmml" xref="alg1.l35.m1.1.1.2">𝑓</ci><ci id="alg1.l35.m1.1.1.3.cmml" xref="alg1.l35.m1.1.1.3">𝑖</ci><ci id="alg1.l35.m1.1.1.4.cmml" xref="alg1.l35.m1.1.1.4">𝑛</ci><ci id="alg1.l35.m1.1.1.5.cmml" xref="alg1.l35.m1.1.1.5">𝑖</ci><ci id="alg1.l35.m1.1.1.6.cmml" xref="alg1.l35.m1.1.1.6">𝑠</ci><ci id="alg1.l35.m1.1.1.7.cmml" xref="alg1.l35.m1.1.1.7">ℎ</ci><ci id="alg1.l35.m1.1.1.8.cmml" xref="alg1.l35.m1.1.1.8">𝑒</ci><ci id="alg1.l35.m1.1.1.9.cmml" xref="alg1.l35.m1.1.1.9">𝑑</ci><ci id="alg1.l35.m1.1.1.10.cmml" xref="alg1.l35.m1.1.1.10">_</ci><ci id="alg1.l35.m1.1.1.11.cmml" xref="alg1.l35.m1.1.1.11">𝑟</ci><ci id="alg1.l35.m1.1.1.12.cmml" xref="alg1.l35.m1.1.1.12">𝑒</ci><ci id="alg1.l35.m1.1.1.13.cmml" xref="alg1.l35.m1.1.1.13">𝑞</ci><ci id="alg1.l35.m1.1.1.14.cmml" xref="alg1.l35.m1.1.1.14">𝑢</ci><ci id="alg1.l35.m1.1.1.15.cmml" xref="alg1.l35.m1.1.1.15">𝑒</ci><ci id="alg1.l35.m1.1.1.16.cmml" xref="alg1.l35.m1.1.1.16">𝑠</ci><ci id="alg1.l35.m1.1.1.17.cmml" xref="alg1.l35.m1.1.1.17">𝑡</ci><ci id="alg1.l35.m1.1.1.18.cmml" xref="alg1.l35.m1.1.1.18">𝑠</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l35.m1.1c">finished\_requests</annotation></semantics></math>

</div>
</div>
</figure>
</section>
<section id="S5.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.3 </span>Other Optimizations</h3>

<div id="S5.SS3.p1" class="ltx_para">
<p id="S5.SS3.p1.1" class="ltx_p">The development of new CUDA kernels is essential to enhance the efficiency of KV cache reuse.
Existing systems mostly implement CUDA kernels for two main functions: prefill and decoding. The prefill kernel assumes that the lengths of the query and key are identical. Conversely, the decode kernel assumes that the query length is one.
However, in the case of KV cache reuse, these kernels cannot be directly applied.
We introduce a new “extend” kernel for this computation, capable of reading the KV cache from both contiguous buffers and non-contiguous memory pools. This fused extend kernel operates without extra copy overhead.</p>
</div>
</section>
</section>
<section id="S6" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">6 </span>Evaluation</h2>

<div id="S6.p1" class="ltx_para">
<p id="S6.p1.1" class="ltx_p">SGLang and SGVM are implemented in approximately 9K lines of Python code. SGVM reuses some components from LightLLM <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib35" title="" class="ltx_ref">35</a>]</cite>, an LLM serving system based on PyTorch <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib43" title="" class="ltx_ref">43</a>]</cite> and Triton <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib57" title="" class="ltx_ref">57</a>]</cite>.</p>
</div>
<div id="S6.p2" class="ltx_para">
<p id="S6.p2.1" class="ltx_p">We evaluate the performance of SGLang across a diverse range of LLM workloads. Subsequently, we conduct case studies and ablation studies to demonstrate the effectiveness of specific components.</p>
</div>
<section id="S6.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">6.1 </span>Setup</h3>

<div id="S6.SS1.p1" class="ltx_para ltx_noindent">
<p id="S6.SS1.p1.1" class="ltx_p"><span id="S6.SS1.p1.1.1" class="ltx_text ltx_font_bold">Models.</span>
We test the Llama 2 chat models <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib58" title="" class="ltx_ref">58</a>]</cite>, CodeLlama models <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib47" title="" class="ltx_ref">47</a>]</cite>, and vision language model LLaVA <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib29" title="" class="ltx_ref">29</a>]</cite> models of various sizes. The number of parameters ranges from 7B to 33B. We use float16 precision for Llama and CodeLlama models, and 4-bit quantized precision for LLaVA models.</p>
</div>
<div id="S6.SS1.p2" class="ltx_para ltx_noindent">
<p id="S6.SS1.p2.1" class="ltx_p"><span id="S6.SS1.p2.1.1" class="ltx_text ltx_font_bold">Hardware.</span>
Most experiments are conducted on AWS EC2 G5 instances unless otherwise stated. The GPUs on G5 instances are NVIDIA A10G GPUs, each with 24 GB of memory. The CPUs are AMD EPYC processors. For most experiments, we run 7B models with a single A10G GPU.
To see how our system handles larger models, we also test a 33B model on an A100 GPU (80GB).</p>
</div>
<div id="S6.SS1.p3" class="ltx_para ltx_noindent">
<p id="S6.SS1.p3.1" class="ltx_p"><span id="S6.SS1.p3.1.1" class="ltx_text ltx_font_bold">Baseline.</span>
We compare SGLang against both high-level programming systems with their respective languages and default runtimes, as well as low-level inference engines with standard OpenAI-like Completion APIs. The baselines include:</p>
<ul id="S6.I1" class="ltx_itemize">
<li id="S6.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S6.I1.i1.p1" class="ltx_para">
<p id="S6.I1.i1.p1.1" class="ltx_p">Guidance <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib16" title="" class="ltx_ref">16</a>]</cite>, a language for controlling LLMs. We use Guidance v0.1.4 with the llama.cpp backend.</p>
</div>
</li>
<li id="S6.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S6.I1.i2.p1" class="ltx_para">
<p id="S6.I1.i2.p1.1" class="ltx_p">LMQL <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib4" title="" class="ltx_ref">4</a>]</cite>, a query language for LLMs. We use LMQL v0.7.3 with the Hugging Face Transformers backend.</p>
</div>
</li>
<li id="S6.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S6.I1.i3.p1" class="ltx_para">
<p id="S6.I1.i3.p1.1" class="ltx_p">vLLM <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib23" title="" class="ltx_ref">23</a>]</cite>, a high-throughput inference engine. We use vLLM v0.2.2 and its default API server.</p>
</div>
</li>
</ul>
</div>
<div id="S6.SS1.p4" class="ltx_para ltx_noindent">
<p id="S6.SS1.p4.1" class="ltx_p"><span id="S6.SS1.p4.1.1" class="ltx_text ltx_font_bold">Metrics.</span>
We report two performance metrics: throughput and latency. For throughput, we run a sufficiently large batch of program instances to compute the maximum throughput, comparing the number of program instances executed per second (programs per second, p/s). For latency, we execute a single program at a time without batching and report the average latency for multiple instances. Unless otherwise stated, most of our optimizations do not change the computation results.</p>
</div>
<div id="S6.SS1.p5" class="ltx_para">
<p id="S6.SS1.p5.1" class="ltx_p">While our primary focus is on runtime efficiency, we also conduct several case studies to compare the number of lines of code for productivity.</p>
</div>
</section>
<section id="S6.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">6.2 </span>End-to-End Benchmark</h3>

<section id="S6.SS2.SSS1" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">6.2.1 </span>Few-Shot In-Context Learning</h4>

<div id="S6.SS2.SSS1.p1" class="ltx_para">
<p id="S6.SS2.SSS1.p1.1" class="ltx_p">We start with the most basic few-shot learning prompting.
We evaluate the maximum throughput of systems during batch processing of few-shot learning requests using three benchmarks, each with 5 shots. Results are shown in <a href="#S6.F7" title="Figure 7 ‣ 6.2.1 Few-Shot In-Context Learning ‣ 6.2 End-to-End Benchmark ‣ 6 Evaluation ‣ Efficiently Programming Large Language Models using SGLang" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 7</span></a>.</p>
</div>
<div id="S6.SS2.SSS1.p2" class="ltx_para">
<p id="S6.SS2.SSS1.p2.1" class="ltx_p">MMLU <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib18" title="" class="ltx_ref">18</a>]</cite> is a multi-choice knowledge benchmark where the LLM outputs a single token (A/B/C/D) for answer selection. LMQL exhibits the worst performance due to its slow backend and complex pre- and post-processing. Guidance, being faster than LMQL, benefits from reduced processing overhead and some KV cache reuse. However, both systems support only a batch size of one. vLLM performs similarly to Guidance, but it lacks automatic KV cache reuse for the shared prefix. Although the vLLM paper discusses this feature, the released code does not support it due to the absence of a reliable high-performance kernel.
Even if a high-performance kernel is implemented for vLLM, manual configuration is required to use this feature.
Conversely, SGLang automatically achieves sharing with RadixAttention and attains a throughput that is <math id="S6.SS2.SSS1.p2.1.m1.1" class="ltx_math_unparsed" alttext="4.4\times" display="inline"><semantics id="S6.SS2.SSS1.p2.1.m1.1a"><mrow id="S6.SS2.SSS1.p2.1.m1.1b"><mn id="S6.SS2.SSS1.p2.1.m1.1.1">4.4</mn><mo lspace="0.222em" id="S6.SS2.SSS1.p2.1.m1.1.2">×</mo></mrow><annotation encoding="application/x-tex" id="S6.SS2.SSS1.p2.1.m1.1c">4.4\times</annotation></semantics></math> higher than its closest competitor.</p>
</div>
<div id="S6.SS2.SSS1.p3" class="ltx_para">
<p id="S6.SS2.SSS1.p3.1" class="ltx_p">HellaSwag <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib76" title="" class="ltx_ref">76</a>]</cite> is a multi-choice benchmark where each option is a short string. Unlike MMLU, which decodes a single output token, HellaSwag requires an LLM to calculate the probability of each string choice by performing a forward pass. LMQL exhibits slowness due to high overhead.
vLLM lacks a “select” primitive. Instead, it concatenates each choice with the prompt and computes the probability for the entire concatenated prompt for all choices, leading to redundant calculations.
Guidance, although offering a “select” feature, is limited to a batch size of 1. SGLang outperforms others by <math id="S6.SS2.SSS1.p3.1.m1.1" class="ltx_math_unparsed" alttext="2\times" display="inline"><semantics id="S6.SS2.SSS1.p3.1.m1.1a"><mrow id="S6.SS2.SSS1.p3.1.m1.1b"><mn id="S6.SS2.SSS1.p3.1.m1.1.1">2</mn><mo lspace="0.222em" id="S6.SS2.SSS1.p3.1.m1.1.2">×</mo></mrow><annotation encoding="application/x-tex" id="S6.SS2.SSS1.p3.1.m1.1c">2\times</annotation></semantics></math> with automatic KV cache sharing and batching.</p>
</div>
<div id="S6.SS2.SSS1.p4" class="ltx_para">
<p id="S6.SS2.SSS1.p4.1" class="ltx_p">GSM-8K <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib9" title="" class="ltx_ref">9</a>]</cite> is a benchmark that requires LLMs to generate a reasoning process and produce an integer answer. Since the benchmark demands longer output lengths, batching is crucial for enhanced throughput.
However, LMQL and guidance currently lack support for batching, leading to reduced performance. vLLM is also slow due to its inability to reuse the KV cache for the shared prefix.
In contrast, SGLang achieves a <math id="S6.SS2.SSS1.p4.1.m1.1" class="ltx_math_unparsed" alttext="4.5\times" display="inline"><semantics id="S6.SS2.SSS1.p4.1.m1.1a"><mrow id="S6.SS2.SSS1.p4.1.m1.1b"><mn id="S6.SS2.SSS1.p4.1.m1.1.1">4.5</mn><mo lspace="0.222em" id="S6.SS2.SSS1.p4.1.m1.1.2">×</mo></mrow><annotation encoding="application/x-tex" id="S6.SS2.SSS1.p4.1.m1.1c">4.5\times</annotation></semantics></math> increase in throughput, thanks to the integration of RadixAttention and contiguous batching.</p>
</div>
<figure id="S6.F7" class="ltx_figure"><img src="/html/2312.07104/assets/x7.png" id="S6.F7.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="202" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S6.F7.2.1.1" class="ltx_text" style="font-size:90%;">Figure 7</span>: </span><span id="S6.F7.3.2" class="ltx_text" style="font-size:90%;">Throughput on few-shot in-context learning tasks.</span></figcaption>
</figure>
<div id="S6.SS2.SSS1.p5" class="ltx_para">
<p id="S6.SS2.SSS1.p5.1" class="ltx_p">Due to its overhead, LMQL often performs significantly worse than other systems in both short and long-generation tasks, so we will not include it in our future benchmarks.</p>
</div>
</section>
<section id="S6.SS2.SSS2" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">6.2.2 </span>LLM-Based Agents</h4>

<div id="S6.SS2.SSS2.p1" class="ltx_para">
<p id="S6.SS2.SSS2.p1.1" class="ltx_p">We benchmark on two highly regarded agent frameworks: ReAct agent <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib71" title="" class="ltx_ref">71</a>]</cite>, which processes tasks in an interleaved manner by chaining multiple LLM calls with actions like web search; and generative agents <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib42" title="" class="ltx_ref">42</a>]</cite>, employed in a gaming environment where agents interact with each other and make decisions by invoking LLMs. Both original papers utilized OpenAI models.
To accurately gauge potential performance gains, we mirrored the settings described in the papers and used GPT-3.5-turbo as the backbone model, and then traced the input and output of LLM calls to create a dataset.
For ReAct agents, we used the HotpotQA <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib69" title="" class="ltx_ref">69</a>]</cite> evaluation as reported in the paper. For generative agents, we rewrote the top five functions (out of forty traced) from the game using SGLang, which accounts for approximately 70% of all LLM calls.
We then replayed these traces on our experimental platform.</p>
</div>
<div id="S6.SS2.SSS2.p2" class="ltx_para">
<p id="S6.SS2.SSS2.p2.2" class="ltx_p">As depicted in Figure <a href="#S6.F8" title="Figure 8 ‣ 6.2.2 LLM-Based Agents ‣ 6.2 End-to-End Benchmark ‣ 6 Evaluation ‣ Efficiently Programming Large Language Models using SGLang" class="ltx_ref"><span class="ltx_text ltx_ref_tag">8</span></a>, the ReAct agent, using SGLang, achieves a throughput that is <math id="S6.SS2.SSS2.p2.1.m1.1" class="ltx_math_unparsed" alttext="5.6\times" display="inline"><semantics id="S6.SS2.SSS2.p2.1.m1.1a"><mrow id="S6.SS2.SSS2.p2.1.m1.1b"><mn id="S6.SS2.SSS2.p2.1.m1.1.1">5.6</mn><mo lspace="0.222em" id="S6.SS2.SSS2.p2.1.m1.1.2">×</mo></mrow><annotation encoding="application/x-tex" id="S6.SS2.SSS2.p2.1.m1.1c">5.6\times</annotation></semantics></math> higher than that of vLLM and has only <math id="S6.SS2.SSS2.p2.2.m2.1" class="ltx_Math" alttext="13\%" display="inline"><semantics id="S6.SS2.SSS2.p2.2.m2.1a"><mrow id="S6.SS2.SSS2.p2.2.m2.1.1" xref="S6.SS2.SSS2.p2.2.m2.1.1.cmml"><mn id="S6.SS2.SSS2.p2.2.m2.1.1.2" xref="S6.SS2.SSS2.p2.2.m2.1.1.2.cmml">13</mn><mo id="S6.SS2.SSS2.p2.2.m2.1.1.1" xref="S6.SS2.SSS2.p2.2.m2.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S6.SS2.SSS2.p2.2.m2.1b"><apply id="S6.SS2.SSS2.p2.2.m2.1.1.cmml" xref="S6.SS2.SSS2.p2.2.m2.1.1"><csymbol cd="latexml" id="S6.SS2.SSS2.p2.2.m2.1.1.1.cmml" xref="S6.SS2.SSS2.p2.2.m2.1.1.1">percent</csymbol><cn type="integer" id="S6.SS2.SSS2.p2.2.m2.1.1.2.cmml" xref="S6.SS2.SSS2.p2.2.m2.1.1.2">13</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS2.SSS2.p2.2.m2.1c">13\%</annotation></semantics></math> of its latency. This improvement primarily results from the agent’s process of appending current states (thoughts, actions, and observations) to the prompt for subsequent LLM calls. Our RadixAttention mechanism effectively identifies and minimizes redundant computations, a capability not present in the current inference engines. This mechanism also significantly reduces memory usage, allowing more questions to be addressed by a greater number of agents, thereby optimizing GPU utilization and enhancing throughput.</p>
</div>
<div id="S6.SS2.SSS2.p3" class="ltx_para">
<p id="S6.SS2.SSS2.p3.1" class="ltx_p">In the case of the generative agent task, the gaming setting’s constraints permit only a single LLM call per simulation to maintain game logic consistency. In this case, SGLang still outperforms vLLM by 30% in both throughput and latency. This efficiency is due to the caching of agents’ predefined routines, reducing redundant computations.
The task prompt templates contain multiple arguments that vary at different speeds based on timestamps, resulting in complex prefix reuse patterns. Nevertheless, the SGLang runtime can automatically detect these patterns and reuse the KV cache.</p>
</div>
<figure id="S6.F8" class="ltx_figure"><img src="/html/2312.07104/assets/x8.png" id="S6.F8.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="415" height="206" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S6.F8.2.1.1" class="ltx_text" style="font-size:90%;">Figure 8</span>: </span><span id="S6.F8.3.2" class="ltx_text" style="font-size:90%;">Throughput and latency on agent tasks.</span></figcaption>
</figure>
<figure id="S6.F9" class="ltx_figure"><img src="/html/2312.07104/assets/x9.png" id="S6.F9.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="97" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S6.F9.2.1.1" class="ltx_text" style="font-size:90%;">Figure 9</span>: </span><span id="S6.F9.3.2" class="ltx_text" style="font-size:90%;">(a) Throughput on reasoning tasks. (b) Latency on reasoning tasks. (c) Throughput on content generation tasks. (d) Latency on content generation tasks. (e) Throughput on long-context tasks.</span></figcaption>
</figure>
</section>
<section id="S6.SS2.SSS3" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">6.2.3 </span>Reasoning</h4>

<div id="S6.SS2.SSS3.p1" class="ltx_para">
<p id="S6.SS2.SSS3.p1.1" class="ltx_p">We benchmark the performance of systems using advanced prompting techniques for reasoning tasks. These tasks involve solving GSM-8K problems in a zero-shot setting. Both throughput and latency metrics are reported in <a href="#S6.F9" title="Figure 9 ‣ 6.2.2 LLM-Based Agents ‣ 6.2 End-to-End Benchmark ‣ 6 Evaluation ‣ Efficiently Programming Large Language Models using SGLang" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 9</span></a> (a)(b).</p>
</div>
<div id="S6.SS2.SSS3.p2" class="ltx_para">
<p id="S6.SS2.SSS3.p2.1" class="ltx_p">Multi-chain reasoning <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib73" title="" class="ltx_ref">73</a>]</cite> involves generating several solutions to one question and using meta-reasoning across these solutions for the final answer.
This process allows for sharing the question and parallelizing the generation of multiple solutions.
For each solution generation, we use various eliciting prompts such as "Think step-by-step" and "Take a deep breath". Unlike simple parallel sampling, this flexible approach with different eliciting prompts is not supported by the vLLM API. Hence, vLLM exhibits lower throughput due to its inability to share prefixes flexibly.
Guidance has lower throughput due to no batching.
In contrast, SGLang allows users to intuitively implement this technique without worrying about manual optimizations, yet achieve high throughput.
The improvement in throughput is not as substantial as in previous cases of few-shot learning, due to the zero-shot setting having a shorter shareable prefix.</p>
</div>
<div id="S6.SS2.SSS3.p3" class="ltx_para">
<p id="S6.SS2.SSS3.p3.1" class="ltx_p">Regarding latency, the SGLang interpreter can automatically resolve dependencies and submit requests in parallel. For a fair comparison, we directly translate the SGLang program into an equivalent Python program without explicitly using threading or asyncio to avoid added complexity. Consequently, the translated program does not parallelize parallelizable requests, resulting in higher latency.</p>
</div>
<div id="S6.SS2.SSS3.p4" class="ltx_para">
<p id="S6.SS2.SSS3.p4.1" class="ltx_p">Tree-of-thought <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib70" title="" class="ltx_ref">70</a>]</cite> incorporates LLMs into the breadth-first tree-search algorithm. The LLMs must propose the next steps for branch exploration, execute these proposals, and evaluate the solutions.
This method leads to complex sharing patterns beyond the capabilities of existing systems.
Once again, SGLang surpasses others in terms of both higher throughput and lower latency in this benchmark.</p>
</div>
</section>
<section id="S6.SS2.SSS4" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">6.2.4 </span>Content Generation</h4>

<div id="S6.SS2.SSS4.p1" class="ltx_para">
<p id="S6.SS2.SSS4.p1.1" class="ltx_p">We evaluate two advanced prompting methods for content generation: tip generation and article review.
The skeleton-of-thought technique <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib37" title="" class="ltx_ref">37</a>]</cite> first creates a brief outline of tips, then elaborates each into a detailed paragraph. The branch-solve-merge approach <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib48" title="" class="ltx_ref">48</a>]</cite> assesses an article from multiple dimensions and then assigns a final grade.
Results are shown in <a href="#S6.F9" title="Figure 9 ‣ 6.2.2 LLM-Based Agents ‣ 6.2 End-to-End Benchmark ‣ 6 Evaluation ‣ Efficiently Programming Large Language Models using SGLang" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 9</span></a> (c)(d).
In both cases, there are opportunities for sharing and intra-program parallelism. SGLang systematically exploits them through an intuitive programming model, achieving higher throughput and reduced latency.</p>
</div>
</section>
<section id="S6.SS2.SSS5" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">6.2.5 </span>Long Document Processing</h4>

<div id="S6.SS2.SSS5.p1" class="ltx_para">
<p id="S6.SS2.SSS5.p1.1" class="ltx_p">We benchmark the performance on long document processing tasks. When the document is long, KV cache reuse becomes more important as the forward pass to encode the prompt becomes significantly longer. We use CodeLlama models, which were fine-tuned on a context length of 16K tokens. On our setup, running a 7B model in fp16 precision on a 24 GB GPU, we can store at most 15K tokens. We select inputs exceeding 10K tokens in length, meaning the system can process only one request at a time at most.
The first benchmark tests answering questions using multiple document segments from a paper, akin to a retrieval augmented generation setup. The second benchmark involves summarizing a city’s information from its Wikipedia page and outputting a JSON object.</p>
</div>
<div id="S6.SS2.SSS5.p2" class="ltx_para">
<p id="S6.SS2.SSS5.p2.1" class="ltx_p">For the first benchmark, we employ the fork/join primitives in SGLang to implement parallel-context window <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib46" title="" class="ltx_ref">46</a>]</cite>. By using this technique, we can encode multiple documents separately in parallel and concatenate the resulting KV cache, without letting them attend to each other. This approach reduces the quadratic computational cost of self-attention for long prompts. While this optimization alters the computation, it does not negatively impact accuracy in our benchmark.
For the second benchmark, we reuse the KV cache of the long prompts when decoding multiple fields of the JSON object. The results are shown in <a href="#S6.F9" title="Figure 9 ‣ 6.2.2 LLM-Based Agents ‣ 6.2 End-to-End Benchmark ‣ 6 Evaluation ‣ Efficiently Programming Large Language Models using SGLang" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 9</span></a> (e). SGLang achieves 1.2× and 2.9× improvement on these two benchmarks, respectively.
Since the maximum batch size is one, latency improvements are the same as throughput improvements.</p>
</div>
</section>
</section>
<section id="S6.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">6.3 </span>Case Study</h3>

<section id="S6.SS3.SSS1" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">6.3.1 </span>Vision Language Models</h4>

<div id="S6.SS3.SSS1.p1" class="ltx_para">
<p id="S6.SS3.SSS1.p1.1" class="ltx_p">We conduct a case study to demonstrate the extensibility of SGLang by supporting a new vision language model, LLaVA v1.5 <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib29" title="" class="ltx_ref">29</a>]</cite>, on a new hardware platform, Apple Silicon.
To achieve this, we add a new primitive, <span id="S6.SS3.SSS1.p1.1.1" class="ltx_text ltx_font_typewriter">sgl.image</span>, to accept image inputs, and incorporate a new backend, llama.cpp <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib15" title="" class="ltx_ref">15</a>]</cite>, an inference engine compatible with a wide range of hardware.</p>
</div>
<div id="S6.SS3.SSS1.p2" class="ltx_para">
<p id="S6.SS3.SSS1.p2.1" class="ltx_p">We integrate <span id="S6.SS3.SSS1.p2.1.1" class="ltx_text ltx_font_typewriter">sgl.image</span> as a new IR node into the interpreter and reuse the SGVM protocol to build a server based on llama.cpp. This integration is achieved in around 400 lines of code. Additionally, the code that uses <span id="S6.SS3.SSS1.p2.1.2" class="ltx_text ltx_font_typewriter">sgl.image</span> can be executed on GPT-4 Vision through the OpenAI back-end without code modification.
We create a cache for image tokens based on the image’s hash ID, similar to how we maintain RadixAttention.
We run the LLaVA-Bench and report the throughput using SGLang and the default llama.cpp server on an M2 Ultra. The LLaVA-Bench contains a total of 24 images and 60 questions, with 2.5 questions on each image on average.
SGLang increases the throughput of llama.cpp by 1.7<math id="S6.SS3.SSS1.p2.1.m1.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S6.SS3.SSS1.p2.1.m1.1a"><mo id="S6.SS3.SSS1.p2.1.m1.1.1" xref="S6.SS3.SSS1.p2.1.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S6.SS3.SSS1.p2.1.m1.1b"><times id="S6.SS3.SSS1.p2.1.m1.1.1.cmml" xref="S6.SS3.SSS1.p2.1.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S6.SS3.SSS1.p2.1.m1.1c">\times</annotation></semantics></math>. The speedup is mainly affected by the number of questions per shared image and the output length.</p>
</div>
</section>
<section id="S6.SS3.SSS2" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">6.3.2 </span>Compiler Optimizations</h4>

<div id="S6.SS3.SSS2.p1" class="ltx_para">
<p id="S6.SS3.SSS2.p1.1" class="ltx_p">We evaluate the effectiveness of two compiler optimizations. The first is GPT-4-assisted code movement for prefix sharing. We collect 20 prompt templates from the internet and implement them in SGLang.
We utilize 5 of these templates for few-shot training examples and the remaining 15 as test cases. Our evaluation shows that, for 12 out of the 15 templates, GPT-4 successfully reorders the graph nodes without altering the semantics, as confirmed by manual inspection of the modified prompts.
On average, this optimization results in a 60-token increase in the sharable prefix length, showcasing GPT-4’s effectiveness. Failures in creating optimized prompt order come from an incorrect understanding of the semantic meaning behind the graph nodes. It is too aggressive and puts all constants upfront even when such ordering changes the original semantics.
This case study aims to explore the use of GPT-4 for compiler optimizations. More work is needed to make these kinds of optimizations reliable in the future.</p>
</div>
<div id="S6.SS3.SSS2.p2" class="ltx_para">
<p id="S6.SS3.SSS2.p2.2" class="ltx_p">The second optimization involves prefetching annotation insertion.
For frequently used long prompts, we cache them in CPU memory. The compiler automatically inserts prefetch nodes into the computational graph. During runtime, when the server encounters a prefetch node, it evicts nodes from the radix tree and swaps in the cached prefix.
As the cost of swapping can be substantially overlapped, this reduces the first-token latency of the subsequent request.
We test SGLang programs that contain function calls with a <math id="S6.SS3.SSS2.p2.1.m1.1" class="ltx_Math" alttext="4" display="inline"><semantics id="S6.SS3.SSS2.p2.1.m1.1a"><mn id="S6.SS3.SSS2.p2.1.m1.1.1" xref="S6.SS3.SSS2.p2.1.m1.1.1.cmml">4</mn><annotation-xml encoding="MathML-Content" id="S6.SS3.SSS2.p2.1.m1.1b"><cn type="integer" id="S6.SS3.SSS2.p2.1.m1.1.1.cmml" xref="S6.SS3.SSS2.p2.1.m1.1.1">4</cn></annotation-xml><annotation encoding="application/x-tex" id="S6.SS3.SSS2.p2.1.m1.1c">4</annotation></semantics></math>k prefix length. By prefetching the <math id="S6.SS3.SSS2.p2.2.m2.1" class="ltx_Math" alttext="4" display="inline"><semantics id="S6.SS3.SSS2.p2.2.m2.1a"><mn id="S6.SS3.SSS2.p2.2.m2.1.1" xref="S6.SS3.SSS2.p2.2.m2.1.1.cmml">4</mn><annotation-xml encoding="MathML-Content" id="S6.SS3.SSS2.p2.2.m2.1b"><cn type="integer" id="S6.SS3.SSS2.p2.2.m2.1.1.cmml" xref="S6.SS3.SSS2.p2.2.m2.1.1">4</cn></annotation-xml><annotation encoding="application/x-tex" id="S6.SS3.SSS2.p2.2.m2.1c">4</annotation></semantics></math>k prefix, the first token latency of the called function has been reduced from 1 second to 0.2 seconds, which is a 80% reduction.</p>
</div>
</section>
<section id="S6.SS3.SSS3" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">6.3.3 </span>Productivity</h4>

<div id="S6.SS3.SSS3.p1" class="ltx_para">
<p id="S6.SS3.SSS3.p1.1" class="ltx_p">We conducted a case study to demonstrate SGLang’s productivity improvement by comparing its Lines of Code (LoC) with OpenAI APIs. We designed a prompting flow for article analysis: if it’s about a celebrity, the program generates a bio and introduction; otherwise, it summarizes the article. Implementing this flow in both OpenAI APIs and SGLang, we counted effective LoC, excluding comments and empty lines. The results show the OpenAI API required 206 lines, while SGLang needed only 91, marking a 55% reduction.</p>
</div>
<div id="S6.SS3.SSS3.p2" class="ltx_para">
<p id="S6.SS3.SSS3.p2.1" class="ltx_p">We delve into the implementations and summarize the insights regarding why SGLang is capable of using fewer LoC.
First, whereas users must manually extract content from OpenAI API responses and chain them to form a multi-step flow, SGLang simplifies this process by leveraging the string concatenation operator.
Second, since the flow involves several calls with restricted answers, such as true/false and integers, users are required to construct an appropriate logit bias to ensure OpenAI returns one of the expected answers. In contrast, SGLang provides primitives that automatically construct logit bias for this purpose, allowing users to merely specify the acceptable data type and choices.
Third, in cases where skeleton-based prompting is used to reduce end-to-end latency, OpenAI API users need to initiate a thread pool for parallel request submission explicitly. However, SGLang’s built-in primitive <span id="S6.SS3.SSS3.p2.1.1" class="ltx_text ltx_font_typewriter">fork</span> automatically parallelizes requests.</p>
</div>
</section>
<section id="S6.SS3.SSS4" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">6.3.4 </span>Larger Models</h4>

<div id="S6.SS3.SSS4.p1" class="ltx_para">
<p id="S6.SS3.SSS4.p1.1" class="ltx_p">While most of our experiments are conducted using a 7B model, our techniques also perform effectively on larger models. We test a 13B model on an A100 (40GB) and a 33B model on an A100 (80GB) using the ReAct agent benchmark, achieving a similar speedup (<math id="S6.SS3.SSS4.p1.1.m1.1" class="ltx_math_unparsed" alttext="5\times" display="inline"><semantics id="S6.SS3.SSS4.p1.1.m1.1a"><mrow id="S6.SS3.SSS4.p1.1.m1.1b"><mn id="S6.SS3.SSS4.p1.1.m1.1.1">5</mn><mo lspace="0.222em" id="S6.SS3.SSS4.p1.1.m1.1.2">×</mo></mrow><annotation encoding="application/x-tex" id="S6.SS3.SSS4.p1.1.m1.1c">5\times</annotation></semantics></math>).</p>
</div>
</section>
</section>
<section id="S6.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">6.4 </span>Ablation Study</h3>

<section id="S6.SS4.SSS1" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">6.4.1 </span>Scheduling Policy</h4>

<figure id="S6.F10" class="ltx_figure"><img src="/html/2312.07104/assets/x10.png" id="S6.F10.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="200" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S6.F10.2.1.1" class="ltx_text" style="font-size:90%;">Figure 10</span>: </span><span id="S6.F10.3.2" class="ltx_text" style="font-size:90%;">Performance of different scheduling policies.</span></figcaption>
</figure>
<div id="S6.SS4.SSS1.p1" class="ltx_para">
<p id="S6.SS4.SSS1.p1.1" class="ltx_p">We evaluate the effectiveness of our cache-aware scheduling by comparing it with several other policies. The results are shown in <a href="#S6.F10" title="Figure 10 ‣ 6.4.1 Scheduling Policy ‣ 6.4 Ablation Study ‣ 6 Evaluation ‣ Efficiently Programming Large Language Models using SGLang" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 10</span></a>. “LMP”, or longest matching prefix, is the default policy outlined in <a href="#S5.SS2" title="5.2 Cache-Aware Scheduling ‣ 5 Runtime with RadixAttention ‣ Efficiently Programming Large Language Models using SGLang" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Section 5.2</span></a>. “Random” employs a random order. “FCFS” adheres to a first-come-first-serve policy. “No interpreter help” omits the prefix hints from the interpreter, which is explained in detail below.
Although RadixAttention can automatically detect shared prefixes, our runtime implementation is simplified: we only compare a pending request with the existing cache tree. We do not construct a tree index for all requests in the waiting queue. This approach may lead to suboptimal performance when there is a large batch of pending requests that share a prefix but have yet to be processed and inserted into the tree. To mitigate this, we use the interpreter to help with the scheduling. When the interpreter identifies a highly likely shared prefix (e.g., during a fork), it issues a forward request for this prefix first and stores it in the tree, and then it sends out the requests that can reuse the prefix. This method leverages hints from the interpreter to lessen the load on backend scheduling, underscoring the benefits of our co-design strategy.
As depicted in <a href="#S6.F10" title="Figure 10 ‣ 6.4.1 Scheduling Policy ‣ 6.4 Ablation Study ‣ 6 Evaluation ‣ Efficiently Programming Large Language Models using SGLang" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 10</span></a>, “LPM” exhibits superior performance, while other policies struggle with a lower cache hit rate and reduced throughput.
All policies perform similarly for simple tasks like MMLU.</p>
</div>
</section>
<section id="S6.SS4.SSS2" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">6.4.2 </span>Optimization Breakdown</h4>

<div id="S6.SS4.SSS2.p1" class="ltx_para">
<p id="S6.SS4.SSS2.p1.1" class="ltx_p">We study the effectiveness of optimizations by disabling each one individually. <a href="#S6.F11" title="Figure 11 ‣ 6.4.2 Optimization Breakdown ‣ 6.4 Ablation Study ‣ 6 Evaluation ‣ Efficiently Programming Large Language Models using SGLang" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 11</span></a> illustrates the results.
“No Extend Kernel” disables our optimized kernel, leading to degraded performance, particularly in long-context tasks. “No Interpreter Parallelism” turns off intra-program parallelism, resulting in reduced parallelism and cache locality.
“No Cache” deactivates the cache, significantly deteriorating throughput across all benchmarks. “No Radix Tree” uses a list instead of a tree to manage the cache, leading to increased memory usage and copy overhead, and consequently, lower performance.</p>
</div>
<figure id="S6.F11" class="ltx_figure"><img src="/html/2312.07104/assets/x11.png" id="S6.F11.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="186" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S6.F11.2.1.1" class="ltx_text" style="font-size:90%;">Figure 11</span>: </span><span id="S6.F11.3.2" class="ltx_text" style="font-size:90%;">Optimization breakdown.</span></figcaption>
</figure>
</section>
<section id="S6.SS4.SSS3" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">6.4.3 </span>Improvements under Different Sharing Settings</h4>

<div id="S6.SS4.SSS3.p1" class="ltx_para">
<p id="S6.SS4.SSS3.p1.1" class="ltx_p">We study how the improvement of prefix sharing is influenced by different parameters, including the lengths of shareable and unshareable inputs, and sharing structure configurations.</p>
</div>
<div id="S6.SS4.SSS3.p2" class="ltx_para">
<p id="S6.SS4.SSS3.p2.1" class="ltx_p">As shown in <a href="#S6.F12" title="Figure 12 ‣ 6.4.4 RadixAttention Overhead Analysis ‣ 6.4 Ablation Study ‣ 6 Evaluation ‣ Efficiently Programming Large Language Models using SGLang" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 12</span></a>, in the left subfigure, we establish a base setting with a shareable input length of 256, an unshareable input length of 256, and an output length of 256. We then gradually increase either the shareable input length or the unshareable length and observe the changes in throughput. The results demonstrate that the RadixAttention cache significantly enhances throughput compared to the “No Cache” scenario, with more notable improvements observed when the shareable length is longer.</p>
</div>
<div id="S6.SS4.SSS3.p3" class="ltx_para">
<p id="S6.SS4.SSS3.p3.1" class="ltx_p">In the right subfigure, we set a base scenario with 5 articles and 2 questions per article, where the article comprises 1000 tokens and each question 256 tokens. There are <math id="S6.SS4.SSS3.p3.1.m1.1" class="ltx_Math" alttext="5\times 2=10" display="inline"><semantics id="S6.SS4.SSS3.p3.1.m1.1a"><mrow id="S6.SS4.SSS3.p3.1.m1.1.1" xref="S6.SS4.SSS3.p3.1.m1.1.1.cmml"><mrow id="S6.SS4.SSS3.p3.1.m1.1.1.2" xref="S6.SS4.SSS3.p3.1.m1.1.1.2.cmml"><mn id="S6.SS4.SSS3.p3.1.m1.1.1.2.2" xref="S6.SS4.SSS3.p3.1.m1.1.1.2.2.cmml">5</mn><mo lspace="0.222em" rspace="0.222em" id="S6.SS4.SSS3.p3.1.m1.1.1.2.1" xref="S6.SS4.SSS3.p3.1.m1.1.1.2.1.cmml">×</mo><mn id="S6.SS4.SSS3.p3.1.m1.1.1.2.3" xref="S6.SS4.SSS3.p3.1.m1.1.1.2.3.cmml">2</mn></mrow><mo id="S6.SS4.SSS3.p3.1.m1.1.1.1" xref="S6.SS4.SSS3.p3.1.m1.1.1.1.cmml">=</mo><mn id="S6.SS4.SSS3.p3.1.m1.1.1.3" xref="S6.SS4.SSS3.p3.1.m1.1.1.3.cmml">10</mn></mrow><annotation-xml encoding="MathML-Content" id="S6.SS4.SSS3.p3.1.m1.1b"><apply id="S6.SS4.SSS3.p3.1.m1.1.1.cmml" xref="S6.SS4.SSS3.p3.1.m1.1.1"><eq id="S6.SS4.SSS3.p3.1.m1.1.1.1.cmml" xref="S6.SS4.SSS3.p3.1.m1.1.1.1"></eq><apply id="S6.SS4.SSS3.p3.1.m1.1.1.2.cmml" xref="S6.SS4.SSS3.p3.1.m1.1.1.2"><times id="S6.SS4.SSS3.p3.1.m1.1.1.2.1.cmml" xref="S6.SS4.SSS3.p3.1.m1.1.1.2.1"></times><cn type="integer" id="S6.SS4.SSS3.p3.1.m1.1.1.2.2.cmml" xref="S6.SS4.SSS3.p3.1.m1.1.1.2.2">5</cn><cn type="integer" id="S6.SS4.SSS3.p3.1.m1.1.1.2.3.cmml" xref="S6.SS4.SSS3.p3.1.m1.1.1.2.3">2</cn></apply><cn type="integer" id="S6.SS4.SSS3.p3.1.m1.1.1.3.cmml" xref="S6.SS4.SSS3.p3.1.m1.1.1.3">10</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS4.SSS3.p3.1.m1.1c">5\times 2=10</annotation></semantics></math> requests in this base setting, with each article being shared by two questions. Subsequently, we increment either the number of articles or the number of questions per article. The findings indicate that the RadixAttention cache exhibits superior performance when the number of questions per article increases, as this creates more opportunities for sharing.</p>
</div>
</section>
<section id="S6.SS4.SSS4" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">6.4.4 </span>RadixAttention Overhead Analysis</h4>

<div id="S6.SS4.SSS4.p1" class="ltx_para">
<p id="S6.SS4.SSS4.p1.1" class="ltx_p">We analyze the overhead of maintaining the radix tree at runtime. The tree structure is maintained on the CPU. In a typical scenario, which has a shareable input length of 256, an unshareable input length of 256, and 128 requests (organized into 16 subtrees, with 8 branches for each subtree), the forward pass in our setup takes 17.6 seconds. Meanwhile, the tree operation only takes 0.07 seconds.
We also construct a request trace with no cache hits and observe the tree maintenance incurs only a 0.5% overhead.
Therefore, the overhead is negligible. Furthermore, the overhead of tree maintenance can be further reduced by using a high-performance C++ implementation and by overlapping CPU and GPU operations.</p>
</div>
<figure id="S6.F12" class="ltx_figure"><img src="/html/2312.07104/assets/x12.png" id="S6.F12.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="216" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S6.F12.2.1.1" class="ltx_text" style="font-size:90%;">Figure 12</span>: </span><span id="S6.F12.3.2" class="ltx_text" style="font-size:90%;">Improvements under Different Sharing Settings</span></figcaption>
</figure>
</section>
</section>
</section>
<section id="S7" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">7 </span>Discussion and Limitations</h2>

<div id="S7.p1" class="ltx_para">
<p id="S7.p1.1" class="ltx_p">Here are SGLang’s limitations and future development plans:</p>
<ul id="S7.I1" class="ltx_itemize">
<li id="S7.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S7.I1.i1.p1" class="ltx_para">
<p id="S7.I1.i1.p1.1" class="ltx_p">Our interpreter runs arbitrary Python code, but the tracing and compiler lack support for data-dependent control flows. Introducing new control flow nodes to the SGLang IR, similar to TensorFlow’s dynamic flow <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib75" title="" class="ltx_ref">75</a>]</cite>, is required to address this.</p>
</div>
</li>
<li id="S7.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S7.I1.i2.p1" class="ltx_para">
<p id="S7.I1.i2.p1.1" class="ltx_p">The current tokenization algorithm may create artifacts at prompt and generation boundaries. Implementing the token healing technique <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib16" title="" class="ltx_ref">16</a>]</cite> could resolve this.</p>
</div>
</li>
<li id="S7.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S7.I1.i3.p1" class="ltx_para">
<p id="S7.I1.i3.p1.1" class="ltx_p">We have not yet implemented grammar-constrained decoding. Efficient implementation would necessitate techniques from <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib65" title="" class="ltx_ref">65</a>, <a href="#bib.bib22" title="" class="ltx_ref">22</a>]</cite> and careful batching.</p>
</div>
</li>
<li id="S7.I1.i4" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S7.I1.i4.p1" class="ltx_para">
<p id="S7.I1.i4.p1.1" class="ltx_p">Plans include integrating more input and output modalities, such as incorporating text-to-image models into SGLang and exploring audio and video capabilities.</p>
</div>
</li>
</ul>
</div>
</section>
<section id="S8" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">8 </span>Related Work</h2>

<section id="S8.SS0.SSS0.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Programming with Language Models</h5>

<div id="S8.SS0.SSS0.Px1.p1" class="ltx_para">
<p id="S8.SS0.SSS0.Px1.p1.1" class="ltx_p">The trend in programming LLM includes the development of better prompting techniques <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib64" title="" class="ltx_ref">64</a>, <a href="#bib.bib37" title="" class="ltx_ref">37</a>, <a href="#bib.bib70" title="" class="ltx_ref">70</a>, <a href="#bib.bib3" title="" class="ltx_ref">3</a>]</cite>, tools usage <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib49" title="" class="ltx_ref">49</a>, <a href="#bib.bib50" title="" class="ltx_ref">50</a>, <a href="#bib.bib44" title="" class="ltx_ref">44</a>]</cite>, agent frameworks <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib67" title="" class="ltx_ref">67</a>, <a href="#bib.bib54" title="" class="ltx_ref">54</a>, <a href="#bib.bib71" title="" class="ltx_ref">71</a>, <a href="#bib.bib53" title="" class="ltx_ref">53</a>, <a href="#bib.bib27" title="" class="ltx_ref">27</a>, <a href="#bib.bib41" title="" class="ltx_ref">41</a>, <a href="#bib.bib42" title="" class="ltx_ref">42</a>]</cite>, and programming systems <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib4" title="" class="ltx_ref">4</a>, <a href="#bib.bib16" title="" class="ltx_ref">16</a>, <a href="#bib.bib24" title="" class="ltx_ref">24</a>, <a href="#bib.bib21" title="" class="ltx_ref">21</a>, <a href="#bib.bib30" title="" class="ltx_ref">30</a>, <a href="#bib.bib65" title="" class="ltx_ref">65</a>]</cite>. Among these, Guidance and LMQL are the most similar to our work. We discuss the differences between SGLang and existing LLM programming systems in <a href="#S2.SS4" title="2.4 Existing LLM Programming Systems ‣ 2 Background ‣ Efficiently Programming Large Language Models using SGLang" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Section 2.4</span></a> and evaluate their performance in <a href="#S6" title="6 Evaluation ‣ Efficiently Programming Large Language Models using SGLang" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Section 6</span></a>. In short, the co-design approach of SGLang significantly enhances execution efficiency.</p>
</div>
</section>
<section id="S8.SS0.SSS0.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Inference Optimizations for LLMs.</h5>

<div id="S8.SS0.SSS0.Px2.p1" class="ltx_para">
<p id="S8.SS0.SSS0.Px2.p1.1" class="ltx_p">Common inference optimizations for LLMs include batching <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib74" title="" class="ltx_ref">74</a>]</cite>, memory optimizations <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib52" title="" class="ltx_ref">52</a>, <a href="#bib.bib23" title="" class="ltx_ref">23</a>, <a href="#bib.bib2" title="" class="ltx_ref">2</a>]</cite>, GPU kernel optimizations <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib10" title="" class="ltx_ref">10</a>]</cite>, model parallelism <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib45" title="" class="ltx_ref">45</a>]</cite>, parameter sharing <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib51" title="" class="ltx_ref">51</a>]</cite>, speculative execution <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib56" title="" class="ltx_ref">56</a>, <a href="#bib.bib25" title="" class="ltx_ref">25</a>, <a href="#bib.bib34" title="" class="ltx_ref">34</a>]</cite>, scheduling <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib66" title="" class="ltx_ref">66</a>, <a href="#bib.bib17" title="" class="ltx_ref">17</a>]</cite>, caching <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib55" title="" class="ltx_ref">55</a>]</cite>,
quantization <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib68" title="" class="ltx_ref">68</a>, <a href="#bib.bib28" title="" class="ltx_ref">28</a>, <a href="#bib.bib11" title="" class="ltx_ref">11</a>, <a href="#bib.bib14" title="" class="ltx_ref">14</a>]</cite>, and sparsification <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib13" title="" class="ltx_ref">13</a>]</cite>. These optimizations are complementary to our work and can be, or have already been, integrated into our backend.</p>
</div>
</section>
<section id="S8.SS0.SSS0.Px3" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Compilers for Machine Learning.</h5>

<div id="S8.SS0.SSS0.Px3.p1" class="ltx_para">
<p id="S8.SS0.SSS0.Px3.p1.1" class="ltx_p">Various domain-specific compilers for machine learning have been developed. These compilers are capable of optimizing a single tensor operator <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib7" title="" class="ltx_ref">7</a>, <a href="#bib.bib12" title="" class="ltx_ref">12</a>, <a href="#bib.bib77" title="" class="ltx_ref">77</a>, <a href="#bib.bib33" title="" class="ltx_ref">33</a>, <a href="#bib.bib57" title="" class="ltx_ref">57</a>, <a href="#bib.bib72" title="" class="ltx_ref">72</a>, <a href="#bib.bib62" title="" class="ltx_ref">62</a>]</cite>, a computational graph <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib20" title="" class="ltx_ref">20</a>, <a href="#bib.bib59" title="" class="ltx_ref">59</a>]</cite>, a distributed computation graph <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib78" title="" class="ltx_ref">78</a>, <a href="#bib.bib32" title="" class="ltx_ref">32</a>]</cite>, and classical algorithms <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib36" title="" class="ltx_ref">36</a>]</cite>. Distinct from these more general-purpose or low-level compilers, SGLang is specifically designed for LLM applications and incorporates LLM-specific optimizations.</p>
</div>
</section>
</section>
<section id="S9" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">9 </span>Conclusion</h2>

<div id="S9.p1" class="ltx_para">
<p id="S9.p1.1" class="ltx_p">In this paper, we introduce SGLang, an efficient programming system for LLMs.
SGLang greatly accelerates common LLM workloads by up to <math id="S9.p1.1.m1.1" class="ltx_math_unparsed" alttext="5\times" display="inline"><semantics id="S9.p1.1.m1.1a"><mrow id="S9.p1.1.m1.1b"><mn id="S9.p1.1.m1.1.1">5</mn><mo lspace="0.222em" id="S9.p1.1.m1.1.2">×</mo></mrow><annotation encoding="application/x-tex" id="S9.p1.1.m1.1c">5\times</annotation></semantics></math> with reduced code complexity and improved control.
Through the co-design of the language and runtime, we aim to present a more integrated approach to optimizing LLM programming paradigms.</p>
</div>
</section>
<section id="bib" class="ltx_bibliography">
<h2 class="ltx_title ltx_title_bibliography">References</h2>

<ul class="ltx_biblist">
<li id="bib.bib1" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[1]</span>
<span class="ltx_bibblock">
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al.

</span>
<span class="ltx_bibblock">Flamingo: a visual language model for few-shot learning.

</span>
<span class="ltx_bibblock"><span id="bib.bib1.1.1" class="ltx_text ltx_font_italic">Advances in Neural Information Processing Systems</span>, 35:23716–23736, 2022.

</span>
</li>
<li id="bib.bib2" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[2]</span>
<span class="ltx_bibblock">
Reza Yazdani Aminabadi, Samyam Rajbhandari, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Olatunji Ruwase, Shaden Smith, Minjia Zhang, Jeff Rasley, et al.

</span>
<span class="ltx_bibblock">Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale.

</span>
<span class="ltx_bibblock">In <span id="bib.bib2.1.1" class="ltx_text ltx_font_italic">SC22: International Conference for High Performance Computing, Networking, Storage and Analysis</span>, pages 1–15. IEEE, 2022.

</span>
</li>
<li id="bib.bib3" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[3]</span>
<span class="ltx_bibblock">
Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Michal Podstawski, Hubert Niewiadomski, Piotr Nyczyk, et al.

</span>
<span class="ltx_bibblock">Graph of thoughts: Solving elaborate problems with large language models.

</span>
<span class="ltx_bibblock"><span id="bib.bib3.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2308.09687</span>, 2023.

</span>
</li>
<li id="bib.bib4" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[4]</span>
<span class="ltx_bibblock">
Luca Beurer-Kellner, Marc Fischer, and Martin Vechev.

</span>
<span class="ltx_bibblock">Prompting is programming: A query language for large language models.

</span>
<span class="ltx_bibblock"><span id="bib.bib4.1.1" class="ltx_text ltx_font_italic">Proceedings of the ACM on Programming Languages</span>, 7(PLDI):1946–1969, 2023.

</span>
</li>
<li id="bib.bib5" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[5]</span>
<span class="ltx_bibblock">
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.

</span>
<span class="ltx_bibblock">Language models are few-shot learners.

</span>
<span class="ltx_bibblock"><span id="bib.bib5.1.1" class="ltx_text ltx_font_italic">Advances in neural information processing systems</span>, 33:1877–1901, 2020.

</span>
</li>
<li id="bib.bib6" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[6]</span>
<span class="ltx_bibblock">
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al.

</span>
<span class="ltx_bibblock">Sparks of artificial general intelligence: Early experiments with gpt-4.

</span>
<span class="ltx_bibblock"><span id="bib.bib6.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2303.12712</span>, 2023.

</span>
</li>
<li id="bib.bib7" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[7]</span>
<span class="ltx_bibblock">
Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Haichen Shen, Meghan Cowan, Leyuan Wang, Yuwei Hu, Luis Ceze, et al.

</span>
<span class="ltx_bibblock"><math id="bib.bib7.1.m1.1" class="ltx_Math" alttext="\{" display="inline"><semantics id="bib.bib7.1.m1.1a"><mo stretchy="false" id="bib.bib7.1.m1.1.1" xref="bib.bib7.1.m1.1.1.cmml">{</mo><annotation-xml encoding="MathML-Content" id="bib.bib7.1.m1.1b"><ci id="bib.bib7.1.m1.1.1.cmml" xref="bib.bib7.1.m1.1.1">{</ci></annotation-xml><annotation encoding="application/x-tex" id="bib.bib7.1.m1.1c">\{</annotation></semantics></math>TVM<math id="bib.bib7.2.m2.1" class="ltx_Math" alttext="\}" display="inline"><semantics id="bib.bib7.2.m2.1a"><mo stretchy="false" id="bib.bib7.2.m2.1.1" xref="bib.bib7.2.m2.1.1.cmml">}</mo><annotation-xml encoding="MathML-Content" id="bib.bib7.2.m2.1b"><ci id="bib.bib7.2.m2.1.1.cmml" xref="bib.bib7.2.m2.1.1">}</ci></annotation-xml><annotation encoding="application/x-tex" id="bib.bib7.2.m2.1c">\}</annotation></semantics></math>: An automated <math id="bib.bib7.3.m3.1" class="ltx_Math" alttext="\{" display="inline"><semantics id="bib.bib7.3.m3.1a"><mo stretchy="false" id="bib.bib7.3.m3.1.1" xref="bib.bib7.3.m3.1.1.cmml">{</mo><annotation-xml encoding="MathML-Content" id="bib.bib7.3.m3.1b"><ci id="bib.bib7.3.m3.1.1.cmml" xref="bib.bib7.3.m3.1.1">{</ci></annotation-xml><annotation encoding="application/x-tex" id="bib.bib7.3.m3.1c">\{</annotation></semantics></math>End-to-End<math id="bib.bib7.4.m4.1" class="ltx_Math" alttext="\}" display="inline"><semantics id="bib.bib7.4.m4.1a"><mo stretchy="false" id="bib.bib7.4.m4.1.1" xref="bib.bib7.4.m4.1.1.cmml">}</mo><annotation-xml encoding="MathML-Content" id="bib.bib7.4.m4.1b"><ci id="bib.bib7.4.m4.1.1.cmml" xref="bib.bib7.4.m4.1.1">}</ci></annotation-xml><annotation encoding="application/x-tex" id="bib.bib7.4.m4.1c">\}</annotation></semantics></math> optimizing compiler for deep learning.

</span>
<span class="ltx_bibblock">In <span id="bib.bib7.5.1" class="ltx_text ltx_font_italic">13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18)</span>, pages 578–594, 2018.

</span>
</li>
<li id="bib.bib8" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[8]</span>
<span class="ltx_bibblock">
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al.

</span>
<span class="ltx_bibblock">Palm: Scaling language modeling with pathways.

</span>
<span class="ltx_bibblock"><span id="bib.bib8.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2204.02311</span>, 2022.

</span>
</li>
<li id="bib.bib9" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[9]</span>
<span class="ltx_bibblock">
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al.

</span>
<span class="ltx_bibblock">Training verifiers to solve math word problems.

</span>
<span class="ltx_bibblock"><span id="bib.bib9.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2110.14168</span>, 2021.

</span>
</li>
<li id="bib.bib10" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[10]</span>
<span class="ltx_bibblock">
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré.

</span>
<span class="ltx_bibblock">Flashattention: Fast and memory-efficient exact attention with io-awareness.

</span>
<span class="ltx_bibblock"><span id="bib.bib10.1.1" class="ltx_text ltx_font_italic">Advances in Neural Information Processing Systems</span>, 35:16344–16359, 2022.

</span>
</li>
<li id="bib.bib11" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[11]</span>
<span class="ltx_bibblock">
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer.

</span>
<span class="ltx_bibblock">Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale.

</span>
<span class="ltx_bibblock"><span id="bib.bib11.1.1" class="ltx_text ltx_font_italic">Advances in Neural Information Processing Systems</span>, 35:30318–30332, 2022.

</span>
</li>
<li id="bib.bib12" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[12]</span>
<span class="ltx_bibblock">
Siyuan Feng, Bohan Hou, Hongyi Jin, Wuwei Lin, Junru Shao, Ruihang Lai, Zihao Ye, Lianmin Zheng, Cody Hao Yu, Yong Yu, et al.

</span>
<span class="ltx_bibblock">Tensorir: An abstraction for automatic tensorized program optimization.

</span>
<span class="ltx_bibblock">In <span id="bib.bib12.1.1" class="ltx_text ltx_font_italic">Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2</span>, pages 804–817, 2023.

</span>
</li>
<li id="bib.bib13" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[13]</span>
<span class="ltx_bibblock">
Elias Frantar and Dan Alistarh.

</span>
<span class="ltx_bibblock">Sparsegpt: Massive language models can be accurately pruned in one-shot.

</span>
<span class="ltx_bibblock">In <span id="bib.bib13.1.1" class="ltx_text ltx_font_italic">International Conference on Machine Learning</span>, pages 10323–10337. PMLR, 2023.

</span>
</li>
<li id="bib.bib14" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[14]</span>
<span class="ltx_bibblock">
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh.

</span>
<span class="ltx_bibblock">Optq: Accurate quantization for generative pre-trained transformers.

</span>
<span class="ltx_bibblock">In <span id="bib.bib14.1.1" class="ltx_text ltx_font_italic">The Eleventh International Conference on Learning Representations</span>, 2022.

</span>
</li>
<li id="bib.bib15" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[15]</span>
<span class="ltx_bibblock">
Georgi Gerganov.

</span>
<span class="ltx_bibblock">Llama.cpp.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://github.com/ggerganov/llama.cpp" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/ggerganov/llama.cpp</a>.

</span>
<span class="ltx_bibblock">Accessed: 2023-11.

</span>
</li>
<li id="bib.bib16" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[16]</span>
<span class="ltx_bibblock">
Guidance AI.

</span>
<span class="ltx_bibblock">A guidance language for controlling large language models.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://github.com/guidance-ai/guidance" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/guidance-ai/guidance</a>.

</span>
<span class="ltx_bibblock">Accessed: 2023-11.

</span>
</li>
<li id="bib.bib17" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[17]</span>
<span class="ltx_bibblock">
Mingcong Han, Hanze Zhang, Rong Chen, and Haibo Chen.

</span>
<span class="ltx_bibblock">Microsecond-scale preemption for concurrent <math id="bib.bib17.1.m1.1" class="ltx_Math" alttext="\{" display="inline"><semantics id="bib.bib17.1.m1.1a"><mo stretchy="false" id="bib.bib17.1.m1.1.1" xref="bib.bib17.1.m1.1.1.cmml">{</mo><annotation-xml encoding="MathML-Content" id="bib.bib17.1.m1.1b"><ci id="bib.bib17.1.m1.1.1.cmml" xref="bib.bib17.1.m1.1.1">{</ci></annotation-xml><annotation encoding="application/x-tex" id="bib.bib17.1.m1.1c">\{</annotation></semantics></math>GPU-accelerated<math id="bib.bib17.2.m2.1" class="ltx_Math" alttext="\}" display="inline"><semantics id="bib.bib17.2.m2.1a"><mo stretchy="false" id="bib.bib17.2.m2.1.1" xref="bib.bib17.2.m2.1.1.cmml">}</mo><annotation-xml encoding="MathML-Content" id="bib.bib17.2.m2.1b"><ci id="bib.bib17.2.m2.1.1.cmml" xref="bib.bib17.2.m2.1.1">}</ci></annotation-xml><annotation encoding="application/x-tex" id="bib.bib17.2.m2.1c">\}</annotation></semantics></math><math id="bib.bib17.3.m3.1" class="ltx_Math" alttext="\{" display="inline"><semantics id="bib.bib17.3.m3.1a"><mo stretchy="false" id="bib.bib17.3.m3.1.1" xref="bib.bib17.3.m3.1.1.cmml">{</mo><annotation-xml encoding="MathML-Content" id="bib.bib17.3.m3.1b"><ci id="bib.bib17.3.m3.1.1.cmml" xref="bib.bib17.3.m3.1.1">{</ci></annotation-xml><annotation encoding="application/x-tex" id="bib.bib17.3.m3.1c">\{</annotation></semantics></math>DNN<math id="bib.bib17.4.m4.1" class="ltx_Math" alttext="\}" display="inline"><semantics id="bib.bib17.4.m4.1a"><mo stretchy="false" id="bib.bib17.4.m4.1.1" xref="bib.bib17.4.m4.1.1.cmml">}</mo><annotation-xml encoding="MathML-Content" id="bib.bib17.4.m4.1b"><ci id="bib.bib17.4.m4.1.1.cmml" xref="bib.bib17.4.m4.1.1">}</ci></annotation-xml><annotation encoding="application/x-tex" id="bib.bib17.4.m4.1c">\}</annotation></semantics></math> inferences.

</span>
<span class="ltx_bibblock">In <span id="bib.bib17.5.1" class="ltx_text ltx_font_italic">16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22)</span>, pages 539–558, 2022.

</span>
</li>
<li id="bib.bib18" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[18]</span>
<span class="ltx_bibblock">
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt.

</span>
<span class="ltx_bibblock">Measuring massive multitask language understanding.

</span>
<span class="ltx_bibblock">In <span id="bib.bib18.1.1" class="ltx_text ltx_font_italic">International Conference on Learning Representations</span>, 2020.

</span>
</li>
<li id="bib.bib19" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[19]</span>
<span class="ltx_bibblock">
Hugging Face.

</span>
<span class="ltx_bibblock">Text generation inference.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://github.com/huggingface/text-generation-inference" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/huggingface/text-generation-inference</a>.

</span>
<span class="ltx_bibblock">Accessed: 2023-11.

</span>
</li>
<li id="bib.bib20" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[20]</span>
<span class="ltx_bibblock">
Zhihao Jia, Oded Padon, James Thomas, Todd Warszawski, Matei Zaharia, and Alex Aiken.

</span>
<span class="ltx_bibblock">Taso: optimizing deep learning computation with automatic generation of graph substitutions.

</span>
<span class="ltx_bibblock">In <span id="bib.bib20.1.1" class="ltx_text ltx_font_italic">Proceedings of the 27th ACM Symposium on Operating Systems Principles</span>, pages 47–62, 2019.

</span>
</li>
<li id="bib.bib21" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[21]</span>
<span class="ltx_bibblock">
Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts.

</span>
<span class="ltx_bibblock">Dspy: Compiling declarative language model calls into self-improving pipelines.

</span>
<span class="ltx_bibblock"><span id="bib.bib21.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2310.03714</span>, 2023.

</span>
</li>
<li id="bib.bib22" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[22]</span>
<span class="ltx_bibblock">
Michael Kuchnik, Virginia Smith, and George Amvrosiadis.

</span>
<span class="ltx_bibblock">Validating large language models with relm.

</span>
<span class="ltx_bibblock"><span id="bib.bib22.1.1" class="ltx_text ltx_font_italic">Proceedings of Machine Learning and Systems</span>, 5, 2023.

</span>
</li>
<li id="bib.bib23" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[23]</span>
<span class="ltx_bibblock">
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica.

</span>
<span class="ltx_bibblock">Efficient memory management for large language model serving with pagedattention.

</span>
<span class="ltx_bibblock">In <span id="bib.bib23.1.1" class="ltx_text ltx_font_italic">Proceedings of the 29th Symposium on Operating Systems Principles</span>, pages 611–626, 2023.

</span>
</li>
<li id="bib.bib24" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[24]</span>
<span class="ltx_bibblock">
LangChain AI.

</span>
<span class="ltx_bibblock">Langchain.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://github.com/langchain-ai/langchain" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/langchain-ai/langchain</a>.

</span>
<span class="ltx_bibblock">Accessed: 2023-11.

</span>
</li>
<li id="bib.bib25" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[25]</span>
<span class="ltx_bibblock">
Yaniv Leviathan, Matan Kalman, and Yossi Matias.

</span>
<span class="ltx_bibblock">Fast inference from transformers via speculative decoding.

</span>
<span class="ltx_bibblock">In <span id="bib.bib25.1.1" class="ltx_text ltx_font_italic">International Conference on Machine Learning</span>, pages 19274–19286. PMLR, 2023.

</span>
</li>
<li id="bib.bib26" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[26]</span>
<span class="ltx_bibblock">
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al.

</span>
<span class="ltx_bibblock">Retrieval-augmented generation for knowledge-intensive nlp tasks.

</span>
<span class="ltx_bibblock"><span id="bib.bib26.1.1" class="ltx_text ltx_font_italic">Advances in Neural Information Processing Systems</span>, 33:9459–9474, 2020.

</span>
</li>
<li id="bib.bib27" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[27]</span>
<span class="ltx_bibblock">
Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem.

</span>
<span class="ltx_bibblock">Camel: Communicative agents for "mind" exploration of large language model society.

</span>
<span class="ltx_bibblock">In <span id="bib.bib27.1.1" class="ltx_text ltx_font_italic">Thirty-seventh Conference on Neural Information Processing Systems</span>, 2023.

</span>
</li>
<li id="bib.bib28" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[28]</span>
<span class="ltx_bibblock">
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han.

</span>
<span class="ltx_bibblock">Awq: Activation-aware weight quantization for llm compression and acceleration.

</span>
<span class="ltx_bibblock"><span id="bib.bib28.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2306.00978</span>, 2023.

</span>
</li>
<li id="bib.bib29" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[29]</span>
<span class="ltx_bibblock">
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee.

</span>
<span class="ltx_bibblock">Improved baselines with visual instruction tuning.

</span>
<span class="ltx_bibblock"><span id="bib.bib29.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2310.03744</span>, 2023.

</span>
</li>
<li id="bib.bib30" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[30]</span>
<span class="ltx_bibblock">
Jerry Liu.

</span>
<span class="ltx_bibblock">LlamaIndex, November 2022.

</span>
</li>
<li id="bib.bib31" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[31]</span>
<span class="ltx_bibblock">
Xiaoxia Liu, Jingyi Wang, Jun Sun, Xiaohan Yuan, Guoliang Dong, Peng Di, Wenhai Wang, and Dongxia Wang.

</span>
<span class="ltx_bibblock">Prompting frameworks for large language models: A survey.

</span>
<span class="ltx_bibblock"><span id="bib.bib31.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2311.12785</span>, 2023.

</span>
</li>
<li id="bib.bib32" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[32]</span>
<span class="ltx_bibblock">
Wenyan Lu, Guihai Yan, Jiajun Li, Shijun Gong, Yinhe Han, and Xiaowei Li.

</span>
<span class="ltx_bibblock">Flexflow: A flexible dataflow accelerator architecture for convolutional neural networks.

</span>
<span class="ltx_bibblock">In <span id="bib.bib32.1.1" class="ltx_text ltx_font_italic">2017 IEEE International Symposium on High Performance Computer Architecture (HPCA)</span>, pages 553–564. IEEE, 2017.

</span>
</li>
<li id="bib.bib33" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[33]</span>
<span class="ltx_bibblock">
Lingxiao Ma, Zhiqiang Xie, Zhi Yang, Jilong Xue, Youshan Miao, Wei Cui, Wenxiang Hu, Fan Yang, Lintao Zhang, and Lidong Zhou.

</span>
<span class="ltx_bibblock">Rammer: Enabling holistic deep learning compiler optimizations with <math id="bib.bib33.1.m1.1" class="ltx_Math" alttext="\{" display="inline"><semantics id="bib.bib33.1.m1.1a"><mo stretchy="false" id="bib.bib33.1.m1.1.1" xref="bib.bib33.1.m1.1.1.cmml">{</mo><annotation-xml encoding="MathML-Content" id="bib.bib33.1.m1.1b"><ci id="bib.bib33.1.m1.1.1.cmml" xref="bib.bib33.1.m1.1.1">{</ci></annotation-xml><annotation encoding="application/x-tex" id="bib.bib33.1.m1.1c">\{</annotation></semantics></math>rTasks<math id="bib.bib33.2.m2.1" class="ltx_Math" alttext="\}" display="inline"><semantics id="bib.bib33.2.m2.1a"><mo stretchy="false" id="bib.bib33.2.m2.1.1" xref="bib.bib33.2.m2.1.1.cmml">}</mo><annotation-xml encoding="MathML-Content" id="bib.bib33.2.m2.1b"><ci id="bib.bib33.2.m2.1.1.cmml" xref="bib.bib33.2.m2.1.1">}</ci></annotation-xml><annotation encoding="application/x-tex" id="bib.bib33.2.m2.1c">\}</annotation></semantics></math>.

</span>
<span class="ltx_bibblock">In <span id="bib.bib33.3.1" class="ltx_text ltx_font_italic">14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20)</span>, pages 881–897, 2020.

</span>
</li>
<li id="bib.bib34" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[34]</span>
<span class="ltx_bibblock">
Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Rae Ying Yee Wong, Zhuoming Chen, Daiyaan Arfeen, Reyna Abhyankar, and Zhihao Jia.

</span>
<span class="ltx_bibblock">Specinfer: Accelerating generative llm serving with speculative inference and token tree verification.

</span>
<span class="ltx_bibblock"><span id="bib.bib34.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2305.09781</span>, 2023.

</span>
</li>
<li id="bib.bib35" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[35]</span>
<span class="ltx_bibblock">
Model TC.

</span>
<span class="ltx_bibblock">Lightllm.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://github.com/ModelTC/lightllm" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/ModelTC/lightllm</a>.

</span>
<span class="ltx_bibblock">Accessed: 2023-11.

</span>
</li>
<li id="bib.bib36" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[36]</span>
<span class="ltx_bibblock">
Supun Nakandala, Karla Saur, Gyeong-In Yu, Konstantinos Karanasos, Carlo Curino, Markus Weimer, and Matteo Interlandi.

</span>
<span class="ltx_bibblock">A tensor compiler for unified machine learning prediction serving.

</span>
<span class="ltx_bibblock">In <span id="bib.bib36.1.1" class="ltx_text ltx_font_italic">14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20)</span>, pages 899–917, 2020.

</span>
</li>
<li id="bib.bib37" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[37]</span>
<span class="ltx_bibblock">
Xuefei Ning, Zinan Lin, Zixuan Zhou, Huazhong Yang, and Yu Wang.

</span>
<span class="ltx_bibblock">Skeleton-of-thought: Large language models can do parallel decoding.

</span>
<span class="ltx_bibblock"><span id="bib.bib37.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2307.15337</span>, 2023.

</span>
</li>
<li id="bib.bib38" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[38]</span>
<span class="ltx_bibblock">
NVIDIA.

</span>
<span class="ltx_bibblock">Tensorrt-llm.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://github.com/NVIDIA/TensorRT-LLM" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/NVIDIA/TensorRT-LLM</a>.

</span>
<span class="ltx_bibblock">Accessed: 2023-11.

</span>
</li>
<li id="bib.bib39" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[39]</span>
<span class="ltx_bibblock">
OpenAI.

</span>
<span class="ltx_bibblock">Openai api reference.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://platform.openai.com/docs/api-reference" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://platform.openai.com/docs/api-reference</a>.

</span>
<span class="ltx_bibblock">Accessed: 2023-11.

</span>
</li>
<li id="bib.bib40" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[40]</span>
<span class="ltx_bibblock">
OpenAI.

</span>
<span class="ltx_bibblock">Gpt-4 technical report, 2023.

</span>
</li>
<li id="bib.bib41" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[41]</span>
<span class="ltx_bibblock">
Charles Packer, Vivian Fang, Shishir G. Patil, Kevin Lin, Sarah Wooders, and Joseph E. Gonzalez.

</span>
<span class="ltx_bibblock">Memgpt: Towards llms as operating systems, 2023.

</span>
</li>
<li id="bib.bib42" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[42]</span>
<span class="ltx_bibblock">
Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein.

</span>
<span class="ltx_bibblock">Generative agents: Interactive simulacra of human behavior.

</span>
<span class="ltx_bibblock">In <span id="bib.bib42.1.1" class="ltx_text ltx_font_italic">In the 36th Annual ACM Symposium on User Interface Software and Technology (UIST ’23)</span>, UIST ’23, New York, NY, USA, 2023. Association for Computing Machinery.

</span>
</li>
<li id="bib.bib43" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[43]</span>
<span class="ltx_bibblock">
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.

</span>
<span class="ltx_bibblock">Pytorch: An imperative style, high-performance deep learning library.

</span>
<span class="ltx_bibblock"><span id="bib.bib43.1.1" class="ltx_text ltx_font_italic">Advances in neural information processing systems</span>, 32, 2019.

</span>
</li>
<li id="bib.bib44" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[44]</span>
<span class="ltx_bibblock">
Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez.

</span>
<span class="ltx_bibblock">Gorilla: Large language model connected with massive apis.

</span>
<span class="ltx_bibblock"><span id="bib.bib44.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2305.15334</span>, 2023.

</span>
</li>
<li id="bib.bib45" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[45]</span>
<span class="ltx_bibblock">
Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean.

</span>
<span class="ltx_bibblock">Efficiently scaling transformer inference.

</span>
<span class="ltx_bibblock"><span id="bib.bib45.1.1" class="ltx_text ltx_font_italic">Proceedings of Machine Learning and Systems</span>, 5, 2023.

</span>
</li>
<li id="bib.bib46" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[46]</span>
<span class="ltx_bibblock">
Nir Ratner, Yoav Levine, Yonatan Belinkov, Ori Ram, Inbal Magar, Omri Abend, Ehud Karpas, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham.

</span>
<span class="ltx_bibblock">Parallel context windows for large language models.

</span>
<span class="ltx_bibblock">In <span id="bib.bib46.1.1" class="ltx_text ltx_font_italic">Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)</span>, pages 6383–6402, 2023.

</span>
</li>
<li id="bib.bib47" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[47]</span>
<span class="ltx_bibblock">
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al.

</span>
<span class="ltx_bibblock">Code llama: Open foundation models for code.

</span>
<span class="ltx_bibblock"><span id="bib.bib47.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2308.12950</span>, 2023.

</span>
</li>
<li id="bib.bib48" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[48]</span>
<span class="ltx_bibblock">
Swarnadeep Saha, Omer Levy, Asli Celikyilmaz, Mohit Bansal, Jason Weston, and Xian Li.

</span>
<span class="ltx_bibblock">Branch-solve-merge improves large language model evaluation and generation.

</span>
<span class="ltx_bibblock"><span id="bib.bib48.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2310.15123</span>, 2023.

</span>
</li>
<li id="bib.bib49" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[49]</span>
<span class="ltx_bibblock">
Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom.

</span>
<span class="ltx_bibblock">Toolformer: Language models can teach themselves to use tools.

</span>
<span class="ltx_bibblock"><span id="bib.bib49.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2302.04761</span>, 2023.

</span>
</li>
<li id="bib.bib50" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[50]</span>
<span class="ltx_bibblock">
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang.

</span>
<span class="ltx_bibblock">Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface.

</span>
<span class="ltx_bibblock"><span id="bib.bib50.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2303.17580</span>, 2023.

</span>
</li>
<li id="bib.bib51" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[51]</span>
<span class="ltx_bibblock">
Ying Sheng, Shiyi Cao, Dacheng Li, Coleman Hooper, Nicholas Lee, Shuo Yang, Christopher Chou, Banghua Zhu, Lianmin Zheng, Kurt Keutzer, et al.

</span>
<span class="ltx_bibblock">S-lora: Serving thousands of concurrent lora adapters.

</span>
<span class="ltx_bibblock"><span id="bib.bib51.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2311.03285</span>, 2023.

</span>
</li>
<li id="bib.bib52" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[52]</span>
<span class="ltx_bibblock">
Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang.

</span>
<span class="ltx_bibblock">Flexgen: high-throughput generative inference of large language models with a single gpu.

</span>
<span class="ltx_bibblock">In <span id="bib.bib52.1.1" class="ltx_text ltx_font_italic">International Conference on Machine Learning</span>, pages 31094–31116. PMLR, 2023.

</span>
</li>
<li id="bib.bib53" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[53]</span>
<span class="ltx_bibblock">
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik R Narasimhan, and Shunyu Yao.

</span>
<span class="ltx_bibblock">Reflexion: Language agents with verbal reinforcement learning.

</span>
<span class="ltx_bibblock">In <span id="bib.bib53.1.1" class="ltx_text ltx_font_italic">Thirty-seventh Conference on Neural Information Processing Systems</span>, 2023.

</span>
</li>
<li id="bib.bib54" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[54]</span>
<span class="ltx_bibblock">
Significant Gravitas.

</span>
<span class="ltx_bibblock">AutoGPT.

</span>
</li>
<li id="bib.bib55" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[55]</span>
<span class="ltx_bibblock">
Xiaoniu Song, Yiwen Zhang, Rong Chen, and Haibo Chen.

</span>
<span class="ltx_bibblock">Ugache: A unified gpu cache for embedding-based deep learning.

</span>
<span class="ltx_bibblock">In <span id="bib.bib55.1.1" class="ltx_text ltx_font_italic">Proceedings of the 29th Symposium on Operating Systems Principles</span>, pages 627–641, 2023.

</span>
</li>
<li id="bib.bib56" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[56]</span>
<span class="ltx_bibblock">
Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit.

</span>
<span class="ltx_bibblock">Blockwise parallel decoding for deep autoregressive models.

</span>
<span class="ltx_bibblock"><span id="bib.bib56.1.1" class="ltx_text ltx_font_italic">Advances in Neural Information Processing Systems</span>, 31, 2018.

</span>
</li>
<li id="bib.bib57" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[57]</span>
<span class="ltx_bibblock">
Philippe Tillet, Hsiang-Tsung Kung, and David Cox.

</span>
<span class="ltx_bibblock">Triton: an intermediate language and compiler for tiled neural network computations.

</span>
<span class="ltx_bibblock">In <span id="bib.bib57.1.1" class="ltx_text ltx_font_italic">Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages</span>, pages 10–19, 2019.

</span>
</li>
<li id="bib.bib58" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[58]</span>
<span class="ltx_bibblock">
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.

</span>
<span class="ltx_bibblock">Llama 2: Open foundation and fine-tuned chat models.

</span>
<span class="ltx_bibblock"><span id="bib.bib58.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2307.09288</span>, 2023.

</span>
</li>
<li id="bib.bib59" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[59]</span>
<span class="ltx_bibblock">
Colin Unger, Zhihao Jia, Wei Wu, Sina Lin, Mandeep Baines, Carlos Efrain Quintero Narvaez, Vinay Ramakrishnaiah, Nirmal Prajapati, Pat McCormick, Jamaludin Mohd-Yusof, et al.

</span>
<span class="ltx_bibblock">Unity: Accelerating <math id="bib.bib59.1.m1.1" class="ltx_Math" alttext="\{" display="inline"><semantics id="bib.bib59.1.m1.1a"><mo stretchy="false" id="bib.bib59.1.m1.1.1" xref="bib.bib59.1.m1.1.1.cmml">{</mo><annotation-xml encoding="MathML-Content" id="bib.bib59.1.m1.1b"><ci id="bib.bib59.1.m1.1.1.cmml" xref="bib.bib59.1.m1.1.1">{</ci></annotation-xml><annotation encoding="application/x-tex" id="bib.bib59.1.m1.1c">\{</annotation></semantics></math>DNN<math id="bib.bib59.2.m2.1" class="ltx_Math" alttext="\}" display="inline"><semantics id="bib.bib59.2.m2.1a"><mo stretchy="false" id="bib.bib59.2.m2.1.1" xref="bib.bib59.2.m2.1.1.cmml">}</mo><annotation-xml encoding="MathML-Content" id="bib.bib59.2.m2.1b"><ci id="bib.bib59.2.m2.1.1.cmml" xref="bib.bib59.2.m2.1.1">}</ci></annotation-xml><annotation encoding="application/x-tex" id="bib.bib59.2.m2.1c">\}</annotation></semantics></math> training through joint optimization of algebraic transformations and parallelization.

</span>
<span class="ltx_bibblock">In <span id="bib.bib59.3.1" class="ltx_text ltx_font_italic">16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22)</span>, pages 267–284, 2022.

</span>
</li>
<li id="bib.bib60" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[60]</span>
<span class="ltx_bibblock">
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.

</span>
<span class="ltx_bibblock">Attention is all you need.

</span>
<span class="ltx_bibblock"><span id="bib.bib60.1.1" class="ltx_text ltx_font_italic">Advances in neural information processing systems</span>, 30, 2017.

</span>
</li>
<li id="bib.bib61" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[61]</span>
<span class="ltx_bibblock">
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar.

</span>
<span class="ltx_bibblock">Voyager: An open-ended embodied agent with large language models.

</span>
<span class="ltx_bibblock"><span id="bib.bib61.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv: Arxiv-2305.16291</span>, 2023.

</span>
</li>
<li id="bib.bib62" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[62]</span>
<span class="ltx_bibblock">
Haojie Wang, Jidong Zhai, Mingyu Gao, Zixuan Ma, Shizhi Tang, Liyan Zheng, Yuanzhi Li, Kaiyuan Rong, Yuanyong Chen, and Zhihao Jia.

</span>
<span class="ltx_bibblock"><math id="bib.bib62.1.m1.1" class="ltx_Math" alttext="\{" display="inline"><semantics id="bib.bib62.1.m1.1a"><mo stretchy="false" id="bib.bib62.1.m1.1.1" xref="bib.bib62.1.m1.1.1.cmml">{</mo><annotation-xml encoding="MathML-Content" id="bib.bib62.1.m1.1b"><ci id="bib.bib62.1.m1.1.1.cmml" xref="bib.bib62.1.m1.1.1">{</ci></annotation-xml><annotation encoding="application/x-tex" id="bib.bib62.1.m1.1c">\{</annotation></semantics></math>PET<math id="bib.bib62.2.m2.1" class="ltx_Math" alttext="\}" display="inline"><semantics id="bib.bib62.2.m2.1a"><mo stretchy="false" id="bib.bib62.2.m2.1.1" xref="bib.bib62.2.m2.1.1.cmml">}</mo><annotation-xml encoding="MathML-Content" id="bib.bib62.2.m2.1b"><ci id="bib.bib62.2.m2.1.1.cmml" xref="bib.bib62.2.m2.1.1">}</ci></annotation-xml><annotation encoding="application/x-tex" id="bib.bib62.2.m2.1c">\}</annotation></semantics></math>: Optimizing tensor programs with partially equivalent transformations and automated corrections.

</span>
<span class="ltx_bibblock">In <span id="bib.bib62.3.1" class="ltx_text ltx_font_italic">15th USENIX Symposium on Operating Systems Design and Implementation (OSDI 21)</span>, pages 37–54, 2021.

</span>
</li>
<li id="bib.bib63" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[63]</span>
<span class="ltx_bibblock">
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou.

</span>
<span class="ltx_bibblock">Self-consistency improves chain of thought reasoning in language models.

</span>
<span class="ltx_bibblock">In <span id="bib.bib63.1.1" class="ltx_text ltx_font_italic">The Eleventh International Conference on Learning Representations</span>, 2022.

</span>
</li>
<li id="bib.bib64" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[64]</span>
<span class="ltx_bibblock">
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al.

</span>
<span class="ltx_bibblock">Chain-of-thought prompting elicits reasoning in large language models.

</span>
<span class="ltx_bibblock"><span id="bib.bib64.1.1" class="ltx_text ltx_font_italic">Advances in Neural Information Processing Systems</span>, 35:24824–24837, 2022.

</span>
</li>
<li id="bib.bib65" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[65]</span>
<span class="ltx_bibblock">
Brandon T Willard and Rémi Louf.

</span>
<span class="ltx_bibblock">Efficient guided generation for large language models.

</span>
<span class="ltx_bibblock"><span id="bib.bib65.1.1" class="ltx_text ltx_font_italic">arXiv e-prints</span>, pages arXiv–2307, 2023.

</span>
</li>
<li id="bib.bib66" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[66]</span>
<span class="ltx_bibblock">
Bingyang Wu, Yinmin Zhong, Zili Zhang, Gang Huang, Xuanzhe Liu, and Xin Jin.

</span>
<span class="ltx_bibblock">Fast distributed inference serving for large language models.

</span>
<span class="ltx_bibblock"><span id="bib.bib66.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2305.05920</span>, 2023.

</span>
</li>
<li id="bib.bib67" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[67]</span>
<span class="ltx_bibblock">
Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W White, Doug Burger, and Chi Wang.

</span>
<span class="ltx_bibblock">Autogen: Enabling next-gen llm applications via multi-agent conversation framework.

</span>
<span class="ltx_bibblock">2023.

</span>
</li>
<li id="bib.bib68" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[68]</span>
<span class="ltx_bibblock">
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han.

</span>
<span class="ltx_bibblock">Smoothquant: Accurate and efficient post-training quantization for large language models.

</span>
<span class="ltx_bibblock">In <span id="bib.bib68.1.1" class="ltx_text ltx_font_italic">International Conference on Machine Learning</span>, pages 38087–38099. PMLR, 2023.

</span>
</li>
<li id="bib.bib69" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[69]</span>
<span class="ltx_bibblock">
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning.

</span>
<span class="ltx_bibblock">HotpotQA: A dataset for diverse, explainable multi-hop question answering.

</span>
<span class="ltx_bibblock">In <span id="bib.bib69.1.1" class="ltx_text ltx_font_italic">Conference on Empirical Methods in Natural Language Processing (EMNLP)</span>, 2018.

</span>
</li>
<li id="bib.bib70" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[70]</span>
<span class="ltx_bibblock">
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan.

</span>
<span class="ltx_bibblock">Tree of thoughts: Deliberate problem solving with large language models.

</span>
<span class="ltx_bibblock"><span id="bib.bib70.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2305.10601</span>, 2023.

</span>
</li>
<li id="bib.bib71" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[71]</span>
<span class="ltx_bibblock">
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao.

</span>
<span class="ltx_bibblock">React: Synergizing reasoning and acting in language models.

</span>
<span class="ltx_bibblock">In <span id="bib.bib71.1.1" class="ltx_text ltx_font_italic">The Eleventh International Conference on Learning Representations</span>, 2022.

</span>
</li>
<li id="bib.bib72" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[72]</span>
<span class="ltx_bibblock">
Zihao Ye, Ruihang Lai, Junru Shao, Tianqi Chen, and Luis Ceze.

</span>
<span class="ltx_bibblock">Sparsetir: Composable abstractions for sparse compilation in deep learning.

</span>
<span class="ltx_bibblock">In <span id="bib.bib72.1.1" class="ltx_text ltx_font_italic">Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3</span>, pages 660–678, 2023.

</span>
</li>
<li id="bib.bib73" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[73]</span>
<span class="ltx_bibblock">
Ori Yoran, Tomer Wolfson, Ben Bogin, Uri Katz, Daniel Deutch, and Jonathan Berant.

</span>
<span class="ltx_bibblock">Answering questions by meta-reasoning over multiple chains of thought.

</span>
<span class="ltx_bibblock"><span id="bib.bib73.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2304.13007</span>, 2023.

</span>
</li>
<li id="bib.bib74" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[74]</span>
<span class="ltx_bibblock">
Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun.

</span>
<span class="ltx_bibblock">Orca: A distributed serving system for <math id="bib.bib74.1.m1.1" class="ltx_Math" alttext="\{" display="inline"><semantics id="bib.bib74.1.m1.1a"><mo stretchy="false" id="bib.bib74.1.m1.1.1" xref="bib.bib74.1.m1.1.1.cmml">{</mo><annotation-xml encoding="MathML-Content" id="bib.bib74.1.m1.1b"><ci id="bib.bib74.1.m1.1.1.cmml" xref="bib.bib74.1.m1.1.1">{</ci></annotation-xml><annotation encoding="application/x-tex" id="bib.bib74.1.m1.1c">\{</annotation></semantics></math>Transformer-Based<math id="bib.bib74.2.m2.1" class="ltx_Math" alttext="\}" display="inline"><semantics id="bib.bib74.2.m2.1a"><mo stretchy="false" id="bib.bib74.2.m2.1.1" xref="bib.bib74.2.m2.1.1.cmml">}</mo><annotation-xml encoding="MathML-Content" id="bib.bib74.2.m2.1b"><ci id="bib.bib74.2.m2.1.1.cmml" xref="bib.bib74.2.m2.1.1">}</ci></annotation-xml><annotation encoding="application/x-tex" id="bib.bib74.2.m2.1c">\}</annotation></semantics></math> generative models.

</span>
<span class="ltx_bibblock">In <span id="bib.bib74.3.1" class="ltx_text ltx_font_italic">16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22)</span>, pages 521–538, 2022.

</span>
</li>
<li id="bib.bib75" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[75]</span>
<span class="ltx_bibblock">
Yuan Yu, Martín Abadi, Paul Barham, Eugene Brevdo, Mike Burrows, Andy Davis, Jeff Dean, Sanjay Ghemawat, Tim Harley, Peter Hawkins, et al.

</span>
<span class="ltx_bibblock">Dynamic control flow in large-scale machine learning.

</span>
<span class="ltx_bibblock">In <span id="bib.bib75.1.1" class="ltx_text ltx_font_italic">Proceedings of the Thirteenth EuroSys Conference</span>, pages 1–15, 2018.

</span>
</li>
<li id="bib.bib76" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[76]</span>
<span class="ltx_bibblock">
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi.

</span>
<span class="ltx_bibblock">Hellaswag: Can a machine really finish your sentence?

</span>
<span class="ltx_bibblock">In <span id="bib.bib76.1.1" class="ltx_text ltx_font_italic">Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics</span>, pages 4791–4800, 2019.

</span>
</li>
<li id="bib.bib77" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[77]</span>
<span class="ltx_bibblock">
Lianmin Zheng, Chengfan Jia, Minmin Sun, Zhao Wu, Cody Hao Yu, Ameer Haj-Ali, Yida Wang, Jun Yang, Danyang Zhuo, Koushik Sen, et al.

</span>
<span class="ltx_bibblock">Ansor: Generating <math id="bib.bib77.1.m1.1" class="ltx_Math" alttext="\{" display="inline"><semantics id="bib.bib77.1.m1.1a"><mo stretchy="false" id="bib.bib77.1.m1.1.1" xref="bib.bib77.1.m1.1.1.cmml">{</mo><annotation-xml encoding="MathML-Content" id="bib.bib77.1.m1.1b"><ci id="bib.bib77.1.m1.1.1.cmml" xref="bib.bib77.1.m1.1.1">{</ci></annotation-xml><annotation encoding="application/x-tex" id="bib.bib77.1.m1.1c">\{</annotation></semantics></math>High-Performance<math id="bib.bib77.2.m2.1" class="ltx_Math" alttext="\}" display="inline"><semantics id="bib.bib77.2.m2.1a"><mo stretchy="false" id="bib.bib77.2.m2.1.1" xref="bib.bib77.2.m2.1.1.cmml">}</mo><annotation-xml encoding="MathML-Content" id="bib.bib77.2.m2.1b"><ci id="bib.bib77.2.m2.1.1.cmml" xref="bib.bib77.2.m2.1.1">}</ci></annotation-xml><annotation encoding="application/x-tex" id="bib.bib77.2.m2.1c">\}</annotation></semantics></math> tensor programs for deep learning.

</span>
<span class="ltx_bibblock">In <span id="bib.bib77.3.1" class="ltx_text ltx_font_italic">14th USENIX symposium on operating systems design and implementation (OSDI 20)</span>, pages 863–879, 2020.

</span>
</li>
<li id="bib.bib78" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[78]</span>
<span class="ltx_bibblock">
Lianmin Zheng, Zhuohan Li, Hao Zhang, Yonghao Zhuang, Zhifeng Chen, Yanping Huang, Yida Wang, Yuanzhong Xu, Danyang Zhuo, Eric P Xing, et al.

</span>
<span class="ltx_bibblock">Alpa: Automating inter-and <math id="bib.bib78.1.m1.1" class="ltx_Math" alttext="\{" display="inline"><semantics id="bib.bib78.1.m1.1a"><mo stretchy="false" id="bib.bib78.1.m1.1.1" xref="bib.bib78.1.m1.1.1.cmml">{</mo><annotation-xml encoding="MathML-Content" id="bib.bib78.1.m1.1b"><ci id="bib.bib78.1.m1.1.1.cmml" xref="bib.bib78.1.m1.1.1">{</ci></annotation-xml><annotation encoding="application/x-tex" id="bib.bib78.1.m1.1c">\{</annotation></semantics></math>Intra-Operator<math id="bib.bib78.2.m2.1" class="ltx_Math" alttext="\}" display="inline"><semantics id="bib.bib78.2.m2.1a"><mo stretchy="false" id="bib.bib78.2.m2.1.1" xref="bib.bib78.2.m2.1.1.cmml">}</mo><annotation-xml encoding="MathML-Content" id="bib.bib78.2.m2.1b"><ci id="bib.bib78.2.m2.1.1.cmml" xref="bib.bib78.2.m2.1.1">}</ci></annotation-xml><annotation encoding="application/x-tex" id="bib.bib78.2.m2.1c">\}</annotation></semantics></math> parallelism for distributed deep learning.

</span>
<span class="ltx_bibblock">In <span id="bib.bib78.3.1" class="ltx_text ltx_font_italic">16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22)</span>, pages 559–578, 2022.

</span>
</li>
</ul>
</section>
<div class="ltx_pagination ltx_role_newpage"></div>
</article>
</div>
<div class="ar5iv-footer"><a href="/html/2312.07103" class="ar5iv-nav-button ar5iv-nav-button-prev">◄</a>
    <a class="ar5iv-home-button" href="/"><img height="40" alt="ar5iv homepage" src="/assets/ar5iv.png"></a>
    <a href="/feeling_lucky" class="ar5iv-text-button">Feeling<br>lucky?</a>
    <a href="/log/2312.07104" class="ar5iv-text-button ar5iv-severity-warning">Conversion<br>report</a>
    <a class="ar5iv-text-button" target="_blank" href="https://github.com/dginev/ar5iv/issues/new?template=improve-article--arxiv-id-.md&title=Improve+article+2312.07104">Report<br>an issue</a>
    <a href="https://arxiv.org/abs/2312.07104" class="ar5iv-text-button arxiv-ui-theme">View&nbsp;original<br>on&nbsp;arXiv</a><a href="/html/2312.07105" class="ar5iv-nav-button ar5iv-nav-button-next">►</a>
</div><footer class="ltx_page_footer">
<a class="ar5iv-toggle-color-scheme" href="javascript:toggleColorScheme()" title="Toggle ar5iv color scheme"><span class="color-scheme-icon"></span></a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/license" target="_blank">Copyright</a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/policies/privacy_policy" target="_blank">Privacy Policy</a>

<div class="ltx_page_logo">Generated  on Tue Feb 27 11:25:01 2024 by <a target="_blank" href="http://dlmf.nist.gov/LaTeXML/" class="ltx_LaTeXML_logo"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg==" alt="Mascot Sammy"></a>
</div></footer>
</div>

    <script>
      var canMathML = typeof(MathMLElement) == "function";
      if (!canMathML) {
        var body = document.querySelector("body");
        body.firstElementChild.setAttribute('style', 'opacity: 0;');
        var loading = document.createElement("div");
        loading.setAttribute("id", "mathjax-loading-spinner");
        var message = document.createElement("div");
        message.setAttribute("id", "mathjax-loading-message");
        message.innerText = "Typesetting Equations...";
        body.prepend(loading);
        body.prepend(message);

        var el = document.createElement("script");
        el.src = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js";
        document.querySelector("head").appendChild(el);

        window.MathJax = {
          startup: {
            pageReady: () => {
              return MathJax.startup.defaultPageReady().then(() => {
                body.removeChild(loading);
                body.removeChild(message);
                body.firstElementChild.removeAttribute('style');
              }); } } };
      }
    </script>
    <script>
    // Auxiliary function, building the preview feature when
    // an inline citation is clicked
    function clicked_cite(e) {
      e.preventDefault();
      let cite = this.closest('.ltx_cite');
      let next = cite.nextSibling;
      if (next && next.nodeType == Node.ELEMENT_NODE && next.getAttribute('class') == "ar5iv-bibitem-preview") {
        next.remove();
        return; }
      // Before adding a preview modal,
      // cleanup older previews, in case they're still open
      document.querySelectorAll('span.ar5iv-bibitem-preview').forEach(function(node) {
        node.remove();
      })

      // Create the preview
      preview = document.createElement('span');
      preview.setAttribute('class','ar5iv-bibitem-preview');
      let target = document.getElementById(this.getAttribute('href').slice(1));
      target.childNodes.forEach(function (child) {
        preview.append(child.cloneNode(true));
      });
      let close_x = document.createElement('button');
      close_x.setAttribute("aria-label","Close modal for bibliography item preview");
      close_x.textContent = "×";
      close_x.setAttribute('class', 'ar5iv-button-close-preview');
      close_x.setAttribute('onclick','this.parentNode.remove()');
      preview.append(close_x);
      preview.querySelectorAll('.ltx_tag_bibitem').forEach(function(node) {
        node.remove();
      });
      cite.parentNode.insertBefore(preview, cite.nextSibling);
      return;
    }
    // Global Document initialization:
    // - assign the preview feature to all inline citation links
    document.querySelectorAll(".ltx_cite .ltx_ref").forEach(function (link) {
      link.addEventListener("click", clicked_cite);
    });
    </script>
    </body>
</html>

2312.07104
Large language models (LLMs)  are increasingly used for complex tasks requiring multiple chained generation calls, advanced prompting techniques, control flow, and interaction with external environments. However, efficient systems for programming and executing these applications are lacking.
To bridge this gap, we introduce SGLang, aStructuredGenerationLanguage for LLMs. SGLang is designed for the efficient programming of LLMs and incorporates primitives for common LLM programming patterns. We have implemented SGLang as a domain-specific language embedded in Python, and we developed an interpreter, a compiler, and a high-performance runtime for SGLang.
These components work together to enable optimizations such as parallelism, batching, caching, sharing, and other compilation techniques. Additionally, we propose RadixAttention, a novel technique that maintains a Least Recently Used (LRU)  cache of the Key-Value (KV)  cache for all requests in a radix tree, enabling automatic KV cache reuse across multiple generation calls at runtime. SGLang simplifies the writing of LLM programs and boosts execution efficiency.
Our experiments demonstrate that SGLang can speed up common LLM tasks by up to, while reducing code complexity and enhancing control.
abstract
Efficiently Programming Large Language Models using SGLang
Abstract
2312.07104
1IntroductionRecent increases in the capabilities of LLMs have broadened their utility, enabling them to tackle a wider range of general tasks and act as autonomous agents.
In such applications, LLMs engage inmulti-roundplanning, reasoning, and interaction with external environments.
This is accomplished through tool usage, multiple input modalities, and a wide-range of prompting techniques, like few-shot learning,
self-consistency, skeleton-of-thought, and tree-of-thought.
All of these new use cases require multiple, often dependent, LLM generation calls, and critically, the control flow of these programs may depend on the output of LLMs.The emergence of these complex patterns signifies a shift in our interaction with LLMs, moving from simple turn-based chatting to a more sophisticated form of programmatic usage of LLMs (seeSection 2.2) .Figure 1shows an example of such an LLM program, where multiple generation calls are used to accomplish a task. The example uses SGLang, a new domain-specific language we introduce in this paper and explain inSection 4.Figure 1:The implementation of a multi-dimensional essay judge in SGLang utilizes the branch-solve-merge prompting technique.
This function uses LLMs to evaluate the quality of an essay from multiple dimensions, merges the judgments, generates a summary, and assigns a final grade.
The highlighted regions illustrate the use of SGLang APIs.
(1) forkcreates multiple parallel copies of a prompt.
(2) geninvokes an LLM generation and stores the result in a variable.
(3) [variable_name]retrieves the result of the generation.
(4) choicesimposes constraints on the generation.
(5) runexecutes a SGLang function with its arguments.To align with emerging trends, the community has seen the development and rising popularity of new interfaces such as LangChain, LMQL, Guidance, and DSPy.
These tools help users manage prompt templates and program pipelines with ease. However, their primary focus on front-end design often results in compromised runtime performance.Conversely, while back-end inference engines like NVIDIA TensorRT-LLM, Hugging Face TGI, Orca, and vLLMcontinue to reduce latency and improve throughput, their design largely remains anchored in traditional single-generation call interfaces. They typically support APIs similar to the OpenAI Completion API, where a user prompt is processed and a response is returned in a stateless manner.While both front-end and back-end technologies are advancing rapidly, they are evolving along separate trajectories. The front end largely remains oblivious to the LLM inference processes, while the back end often lacks awareness of the application’s structure and is unable to optimize across multiple calls. This disjointed development leads to numerous missed opportunities for performance optimization, such as KV cache sharing and parallelism, as discussed inSection 2.3.In this paper, we present a holistic approach for an efficient LLM programming system by co-designing both the front-end language and the back-end runtime.
The process involves three key challenges: (i)  creating a flexible, domain-specific language with primitives suitable for LLM programming and ecosystem integration; (ii) 
ensuring efficient program execution via automatic parallelism, batching, caching, and sharing across multiple calls and multiple programs; and (iii) 
enhancing program performance through compiler optimizations like code movement and prefetching annotations.On the front end, we introduce SGLang, a domain-specific language embedded in Python. It provides a set of primitives specifically designed to facilitate programming LLMs. These primitives enable the manipulation of prompts and generations (e.g.,extend,gen,select)  and the control of parallelism (e.g.,fork,join) . Furthermore, SGLang is compatible with Python’s control flow and libraries.
Leveraging these primitives along with native Python syntax, users can
intuitively develop sophisticated prompting strategies and agents, while obviating the need for manual optimization.
Moreover, the unified language enhances portability, enabling users to transition easily between different open-source and proprietary models, such as those from OpenAI and Anthropic.We develop both an interpreter and a compiler for SGLang. The interpreter is responsible for managing the prompt state and executing SGLang programs.
Specifically, it treats a prompt as a stream and submits primitive operations to this stream for asynchronous execution, ensuring proper control over synchronization and intra-program parallelism.
Additionally, we can trace a SGLang program and utilize the compiler to compile it into a computational graph.
This approach opens opportunities for more aggressive optimizations. We explore several classical compiler optimizations, e.g., code movement, in the context of LLM programming.On the back end, we propose a novel technique called RadixAttention to automatically reuse the KV cache across multiple generation calls. In existing systems, the KV cache of a request is discarded after processing is completed, which prevents KV cache from being reused across multiple generation calls.
Instead, our system maintains an LRU cache of the KV cache for all requests within a radix tree. This approach allows the runtime to efficiently achieve automatic sharing for various reuse patterns.
Additionally, to increase the cache hit rate, we utilize a cache-aware scheduling policy.In summary, we make the following contributions:•We present SGLang, a domain-specific language with primitives for LLM programming, supporting parallelism, control flow, nested calls, and external calls.•We develop an interpreter capable of executing SGLang programs in tandem with the Python interpreter and managing prompt states as asynchronous streams.•We build a compiler capable of compiling programs into computational graphs and explore classical compiler techniques in the context of LLM programming.•We propose RadixAttention, a novel technique for automatic KV cache reuse across multiple generation calls at runtime.•Using SGLang, we implement various LLM applications, including agent control, logical reasoning, content generation, benchmarking, and long document processing.
Our experiments demonstrate that SGLang can accelerate common LLM tasks by up to, while reducing code complexity and enhancing control.
introduction
Efficiently Programming Large Language Models using SGLang
1Introduction
2312.07104
2Background2.1The Inference Process of LLMsMost LLMs in use today, such as GPT-3, PaLM, and LLaMA, are based on the autoregressive Transformer architecture.
The models predict the probability of the next token in a sequence based on the preceding tokens.
During inference, the model first processes a sequence of input tokens through a forward pass.
It then sequentially decodes output tokens, with each token depending on prior tokens.
We refer to the process of taking a sequence of input tokens and generating a sequence of output tokens as a single generation call.
Throughout this process, each token generates some intermediate tensors, which are used for decoding further tokens.
These intermediate tensors, known as the "KV Cache", are named for the key-value pairs in the self-attention mechanism.
An observation that will be important when discussing optimizations later is that the computation of the KV cache only depends on all previous tokens, so different sequences with the same prefix can reuse the KV cache of the prefix tokens and avoid redundant computation.2.2The Paradigm of Programming LLMsAs LLMs become more powerful, they are increasingly employed as versatile task solvers in complex systems. In these contexts, it is imperative that LLMs operate as dependable and intelligent components.
This evolves the use of LLMs from single, straightforward generation calls to multiple, interdependent generation calls interleaved with control flow, leading to the emergence of the "Programming LLMs" paradigm.
Programming LLMs refers to the development of computer programs that control the generation processes of LLMs. This concept can be exemplified in several ways.First, advanced prompting techniques are examples of programming LLMs. An exemplary case is meta-reasoning. To solve a problem, the LLM initially generates multiple solutions through chain-of-thought prompting. It then performs meta-reasoning across these solutions to derive the final resolution. Another example is skeleton-of-thought. To answer a question, the LLM first generates a skeleton of the answer. It then issues multiple parallel calls to elaborate the skeleton into a detailed explanation. A more complex case is the tree-of-thought approach, which integrates LLM calls within tree search algorithms (e.g., breadth-first search) .Second, in LLM-based agents, the programmatic use of LLMs is crucial. Take the ReAct agent, for example. The behavior of the agent is controlled by loops and conditional statements. The agent performs one action per iteration. In each iteration, the agent chooses the action type (search or finish) , generates action parameters, and processes observations from the external environment.Third, simpler prompting forms also fall under the scope of programming LLMs. Techniques like few-shot prompting, chain-of-thought prompting, and retrieval-augmented generationare basic elements in programming LLMs.Figure 2:KV cache sharing examples. Blue boxes represent shareable prompt parts, green boxes indicate non-shareable parts, and yellow boxes mark non-shareable model outputs. Shareable elements include few-shot learning examples, questions in self-consistency, chat history in multi-turn chats, and search history in tree-of-thought.LanguageSyntaxPrimitivesRuntime BackendsExecution ModesLMQLCustomextend, gen, selectHugging Face Transformers, llama.cpp, OpenAICompilerGuidancePythonextend, gen, selectHugging Face Transformers, llama.cpp, OpenAI, VertexAIInterpreterSGLangPythonextend, gen, select, fork, joinSGVM, OpenAI, Anthropic, Hugging Face TGI∗, llama.cpp∗Interpreter and compilerTable 1:Comparison among LMQL, Guidance, and SGLang. The backends marked with∗are experimental.2.3Challenges and OpportunitiesDesigning programming systems for LLMs presents unique challenges and opportunities.
A good system should help a programmer be productive, offering an intuitive and expressive interface for common programming tasks, including LLM state management, decoding process control, and smooth integration with external tools.
Furthermore, it is essential to ensure performance by optimizing for low latency and high throughput execution.
In particular, the rich structure of LLM programs and the unique properties of LLM inference provide plenty of opportunities for runtime optimizations. These include:•Caching:Often in LLM programs, multiple text segments and generation calls are appended to a single prompt. Caching the computed KV cache for previous tokens across multiple chained calls can reduce redundant computation. This optimization, however, is neither free nor trivial, as it requires additional storage and more complex memory management.•Batching:The decoding process in LLMs is primarily memory-bound, meaning increasing the batch size can significantly enhance throughput. Running a large number of LLM programs in a batch, and utilizing the contiguous batching techniquecan be beneficial.•Sharing:It is common in LLM programs to generate multiple outputs from a single prompt or to fork a new prompt from the current state. Basic prefix sharing has been investigated in vLLM. More advanced sharing patterns like irregular tree-structured sharing can also be employed.Figure 2shows four typical patterns of KV cache sharing across multiple calls; none of the existing systems can automatically handle all of them.•Parallelism:An LLM program often includes multiple generation calls. By creating a dependency graph for these calls, we can identify independent calls and execute them in parallel. This creates intra-program parallelism, which adds additional parallelism to the straightforward inter-program parallelism.
For example, inFigure 1, the generation of multiple judgments can be parallelized.•Compilation:Once a full program is available, it can be compiled into an optimized intermediate representation for more efficient execution. More aggressive optimizations can also be implemented, such as automatically adjusting the original prompts based on user-provided test cases. These optimizations may include expanding, compressing, rearranging parts of the prompts, and selecting different models for different calls.To systematically explore these opportunities, it is essential to co-design the programming interface and runtime. However, inference engines like Orca and vLLM fall short, as they optimize only for single generation calls. This limitation leads to redundant computations and missed opportunities when executing LLM programs that involve multiple calls.2.4Existing LLM Programming SystemsGiven the significance of LLM programming, specialized programming systems have been developed. The top layer comprises high-level application development libraries with pre-built modules, such as LangChain. The middle layer features more flexible pipeline programming frameworks like DSPy. At the bottom layer, there are domain-specific languages designed for controlling a single prompt, including LMQLand Guidance.SGLang, uniquely positioned between the middle and bottom layers, shares similarities with LMQL and Guidance. However, unlike LMQL and Guidance, which primarily focus on language design while often overlooking runtime efficiency, SGLang thoughtfully integrates front-end language features with back-end runtime optimizations. This approach significantly enhances runtime efficiency, sometimes by an order of magnitude or more.
For its front-end, LMQL employs an SQL-like custom syntax, contrasting with the more intuitive Python syntax. Guidance initially used the Handlebars template language and later transitioned to a Pythonic language embedded within Python. The front-end language of SGLang, inspired by Guidance, is being developed concurrently with the new version of Guidance.SGLang introduces new primitives for enhanced expressiveness and runtime efficiency. It also offers the flexibility of both interpreter and compiler modes. A comparison among LMQL, Guidance, and SGLang is presented inTable 1. We omit the comparison with LangChain and DSPy because they are at a higher level. It is possible to compile high-level pipelines from LangChain and DSPy to SGLang.
body
Efficiently Programming Large Language Models using SGLang
2Background
2312.07104
3OverviewFigure 3shows an overview of our system.
At the user level, SGLang programs are written in Python with embedded SGLang primitives for managing prompts and generation calls.
The syntax and primitives of SGLang will be introduced inSection 4.1.
SGLang programs can be executed in two ways: either using an interpreter or a compiler. When using the interpreter, SGLang programs, which are also valid Python programs, are processed by both the Python and SGLang interpreters. The SGLang interpreter handles specific SGLang primitives.
The design of this interpreter is explained inSection 4.2.
Alternatively, a limited set of SGLang programs can be compiled, enabling further optimizations.Section 4.3is dedicated to the SGLang compiler.
Both the interpreter and the compiler work with the SGVM, which is our custom serving engine for LLM generation calls (Section 5) .
A key aspect of SGVM is its RadixAttention feature, which automatically reuses KV caches across multiple calls.Figure 3:The overview of the system.
body
Efficiently Programming Large Language Models using SGLang
3Overview
2312.07104
4Language, Interpreter, and Compiler4.1Syntax and PrimitivesSGLang is implemented as an embedded domain-specific language in Python.Figure 1shows an example of a SGLang program.
This program evaluates the quality of an essay from multiple dimensions using the branch-solve-merge prompting method.
The functionessay_judgetakes two arguments:sandessay.
The variablesmanages the state of a prompt, allowing new strings and SGLang primitives to be appended using+=.
In the function, it first adds the essay to the prompt, and then forks the prompt into three copies.
These forks run in parallel, generating judgments from different dimensions.
Finally, it merges these judgments, generates a summary, and assigns a letter grade to the essay.Figure 4presents another example.
This program generates two tips in parallel for a given topic using the skeleton-of-thought prompting method.
It employs two functions:expand, which elaborates a short tip into a detailed paragraph, andtip_suggestion, which orchestrates the entire process.
It first generates two short tips and then callsexpandto elaborate them in parallel.
Finally, it generates a summary.These examples illustrate the use of SGLang. In SGLang, we provide the following primitives:genfor calling LLM generation,selectfor letting the LLM choose the option with the highest probability from a list,+=orextendfor extending the current prompt,forkfor forking the current prompt state, andjoinfor rejoining the forked prompt states.
Users can interleave these primitives with arbitrary Python control flow and libraries.4.2InterpreterThe most basic way to execute a SGLang program is through an interpreter. In SGLang, a prompt is treated as an asynchronous stream. Executing primitives such as extend, gen, and select on a prompt leads to the interpreter submitting these primitives into the stream for asynchronous execution. These submission calls are non-blocking, allowing the Python interpreter to continue executing Python code without waiting for the LLM generation to finish. This process is akin to launching CUDA kernels to CUDA streams asynchronously. Each prompt is managed by a stream executor in a background thread, enabling users to achieve parallelism.When fetching the generation results from a prompt, the fetch action will be blocked until the desired generation results are ready. This is implemented using event mechanisms, where a generation call sets an event upon producing its output. Thus, the interpreter effectively manages intra-program parallelism and synchronization.Figure 4:The SGLang program for parallel tip suggestion with skeleton-of-thought prompting.4.3CompilerFigure 5:A computational graph for the program inFigure 4. The three streams correspond to three function calls.Another way to run SGLang programs is to compile them as computational graphs and run them with a graph executor. This opens the opportunity for more compilation optimizations, as we can rewrite the graph.4.3.1Tracing and Graph ExecutorWe design an intermediate representation (IR)  for SGLang, representing SGLang program structures and operations as a computational graph.
This graph includes nodes for primitive operators and edges for dependencies. SeeFigure 5for the graph corresponding to the program inFigure 4.
In the program, each call to a decorated function and or fork will create a new prompt state, or a stream.There are several types of nodes. Each operand of+=and+in a SGLang program is represented by an IR node. These includeConstantText,Argument,Gen,Select,Variable,Fork,GetForkItem, andJoin.
There are two types of dependencies.
The first is intra-stream dependency, where operations submitted into a stream using+=must be executed after all preceding operations in that stream. The second is inter-stream dependency, which occurs when one stream needs to fetch variable values from another stream, necessitating synchronization. Operations like fork manipulate multiple streams and thus introduce inter-stream dependencies.To generate the graph, we use tracing to run the program with abstract arguments and construct the graph dynamically.
This method is limited to programs without data-dependent control flow, a limitation we plan to address in future work.
Once constructed, the graph can be executed through a graph executor, eliminating the need to reinterpret the original Python program. This results in benefits like graph rewriting optimizations, reduced runtime overhead, and program serialization.
For execution, stream executors are launched for each data stream, dispatching IR nodes to the streams in topological order.4.3.2Compilation OptimizationsWe explore two compilation optimizations for SGLang IR. The first one is noteworthy as it integrates GPT-4 into compiler optimization processes. The second one is more conventional.
We anticipate more classical compiler techniques can also be applied, such as auto-tuning and instruction selection.Code Movement for Improving Prefix Sharing.This optimization aims to improve prefix sharing by reordering nodes in the graph to increase the length of the constant prefix. It does not strictly preserve the original computation, classifying it as an aggressive optimization. For instance, changing the prompt from
“Here is a question + {question}. Please act as a math expert and solve the given question” to “Please act as a math expert and solve the given question. Here is a question + {question}.”
results in a longer sharable prefix.
This optimization is interesting because traditional program analysis cannot achieve it, due to the presence of natural language instructions in SGLang. Instead, we prompt GPT-4 to reorder graph nodes. We write a prompt with several examples to teach GPT-4 the concepts of SGLang IR, and we find that GPT-4 can successfully apply this optimization for some simple SGLang programs.Prefetching Annotations.This involves automatically adding prefetching nodes to the graph, signaling the runtime to move the cached KV cache from the CPU to the GPU if the cache exists for a long prompt. This can reduce the latency for pipelines with long prompts.Figure 6:Examples of RadixAttention operations with an LRU eviction policy, illustrated across nine time points.
The figure demonstrates the dynamic evolution of the radix tree in response to various requests. These requests include two chat sessions, a batch of few-shot learning inquiries, and a self-consistency sampling.
Each tree edge carries a label denoting a substring or a sequence of tokens. The nodes are color-coded to reflect different states: green for newly added nodes, blue for cached nodes accessed during the time point, and red for nodes that have been evicted.
In step (1) , the radix tree is initially empty.
In step (2) , the server processes an incoming user message "Hello" and responds with the LLM output "Hi". The system prompt "You are a helpful assistant", the user message "Hello!", and the LLM reply "Hi!" are consolidated into the tree as a single edge linked to a new node.
In step (3) , a new prompt arrives and the server finds the prefix of the prompt (i.e., the first turn of the conversation)  in the radix tree and reuses its KV cache. The new turn is appended to the tree as a new node.
In step (4) , a new chat session begins. The node “b” from (3)  is split into two nodes to allow the two chat sessions to share the system prompt.
In step (5) , the second chat session continues. However, due to the memory limit, node "c" from (4)  must be evicted. The new turn is appended after node "d" in (4) .
In step (6) , the server receives a few-shot learning query, processes it, and inserts it into the tree. The root node is split because the new query does not share any prefix with existing nodes.
In step (7) , the server receives a batch of additional few-shot learning queries. These queries share the same set of few-shot examples, so we split node ’e’ from (6)  to enable sharing.
In step (8) , the server receives a new message from the first chat session. It evicts all nodes from the second chat session (node "g" and "h")  as they are least recently used.
In step (9) , the server receives a request to sample more answers for the questions in node "j" from (8) , likely for self-consistency prompting. To make space for these requests, we evict node "i", "k", and "l" in (8) ."
body
Efficiently Programming Large Language Models using SGLang
4Language, Interpreter, and Compiler
2312.07104
5Runtime with RadixAttentionThe primary challenge in optimizing the execution of SGLang programs lies in reusing the KV cache across multiple calls and programs, as discussed inSection 2.3. While some existing systems can handle KV cache reuse in certain scenarios, this often requires manual configurations and ad-hoc adjustments. However, even with manual configurations, no existing system can automatically handle all scenarios due to the variety of possible reuse patterns.This section introduces RadixAttention, a novel technique for automatic KV cache reuse during runtime. Instead of discarding the KV cache after finishing a generation request, our approach retains the KV cache for both prompts and generation results in a radix tree. This structure enables efficient prefix search, reuse, insertion, and eviction. We implement a Least Recently Used (LRU)  eviction policy, complemented by a cache-aware scheduling policy, to enhance the cache hit rate.
Furthermore, RadixAttention is compatible with existing techniques like continuous batching and paged attention.5.1RadixAttentionA radix tree is a data structure that serves as a space-efficient alternative to a trie (prefix tree) . Unlike typical trees, the edges of a radix tree can be labeled with not just single elements, but also with sequences of elements of varying lengths.
This feature significantly boosts the efficiency of radix trees.
In our system, we utilize a radix tree to manage a mapping. This mapping is between sequences of tokens, which act as the keys, and their corresponding KV cache tensors, which serve as the values. These KV cache tensors are stored in a non-contiguous, paged layout, where the size of each page is equivalent to one token.Considering the limited capacity of GPU memory, we cannot retrain infinite KV cache tensors, which necessitates an eviction policy. To tackle this, we implement an LRU eviction policy that recursively evicts leaf nodes.
In the continuous batching setting, we cannot evict nodes that are being used by the currently running batch. Therefore, each node maintains a reference counter indicating how many running requests are using it. A node is evictable if its reference counter is zero.The front end always sends full prompts to the runtime and the runtime will automatically do prefix matching, reuse, and caching.Figure 6illustrates how the radix tree is maintained when processing several incoming requests.
The tree structure is stored on the CPU and the maintenance overhead is small.5.2Cache-Aware SchedulingTo increase the cache hit rate, the scheduler must be cache-aware. Otherwise, when a large batch of requests arrives, the scheduler may switch between different requests, leading to cache thrashing.Algorithm 1presents the algorithm for cache-aware scheduling in RadixAttention with contiguous batching. The key idea is to sort the requests by matched prefix length, as opposed to adhering to a first-come-first-serve schedule.
There are further opportunities for the implementation of more advanced scheduling strategies that assign priority based on the weights of subtrees. Essentially, this would involve counting the total number of requests within each sub-tree(prefix-sharing trees)  and even considering requests that have been historically evicted. The weighted priority can fully utilize the tree-shape locality of some LLM tasks such as tree-of-thought, making the least redundant evictions.Algorithm 1Cache-Aware Scheduling for RadixAttention with Continuous Batching.The radix tree, the memory pool, the current running batch, the waiting queue.Finished requests and updated system state.// Get all requests from the waiting queue// Search for the prefix matching for all waiting requestsfordo// Sort the requests according to the matched prefix lenghts// Select requests for the next batchfordoifthen// Insert requests into the current running batch// Allocate new memory and do eviction if necessaryifnotthen// Process finished requestsfordoreturn5.3Other OptimizationsThe development of new CUDA kernels is essential to enhance the efficiency of KV cache reuse.
Existing systems mostly implement CUDA kernels for two main functions: prefill and decoding. The prefill kernel assumes that the lengths of the query and key are identical. Conversely, the decode kernel assumes that the query length is one.
However, in the case of KV cache reuse, these kernels cannot be directly applied.
We introduce a new “extend” kernel for this computation, capable of reading the KV cache from both contiguous buffers and non-contiguous memory pools. This fused extend kernel operates without extra copy overhead.
body
Efficiently Programming Large Language Models using SGLang
5Runtime with RadixAttention
2312.07104
6EvaluationSGLang and SGVM are implemented in approximately 9K lines of Python code. SGVM reuses some components from LightLLM, an LLM serving system based on PyTorchand Triton.We evaluate the performance of SGLang across a diverse range of LLM workloads. Subsequently, we conduct case studies and ablation studies to demonstrate the effectiveness of specific components.6.1SetupModels.We test the Llama 2 chat models, CodeLlama models, and vision language model LLaVAmodels of various sizes. The number of parameters ranges from 7B to 33B. We use float16 precision for Llama and CodeLlama models, and 4-bit quantized precision for LLaVA models.Hardware.Most experiments are conducted on AWS EC2 G5 instances unless otherwise stated. The GPUs on G5 instances are NVIDIA A10G GPUs, each with 24 GB of memory. The CPUs are AMD EPYC processors. For most experiments, we run 7B models with a single A10G GPU.
To see how our system handles larger models, we also test a 33B model on an A100 GPU (80GB) .Baseline.We compare SGLang against both high-level programming systems with their respective languages and default runtimes, as well as low-level inference engines with standard OpenAI-like Completion APIs. The baselines include:•Guidance, a language for controlling LLMs. We use Guidance v0.1.4 with the llama.cpp backend.•LMQL, a query language for LLMs. We use LMQL v0.7.3 with the Hugging Face Transformers backend.•vLLM, a high-throughput inference engine. We use vLLM v0.2.2 and its default API server.Metrics.We report two performance metrics: throughput and latency. For throughput, we run a sufficiently large batch of program instances to compute the maximum throughput, comparing the number of program instances executed per second (programs per second, p/s) . For latency, we execute a single program at a time without batching and report the average latency for multiple instances. Unless otherwise stated, most of our optimizations do not change the computation results.While our primary focus is on runtime efficiency, we also conduct several case studies to compare the number of lines of code for productivity.6.2End-to-End Benchmark6.2.1Few-Shot In-Context LearningWe start with the most basic few-shot learning prompting.
We evaluate the maximum throughput of systems during batch processing of few-shot learning requests using three benchmarks, each with 5 shots. Results are shown inFigure 7.MMLUis a multi-choice knowledge benchmark where the LLM outputs a single token (A/B/C/D)  for answer selection. LMQL exhibits the worst performance due to its slow backend and complex pre- and post-processing. Guidance, being faster than LMQL, benefits from reduced processing overhead and some KV cache reuse. However, both systems support only a batch size of one. vLLM performs similarly to Guidance, but it lacks automatic KV cache reuse for the shared prefix. Although the vLLM paper discusses this feature, the released code does not support it due to the absence of a reliable high-performance kernel.
Even if a high-performance kernel is implemented for vLLM, manual configuration is required to use this feature.
Conversely, SGLang automatically achieves sharing with RadixAttention and attains a throughput that ishigher than its closest competitor.HellaSwagis a multi-choice benchmark where each option is a short string. Unlike MMLU, which decodes a single output token, HellaSwag requires an LLM to calculate the probability of each string choice by performing a forward pass. LMQL exhibits slowness due to high overhead.
vLLM lacks a “select” primitive. Instead, it concatenates each choice with the prompt and computes the probability for the entire concatenated prompt for all choices, leading to redundant calculations.
Guidance, although offering a “select” feature, is limited to a batch size of 1. SGLang outperforms others bywith automatic KV cache sharing and batching.GSM-8Kis a benchmark that requires LLMs to generate a reasoning process and produce an integer answer. Since the benchmark demands longer output lengths, batching is crucial for enhanced throughput.
However, LMQL and guidance currently lack support for batching, leading to reduced performance. vLLM is also slow due to its inability to reuse the KV cache for the shared prefix.
In contrast, SGLang achieves aincrease in throughput, thanks to the integration of RadixAttention and contiguous batching.Figure 7:Throughput on few-shot in-context learning tasks.Due to its overhead, LMQL often performs significantly worse than other systems in both short and long-generation tasks, so we will not include it in our future benchmarks.6.2.2LLM-Based AgentsWe benchmark on two highly regarded agent frameworks: ReAct agent, which processes tasks in an interleaved manner by chaining multiple LLM calls with actions like web search; and generative agents, employed in a gaming environment where agents interact with each other and make decisions by invoking LLMs. Both original papers utilized OpenAI models.
To accurately gauge potential performance gains, we mirrored the settings described in the papers and used GPT-3.5-turbo as the backbone model, and then traced the input and output of LLM calls to create a dataset.
For ReAct agents, we used the HotpotQAevaluation as reported in the paper. For generative agents, we rewrote the top five functions (out of forty traced)  from the game using SGLang, which accounts for approximately 70% of all LLM calls.
We then replayed these traces on our experimental platform.As depicted in Figure8, the ReAct agent, using SGLang, achieves a throughput that ishigher than that of vLLM and has onlyof its latency. This improvement primarily results from the agent’s process of appending current states (thoughts, actions, and observations)  to the prompt for subsequent LLM calls. Our RadixAttention mechanism effectively identifies and minimizes redundant computations, a capability not present in the current inference engines. This mechanism also significantly reduces memory usage, allowing more questions to be addressed by a greater number of agents, thereby optimizing GPU utilization and enhancing throughput.In the case of the generative agent task, the gaming setting’s constraints permit only a single LLM call per simulation to maintain game logic consistency. In this case, SGLang still outperforms vLLM by 30% in both throughput and latency. This efficiency is due to the caching of agents’ predefined routines, reducing redundant computations.
The task prompt templates contain multiple arguments that vary at different speeds based on timestamps, resulting in complex prefix reuse patterns. Nevertheless, the SGLang runtime can automatically detect these patterns and reuse the KV cache.Figure 8:Throughput and latency on agent tasks.Figure 9:(a)  Throughput on reasoning tasks. (b)  Latency on reasoning tasks. (c)  Throughput on content generation tasks. (d)  Latency on content generation tasks. (e)  Throughput on long-context tasks.6.2.3ReasoningWe benchmark the performance of systems using advanced prompting techniques for reasoning tasks. These tasks involve solving GSM-8K problems in a zero-shot setting. Both throughput and latency metrics are reported inFigure 9(a) (b) .Multi-chain reasoninginvolves generating several solutions to one question and using meta-reasoning across these solutions for the final answer.
This process allows for sharing the question and parallelizing the generation of multiple solutions.
For each solution generation, we use various eliciting prompts such as "Think step-by-step" and "Take a deep breath". Unlike simple parallel sampling, this flexible approach with different eliciting prompts is not supported by the vLLM API. Hence, vLLM exhibits lower throughput due to its inability to share prefixes flexibly.
Guidance has lower throughput due to no batching.
In contrast, SGLang allows users to intuitively implement this technique without worrying about manual optimizations, yet achieve high throughput.
The improvement in throughput is not as substantial as in previous cases of few-shot learning, due to the zero-shot setting having a shorter shareable prefix.Regarding latency, the SGLang interpreter can automatically resolve dependencies and submit requests in parallel. For a fair comparison, we directly translate the SGLang program into an equivalent Python program without explicitly using threading or asyncio to avoid added complexity. Consequently, the translated program does not parallelize parallelizable requests, resulting in higher latency.Tree-of-thoughtincorporates LLMs into the breadth-first tree-search algorithm. The LLMs must propose the next steps for branch exploration, execute these proposals, and evaluate the solutions.
This method leads to complex sharing patterns beyond the capabilities of existing systems.
Once again, SGLang surpasses others in terms of both higher throughput and lower latency in this benchmark.6.2.4Content GenerationWe evaluate two advanced prompting methods for content generation: tip generation and article review.
The skeleton-of-thought techniquefirst creates a brief outline of tips, then elaborates each into a detailed paragraph. The branch-solve-merge approachassesses an article from multiple dimensions and then assigns a final grade.
Results are shown inFigure 9(c) (d) .
In both cases, there are opportunities for sharing and intra-program parallelism. SGLang systematically exploits them through an intuitive programming model, achieving higher throughput and reduced latency.6.2.5Long Document ProcessingWe benchmark the performance on long document processing tasks. When the document is long, KV cache reuse becomes more important as the forward pass to encode the prompt becomes significantly longer. We use CodeLlama models, which were fine-tuned on a context length of 16K tokens. On our setup, running a 7B model in fp16 precision on a 24 GB GPU, we can store at most 15K tokens. We select inputs exceeding 10K tokens in length, meaning the system can process only one request at a time at most.
The first benchmark tests answering questions using multiple document segments from a paper, akin to a retrieval augmented generation setup. The second benchmark involves summarizing a city’s information from its Wikipedia page and outputting a JSON object.For the first benchmark, we employ the fork/join primitives in SGLang to implement parallel-context window. By using this technique, we can encode multiple documents separately in parallel and concatenate the resulting KV cache, without letting them attend to each other. This approach reduces the quadratic computational cost of self-attention for long prompts. While this optimization alters the computation, it does not negatively impact accuracy in our benchmark.
For the second benchmark, we reuse the KV cache of the long prompts when decoding multiple fields of the JSON object. The results are shown inFigure 9(e) . SGLang achieves 1.2× and 2.9× improvement on these two benchmarks, respectively.
Since the maximum batch size is one, latency improvements are the same as throughput improvements.6.3Case Study6.3.1Vision Language ModelsWe conduct a case study to demonstrate the extensibility of SGLang by supporting a new vision language model, LLaVA v1.5, on a new hardware platform, Apple Silicon.
To achieve this, we add a new primitive,sgl.image, to accept image inputs, and incorporate a new backend, llama.cpp, an inference engine compatible with a wide range of hardware.We integratesgl.imageas a new IR node into the interpreter and reuse the SGVM protocol to build a server based on llama.cpp. This integration is achieved in around 400 lines of code. Additionally, the code that usessgl.imagecan be executed on GPT-4 Vision through the OpenAI back-end without code modification.
We create a cache for image tokens based on the image’s hash ID, similar to how we maintain RadixAttention.
We run the LLaVA-Bench and report the throughput using SGLang and the default llama.cpp server on an M2 Ultra. The LLaVA-Bench contains a total of 24 images and 60 questions, with 2.5 questions on each image on average.
SGLang increases the throughput of llama.cpp by 1.7. The speedup is mainly affected by the number of questions per shared image and the output length.6.3.2Compiler OptimizationsWe evaluate the effectiveness of two compiler optimizations. The first is GPT-4-assisted code movement for prefix sharing. We collect 20 prompt templates from the internet and implement them in SGLang.
We utilize 5 of these templates for few-shot training examples and the remaining 15 as test cases. Our evaluation shows that, for 12 out of the 15 templates, GPT-4 successfully reorders the graph nodes without altering the semantics, as confirmed by manual inspection of the modified prompts.
On average, this optimization results in a 60-token increase in the sharable prefix length, showcasing GPT-4’s effectiveness. Failures in creating optimized prompt order come from an incorrect understanding of the semantic meaning behind the graph nodes. It is too aggressive and puts all constants upfront even when such ordering changes the original semantics.
This case study aims to explore the use of GPT-4 for compiler optimizations. More work is needed to make these kinds of optimizations reliable in the future.The second optimization involves prefetching annotation insertion.
For frequently used long prompts, we cache them in CPU memory. The compiler automatically inserts prefetch nodes into the computational graph. During runtime, when the server encounters a prefetch node, it evicts nodes from the radix tree and swaps in the cached prefix.
As the cost of swapping can be substantially overlapped, this reduces the first-token latency of the subsequent request.
We test SGLang programs that contain function calls with ak prefix length. By prefetching thek prefix, the first token latency of the called function has been reduced from 1 second to 0.2 seconds, which is a 80% reduction.6.3.3ProductivityWe conducted a case study to demonstrate SGLang’s productivity improvement by comparing its Lines of Code (LoC)  with OpenAI APIs. We designed a prompting flow for article analysis: if it’s about a celebrity, the program generates a bio and introduction; otherwise, it summarizes the article. Implementing this flow in both OpenAI APIs and SGLang, we counted effective LoC, excluding comments and empty lines. The results show the OpenAI API required 206 lines, while SGLang needed only 91, marking a 55% reduction.We delve into the implementations and summarize the insights regarding why SGLang is capable of using fewer LoC.
First, whereas users must manually extract content from OpenAI API responses and chain them to form a multi-step flow, SGLang simplifies this process by leveraging the string concatenation operator.
Second, since the flow involves several calls with restricted answers, such as true/false and integers, users are required to construct an appropriate logit bias to ensure OpenAI returns one of the expected answers. In contrast, SGLang provides primitives that automatically construct logit bias for this purpose, allowing users to merely specify the acceptable data type and choices.
Third, in cases where skeleton-based prompting is used to reduce end-to-end latency, OpenAI API users need to initiate a thread pool for parallel request submission explicitly. However, SGLang’s built-in primitiveforkautomatically parallelizes requests.6.3.4Larger ModelsWhile most of our experiments are conducted using a 7B model, our techniques also perform effectively on larger models. We test a 13B model on an A100 (40GB)  and a 33B model on an A100 (80GB)  using the ReAct agent benchmark, achieving a similar speedup () .6.4Ablation Study6.4.1Scheduling PolicyFigure 10:Performance of different scheduling policies.We evaluate the effectiveness of our cache-aware scheduling by comparing it with several other policies. The results are shown inFigure 10. “LMP”, or longest matching prefix, is the default policy outlined inSection 5.2. “Random” employs a random order. “FCFS” adheres to a first-come-first-serve policy. “No interpreter help” omits the prefix hints from the interpreter, which is explained in detail below.
Although RadixAttention can automatically detect shared prefixes, our runtime implementation is simplified: we only compare a pending request with the existing cache tree. We do not construct a tree index for all requests in the waiting queue. This approach may lead to suboptimal performance when there is a large batch of pending requests that share a prefix but have yet to be processed and inserted into the tree. To mitigate this, we use the interpreter to help with the scheduling. When the interpreter identifies a highly likely shared prefix (e.g., during a fork) , it issues a forward request for this prefix first and stores it in the tree, and then it sends out the requests that can reuse the prefix. This method leverages hints from the interpreter to lessen the load on backend scheduling, underscoring the benefits of our co-design strategy.
As depicted inFigure 10, “LPM” exhibits superior performance, while other policies struggle with a lower cache hit rate and reduced throughput.
All policies perform similarly for simple tasks like MMLU.6.4.2Optimization BreakdownWe study the effectiveness of optimizations by disabling each one individually.Figure 11illustrates the results.
“No Extend Kernel” disables our optimized kernel, leading to degraded performance, particularly in long-context tasks. “No Interpreter Parallelism” turns off intra-program parallelism, resulting in reduced parallelism and cache locality.
“No Cache” deactivates the cache, significantly deteriorating throughput across all benchmarks. “No Radix Tree” uses a list instead of a tree to manage the cache, leading to increased memory usage and copy overhead, and consequently, lower performance.Figure 11:Optimization breakdown.6.4.3Improvements under Different Sharing SettingsWe study how the improvement of prefix sharing is influenced by different parameters, including the lengths of shareable and unshareable inputs, and sharing structure configurations.As shown inFigure 12, in the left subfigure, we establish a base setting with a shareable input length of 256, an unshareable input length of 256, and an output length of 256. We then gradually increase either the shareable input length or the unshareable length and observe the changes in throughput. The results demonstrate that the RadixAttention cache significantly enhances throughput compared to the “No Cache” scenario, with more notable improvements observed when the shareable length is longer.In the right subfigure, we set a base scenario with 5 articles and 2 questions per article, where the article comprises 1000 tokens and each question 256 tokens. There arerequests in this base setting, with each article being shared by two questions. Subsequently, we increment either the number of articles or the number of questions per article. The findings indicate that the RadixAttention cache exhibits superior performance when the number of questions per article increases, as this creates more opportunities for sharing.6.4.4RadixAttention Overhead AnalysisWe analyze the overhead of maintaining the radix tree at runtime. The tree structure is maintained on the CPU. In a typical scenario, which has a shareable input length of 256, an unshareable input length of 256, and 128 requests (organized into 16 subtrees, with 8 branches for each subtree) , the forward pass in our setup takes 17.6 seconds. Meanwhile, the tree operation only takes 0.07 seconds.
We also construct a request trace with no cache hits and observe the tree maintenance incurs only a 0.5% overhead.
Therefore, the overhead is negligible. Furthermore, the overhead of tree maintenance can be further reduced by using a high-performance C++ implementation and by overlapping CPU and GPU operations.Figure 12:Improvements under Different Sharing Settings
body
Efficiently Programming Large Language Models using SGLang
6Evaluation
2312.07104
7Discussion and LimitationsHere are SGLang’s limitations and future development plans:•Our interpreter runs arbitrary Python code, but the tracing and compiler lack support for data-dependent control flows. Introducing new control flow nodes to the SGLang IR, similar to TensorFlow’s dynamic flow, is required to address this.•The current tokenization algorithm may create artifacts at prompt and generation boundaries. Implementing the token healing techniquecould resolve this.•We have not yet implemented grammar-constrained decoding. Efficient implementation would necessitate techniques fromand careful batching.•Plans include integrating more input and output modalities, such as incorporating text-to-image models into SGLang and exploring audio and video capabilities.
body
Efficiently Programming Large Language Models using SGLang
7Discussion and Limitations
2312.07104
8Related WorkProgramming with Language ModelsThe trend in programming LLM includes the development of better prompting techniques, tools usage, agent frameworks, and programming systems. Among these, Guidance and LMQL are the most similar to our work. We discuss the differences between SGLang and existing LLM programming systems inSection 2.4and evaluate their performance inSection 6. In short, the co-design approach of SGLang significantly enhances execution efficiency.Inference Optimizations for LLMs.Common inference optimizations for LLMs include batching, memory optimizations, GPU kernel optimizations, model parallelism, parameter sharing, speculative execution, scheduling, caching,
quantization, and sparsification. These optimizations are complementary to our work and can be, or have already been, integrated into our backend.Compilers for Machine Learning.Various domain-specific compilers for machine learning have been developed. These compilers are capable of optimizing a single tensor operator, a computational graph, a distributed computation graph, and classical algorithms. Distinct from these more general-purpose or low-level compilers, SGLang is specifically designed for LLM applications and incorporates LLM-specific optimizations.
body
Efficiently Programming Large Language Models using SGLang
8Related Work
2312.07104
9ConclusionIn this paper, we introduce SGLang, an efficient programming system for LLMs.
SGLang greatly accelerates common LLM workloads by up towith reduced code complexity and improved control.
Through the co-design of the language and runtime, we aim to present a more integrated approach to optimizing LLM programming paradigms.
conclusion
Efficiently Programming Large Language Models using SGLang
9Conclusion