<!DOCTYPE html><html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>[2404.03683] Stream of Search (SoS): Learning to Search in Language</title><meta property="og:description" content="Language models are rarely shown fruitful mistakes while training. They then struggle to look beyond the next token, suffering from a snowballing of errors and struggling to predict the consequence of their actions sev…">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="Stream of Search (SoS): Learning to Search in Language">
<meta name="twitter:image:src" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta name="twitter:image:alt" content="ar5iv logo">
<meta property="og:title" content="Stream of Search (SoS): Learning to Search in Language">
<meta property="og:site_name" content="ar5iv">
<meta property="og:image" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta property="og:type" content="article">
<meta property="og:url" content="https://ar5iv.labs.arxiv.org/html/2404.03683">

<!--Generated on Sun May  5 19:04:41 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.-->
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

<script>
  function detectColorScheme(){
    var theme="light";
    var current_theme = localStorage.getItem("ar5iv_theme");
    if(current_theme){
      if(current_theme == "dark"){
        theme = "dark";
      } }
    else if(!window.matchMedia) { return false; }
    else if(window.matchMedia("(prefers-color-scheme: dark)").matches) {
      theme = "dark"; }
    if (theme=="dark") {
      document.documentElement.setAttribute("data-theme", "dark");
    } else {
      document.documentElement.setAttribute("data-theme", "light"); } }

  detectColorScheme();

  function toggleColorScheme(){
    var current_theme = localStorage.getItem("ar5iv_theme");
    if (current_theme) {
      if (current_theme == "light") {
        localStorage.setItem("ar5iv_theme", "dark"); }
      else {
        localStorage.setItem("ar5iv_theme", "light"); } }
    else {
        localStorage.setItem("ar5iv_theme", "dark"); }
    detectColorScheme(); }
</script>
<link media="all" rel="stylesheet" href="/assets/ar5iv-fonts.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv-site.0.2.2.css">
</head>
<body>
<div class="ltx_page_main">
<div class="ltx_page_content">
<article class="ltx_document ltx_authors_1line">
<h1 class="ltx_title ltx_title_document">Stream of Search (SoS): Learning to Search in Language</h1>
<div class="ltx_authors">
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">
<span id="id2.1.id1" class="ltx_text ltx_font_bold">Kanishk Gandhi</span> 
<br class="ltx_break">Stanford University
<br class="ltx_break">
</span><span class="ltx_author_notes">Corresponding author: <span id="id3.2.id1" class="ltx_text ltx_font_typewriter">kanishk.gandhi@stanford.edu</span></span></span>
<span class="ltx_author_before">  </span><span class="ltx_creator ltx_role_author">
<span class="ltx_personname"><span id="id4.1.id1" class="ltx_text ltx_font_bold">Denise Lee</span> 
<br class="ltx_break">Stanford University 
<br class="ltx_break">
</span></span>
<span class="ltx_author_before">  </span><span class="ltx_creator ltx_role_author">
<span class="ltx_personname"><span id="id5.1.id1" class="ltx_text ltx_font_bold">Gabriel Grand</span> 
<br class="ltx_break">MIT 
<br class="ltx_break">
</span></span>
<span class="ltx_author_before">  </span><span class="ltx_creator ltx_role_author">
<span class="ltx_personname"><span id="id6.1.id1" class="ltx_text ltx_font_bold">Muxin Liu</span> 
<br class="ltx_break">Harvey Mudd 
<br class="ltx_break">
</span></span>
<span class="ltx_author_before">  </span><span class="ltx_creator ltx_role_author">
<span class="ltx_personname"><span id="id7.1.id1" class="ltx_text ltx_font_bold">Winson Cheng</span> 
<br class="ltx_break">Stanford University 
<br class="ltx_break">
</span></span>
<span class="ltx_author_before">  </span><span class="ltx_creator ltx_role_author">
<span class="ltx_personname"><span id="id8.1.id1" class="ltx_text ltx_font_bold">Archit Sharma</span> 
<br class="ltx_break">Stanford University 
<br class="ltx_break">
</span></span>
<span class="ltx_author_before">  </span><span class="ltx_creator ltx_role_author">
<span class="ltx_personname"><span id="id9.1.id1" class="ltx_text ltx_font_bold">Noah D. Goodman
<br class="ltx_break"></span>Stanford University
</span></span>
</div>

<div class="ltx_abstract">
<h6 class="ltx_title ltx_title_abstract">Abstract</h6>
<p id="id10.id1" class="ltx_p">Language models are rarely shown fruitful mistakes while training. They then struggle to look beyond the next token, suffering from a snowballing of errors and struggling to predict the consequence of their actions several steps ahead. In this paper, we show how language models can be taught to search by representing the process of search in language, as a flattened string — a <em id="id10.id1.1" class="ltx_emph ltx_font_italic">stream of search (SoS)</em>. We propose a unified language for search that captures an array of different symbolic search strategies.
We demonstrate our approach using the simple yet difficult game of Countdown, where the goal is to combine input numbers with arithmetic operations to reach a target number.
We pretrain a transformer-based language model from scratch on a dataset of streams of search generated by heuristic solvers. We find that SoS pretraining increases search accuracy by 25% over models trained to predict only the optimal search trajectory.
We further finetune this model with two policy improvement methods: Advantage-Induced Policy Alignment (APA) and Self-Taught Reasoner (STaR). The finetuned SoS models solve 36% of previously unsolved problems, including problems that cannot be solved by any of the heuristic solvers.
Our results indicate that language models can learn to solve problems via search, self-improve to flexibly use different search strategies, and potentially discover new ones.
<span id="footnote1" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">1</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">1</sup><span class="ltx_tag ltx_tag_note">1</span>Code Available Here: <a target="_blank" href="https://github.com/kanishkg/stream-of-search" title="" class="ltx_ref ltx_href">https://github.com/kanishkg/stream-of-search</a></span></span></span></p>
</div>
<div id="id1" class="ltx_logical-block">
<div id="id1.p1" class="ltx_para">
<p id="id1.p1.1" class="ltx_p ltx_align_right">“To err is human, to backtrack divine”</p>
<p id="id1.p1.2" class="ltx_p ltx_align_right">—Apocryphal</p>
</div>
</div>
<section id="S1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">1 </span>Introduction</h2>

<div id="S1.p1" class="ltx_para ltx_noindent">
<p id="S1.p1.1" class="ltx_p">Imagine, only ever seeing the right solutions to problems, never a mistake or recovery from it.
You might learn that problems must be solved in one clean pass, rather than through exploration and error.
Most data used to train language models (LMs) only reflects the outcome of a decision making process, not the process itself. LMs never learn to make mistakes. They never learn to search, plan or backtrack. Complex decision-making and reasoning requires search. In this paper we explore the impact of training a LM on the search process, including mistakes, and then allowing them to self-improve.</p>
</div>
<figure id="S1.F1" class="ltx_figure"><img src="/html/2404.03683/assets/x1.png" id="S1.F1.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="422" height="258" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 1: </span>Overview of the Stream of Search (SoS) framework. (a) A search problem in Countdown is instantiated with input numbers and a target number. The input numbers need to be combined with simple arithmetic operations to get to the target. (b) The Stream of Search dataset contains search trajectories generated by diverse search strategies, including exploration and backtracking. (c) The language model is first trained on the SoS dataset and then iteratively improved using policy improvement techniques such as APA and STaR.
</figcaption>
</figure>
<div id="S1.p2" class="ltx_para ltx_noindent">
<p id="S1.p2.1" class="ltx_p">Transformer-based auto-regressive models have been shown to struggle with planning <cite class="ltx_cite ltx_citemacro_citep">(Valmeekam et al., <a href="#bib.bib24" title="" class="ltx_ref">2024</a>; Pallagani et al., <a href="#bib.bib16" title="" class="ltx_ref">2023</a>; Momennejad et al., <a href="#bib.bib15" title="" class="ltx_ref">2024</a>)</cite>.
Recent work has highlighted this weakness in autoregressive models by identifying two main issues <cite class="ltx_cite ltx_citemacro_citep">(LeCun, <a href="#bib.bib12" title="" class="ltx_ref">2023</a>; Bachmann &amp; Nagarajan, <a href="#bib.bib3" title="" class="ltx_ref">2024</a>)</cite>: 1) the snowballing of errors, where a single mistake can compound and lead to increasingly poor performance in subsequent steps <cite class="ltx_cite ltx_citemacro_citep">(Ross et al., <a href="#bib.bib17" title="" class="ltx_ref">2011</a>; Arora et al., <a href="#bib.bib2" title="" class="ltx_ref">2022</a>)</cite>, and 2) a difficulty in ‘lookahead tasks’, where the model must predict the consequences of its actions several steps ahead <cite class="ltx_cite ltx_citemacro_citep">(credit assignment, Cf. Sutton &amp; Barto, <a href="#bib.bib23" title="" class="ltx_ref">2018</a>)</cite>. Both of these issues can be attributed to limited ability to search and backtrack. While recent efforts have combined language models with symbolic search algorithms <cite class="ltx_cite ltx_citemacro_citep">(Ahn et al., <a href="#bib.bib1" title="" class="ltx_ref">2022</a>; Yao et al., <a href="#bib.bib26" title="" class="ltx_ref">2024</a>)</cite> to mitigate some of these problems, they are limited — only supplementing language models during inference —they leave open the question of whether language models could effectively carry out search themselves. The most consequential outcome from learning to search could be during the <em id="S1.p2.1.1" class="ltx_emph ltx_font_italic">training</em> itself <cite class="ltx_cite ltx_citemacro_citep">(Silver et al., <a href="#bib.bib21" title="" class="ltx_ref">2018</a>)</cite>. If language models can learn to search during training, they may be able to discover more flexible search strategies through self-improvement. This could lead to models that are better equipped to handle the challenges posed by error compounding and lookahead tasks.</p>
</div>
<div id="S1.p3" class="ltx_para ltx_noindent">
<p id="S1.p3.1" class="ltx_p">In this paper, we demonstrate that language models can be taught to search and backtrack in language, representing the process as a serialized string, a Stream of Search (SoS). We systematize the different components of search, such as exploration, backtracking, and pruning in a unified language. We instantiate this unified language in the context of a search problem inspired by the game of Countdown, a generalized version of the game of 24 <cite class="ltx_cite ltx_citemacro_citep">(Yao et al., <a href="#bib.bib26" title="" class="ltx_ref">2024</a>; Countdown, <a href="#bib.bib5" title="" class="ltx_ref">2024</a>)</cite>. A problem consists of input numbers and a target number. The goal is to combine the input numbers with arithmetic operations to reach the target (see <a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ Stream of Search (SoS): Learning to Search in Language" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Fig. 1</span></a>a).
Countdown presents a challenging search problem due to its high branching factor and the need to efficiently navigate the combinatorial search space towards the target number.
We generate an initial training dataset of search trajectories or streams of search using different symbolic planners and simple heuristic functions, then train a language model on this diverse dataset. This can be contrasted with training on the optimal path once a solution is found, without the process of search and backtracking.
We find that the Stream of Search LM substantially outperforms models trained to predict the optimal steps. Moreover, we observe improvements in the search and planning abilities of the Stream of Search model when finetuned to optimize for correctness using Advantage-Induced Policy Alignment <cite class="ltx_cite ltx_citemacro_citep">(APA, Zhu et al., <a href="#bib.bib28" title="" class="ltx_ref">2023</a>)</cite> and expert iteration with STaR <cite class="ltx_cite ltx_citemacro_citep">(Zelikman et al., <a href="#bib.bib27" title="" class="ltx_ref">2022</a>; Gulcehre et al., <a href="#bib.bib10" title="" class="ltx_ref">2023</a>)</cite>.</p>
</div>
<div id="S1.p4" class="ltx_para ltx_noindent">
<p id="S1.p4.1" class="ltx_p">Our results indicate that transformer-based language models, when shown how to recover from mistakes and search through different options, can learn to solve problems by searching. More importantly, our results indicate that these models can self-improve to autonomously use different search strategies, solving previously unsolved problems. Finally, we see some evidence that they discover new search strategies when trained to optimize for accuracy.</p>
</div>
</section>
<section id="S2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">2 </span>Related Works</h2>

<div id="S2.p1" class="ltx_para ltx_noindent">
<p id="S2.p1.1" class="ltx_p"><span id="S2.p1.1.1" class="ltx_text ltx_font_bold">Language models as components of a search system.</span> A line of recent work integrates language models as part of larger search and planning systems <cite class="ltx_cite ltx_citemacro_citep">(Yao et al., <a href="#bib.bib26" title="" class="ltx_ref">2024</a>; Ahn et al., <a href="#bib.bib1" title="" class="ltx_ref">2022</a>)</cite>. In these methods, LMs typically play two roles: (1) to generate candidate actions or successor states in the reasoning process, and (2) to evaluate proposed actions or states, by determining validity and/or assigning a heuristic value. A symbolic search algorithm such as BFS or DFS dictates the strategy for exploration, and how the steps or evaluators are called <cite class="ltx_cite ltx_citemacro_citep">(Yao et al., <a href="#bib.bib26" title="" class="ltx_ref">2024</a>; Besta et al., <a href="#bib.bib4" title="" class="ltx_ref">2023</a>)</cite>. While these methods have been shown to improve search accuracy on certain problems, the LM components are typically used only for inference, so their reasoning ability is not improved. In contrast, our work focuses on training LMs that are capable of exploration, backtracking, and other critical components of reasoning. Relative to these “extrinsic” methods, which use fixed search strategies, our method learns an “intrinsic” policy that allows the LM to autonomously search the solution space. In doing so, we avoid the high inference costs <cite class="ltx_cite ltx_citemacro_citep">(Sel et al., <a href="#bib.bib19" title="" class="ltx_ref">2023</a>)</cite> required by tree-of-thoughts style approaches.</p>
</div>
<div id="S2.p2" class="ltx_para ltx_noindent">
<p id="S2.p2.1" class="ltx_p"><span id="S2.p2.1.1" class="ltx_text ltx_font_bold">In-context demonstrations of search.</span> In contrast to using separate symbolic search algorithms to guide search, in-context demonstrations can also be used to demonstrate search procedures in language <cite class="ltx_cite ltx_citemacro_citep">(Gandhi et al., <a href="#bib.bib8" title="" class="ltx_ref">2023</a>; Sel et al., <a href="#bib.bib19" title="" class="ltx_ref">2023</a>)</cite>. These methods allow the language model to perform tree search based on the demonstrated search procedures. While improving efficiency in search, these methods are restricted by the choices of the search procedure demonstrated in the in-context examples, such as the demonstrated exploration strategy or the search heuristic.</p>
</div>
<div id="S2.p3" class="ltx_para ltx_noindent">
<p id="S2.p3.1" class="ltx_p"><span id="S2.p3.1.1" class="ltx_text ltx_font_bold">Process supervision.</span> Process supervision of LMs is another method where an external verifier model is trained to learn the rewards of each intermediate reasoning step and the LM is then trained on this detailed feedback. Process supervision was shown to outperform outcome supervision in mathematical reasoning tasks <cite class="ltx_cite ltx_citemacro_citep">(Lightman et al., <a href="#bib.bib14" title="" class="ltx_ref">2023</a>)</cite>. However, the training of the verifier requires a very large labelled dataset with human-generated annotations for each intermediate step, which may not scale well for other, more complex domains. Comparatively, our method directly improves the model’s planning and search abilities and removes the need to train a verifier reward model.</p>
</div>
<div id="S2.p4" class="ltx_para ltx_noindent">
<p id="S2.p4.1" class="ltx_p"><span id="S2.p4.1.1" class="ltx_text ltx_font_bold">Learning from search trajectories.</span> Similar to our work, <cite class="ltx_cite ltx_citemacro_citet">Yang et al. (<a href="#bib.bib25" title="" class="ltx_ref">2022</a>)</cite> train transformer models on search trajectories to mimic search strategies such as Monte Carlo Tree Search or BFS. While their approach focuses on mimicking a search procedure with a fixed heuristic, we are interested in autonomous usage of different search procedures and the discovery of new ones. In recent concurrent work, <cite class="ltx_cite ltx_citemacro_citet">Lehnert et al. (<a href="#bib.bib13" title="" class="ltx_ref">2024</a>)</cite> employ A* search traces to train a base transformer model, where each trace includes the state and fixed A* heuristic value and search procedure. Their work aims to train transformers to closely imitate A* and enhance its efficiency. In contrast, our method emphasizes <em id="S2.p4.1.2" class="ltx_emph ltx_font_italic">discovery</em> of new search strategies and tackling challenging problems that symbolic algorithms struggle to solve.</p>
</div>
</section>
<section id="S3" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">3 </span>A Language for Search</h2>

<div id="S3.p1" class="ltx_para ltx_noindent">
<p id="S3.p1.4" class="ltx_p">The problem space can be modeled as a Markov Decision Process (MDP), with a set of states, <math id="S3.p1.1.m1.1" class="ltx_Math" alttext="\mathcal{S}" display="inline"><semantics id="S3.p1.1.m1.1a"><mi class="ltx_font_mathcaligraphic" id="S3.p1.1.m1.1.1" xref="S3.p1.1.m1.1.1.cmml">𝒮</mi><annotation-xml encoding="MathML-Content" id="S3.p1.1.m1.1b"><ci id="S3.p1.1.m1.1.1.cmml" xref="S3.p1.1.m1.1.1">𝒮</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.1.m1.1c">\mathcal{S}</annotation></semantics></math>, representing the steps in solving a problem; a set of actions <math id="S3.p1.2.m2.1" class="ltx_Math" alttext="\mathcal{A}" display="inline"><semantics id="S3.p1.2.m2.1a"><mi class="ltx_font_mathcaligraphic" id="S3.p1.2.m2.1.1" xref="S3.p1.2.m2.1.1.cmml">𝒜</mi><annotation-xml encoding="MathML-Content" id="S3.p1.2.m2.1b"><ci id="S3.p1.2.m2.1.1.cmml" xref="S3.p1.2.m2.1.1">𝒜</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.2.m2.1c">\mathcal{A}</annotation></semantics></math>, representing operations that the model can perform on states or to transition between them; a transition function <math id="S3.p1.3.m3.1" class="ltx_Math" alttext="T:\mathcal{S}\times\mathcal{A}\to\mathcal{S}" display="inline"><semantics id="S3.p1.3.m3.1a"><mrow id="S3.p1.3.m3.1.1" xref="S3.p1.3.m3.1.1.cmml"><mi id="S3.p1.3.m3.1.1.2" xref="S3.p1.3.m3.1.1.2.cmml">T</mi><mo lspace="0.278em" rspace="0.278em" id="S3.p1.3.m3.1.1.1" xref="S3.p1.3.m3.1.1.1.cmml">:</mo><mrow id="S3.p1.3.m3.1.1.3" xref="S3.p1.3.m3.1.1.3.cmml"><mrow id="S3.p1.3.m3.1.1.3.2" xref="S3.p1.3.m3.1.1.3.2.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.p1.3.m3.1.1.3.2.2" xref="S3.p1.3.m3.1.1.3.2.2.cmml">𝒮</mi><mo lspace="0.222em" rspace="0.222em" id="S3.p1.3.m3.1.1.3.2.1" xref="S3.p1.3.m3.1.1.3.2.1.cmml">×</mo><mi class="ltx_font_mathcaligraphic" id="S3.p1.3.m3.1.1.3.2.3" xref="S3.p1.3.m3.1.1.3.2.3.cmml">𝒜</mi></mrow><mo stretchy="false" id="S3.p1.3.m3.1.1.3.1" xref="S3.p1.3.m3.1.1.3.1.cmml">→</mo><mi class="ltx_font_mathcaligraphic" id="S3.p1.3.m3.1.1.3.3" xref="S3.p1.3.m3.1.1.3.3.cmml">𝒮</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.p1.3.m3.1b"><apply id="S3.p1.3.m3.1.1.cmml" xref="S3.p1.3.m3.1.1"><ci id="S3.p1.3.m3.1.1.1.cmml" xref="S3.p1.3.m3.1.1.1">:</ci><ci id="S3.p1.3.m3.1.1.2.cmml" xref="S3.p1.3.m3.1.1.2">𝑇</ci><apply id="S3.p1.3.m3.1.1.3.cmml" xref="S3.p1.3.m3.1.1.3"><ci id="S3.p1.3.m3.1.1.3.1.cmml" xref="S3.p1.3.m3.1.1.3.1">→</ci><apply id="S3.p1.3.m3.1.1.3.2.cmml" xref="S3.p1.3.m3.1.1.3.2"><times id="S3.p1.3.m3.1.1.3.2.1.cmml" xref="S3.p1.3.m3.1.1.3.2.1"></times><ci id="S3.p1.3.m3.1.1.3.2.2.cmml" xref="S3.p1.3.m3.1.1.3.2.2">𝒮</ci><ci id="S3.p1.3.m3.1.1.3.2.3.cmml" xref="S3.p1.3.m3.1.1.3.2.3">𝒜</ci></apply><ci id="S3.p1.3.m3.1.1.3.3.cmml" xref="S3.p1.3.m3.1.1.3.3">𝒮</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.3.m3.1c">T:\mathcal{S}\times\mathcal{A}\to\mathcal{S}</annotation></semantics></math>, defining the transition from one state to another based on an action; and a reward function <math id="S3.p1.4.m4.1" class="ltx_Math" alttext="R:\mathcal{S}\to\mathbb{R}" display="inline"><semantics id="S3.p1.4.m4.1a"><mrow id="S3.p1.4.m4.1.1" xref="S3.p1.4.m4.1.1.cmml"><mi id="S3.p1.4.m4.1.1.2" xref="S3.p1.4.m4.1.1.2.cmml">R</mi><mo lspace="0.278em" rspace="0.278em" id="S3.p1.4.m4.1.1.1" xref="S3.p1.4.m4.1.1.1.cmml">:</mo><mrow id="S3.p1.4.m4.1.1.3" xref="S3.p1.4.m4.1.1.3.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.p1.4.m4.1.1.3.2" xref="S3.p1.4.m4.1.1.3.2.cmml">𝒮</mi><mo stretchy="false" id="S3.p1.4.m4.1.1.3.1" xref="S3.p1.4.m4.1.1.3.1.cmml">→</mo><mi id="S3.p1.4.m4.1.1.3.3" xref="S3.p1.4.m4.1.1.3.3.cmml">ℝ</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.p1.4.m4.1b"><apply id="S3.p1.4.m4.1.1.cmml" xref="S3.p1.4.m4.1.1"><ci id="S3.p1.4.m4.1.1.1.cmml" xref="S3.p1.4.m4.1.1.1">:</ci><ci id="S3.p1.4.m4.1.1.2.cmml" xref="S3.p1.4.m4.1.1.2">𝑅</ci><apply id="S3.p1.4.m4.1.1.3.cmml" xref="S3.p1.4.m4.1.1.3"><ci id="S3.p1.4.m4.1.1.3.1.cmml" xref="S3.p1.4.m4.1.1.3.1">→</ci><ci id="S3.p1.4.m4.1.1.3.2.cmml" xref="S3.p1.4.m4.1.1.3.2">𝒮</ci><ci id="S3.p1.4.m4.1.1.3.3.cmml" xref="S3.p1.4.m4.1.1.3.3">ℝ</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.4.m4.1c">R:\mathcal{S}\to\mathbb{R}</annotation></semantics></math>, assigning a reward for reaching the goal state.</p>
</div>
<div id="S3.p2" class="ltx_para ltx_noindent">
<p id="S3.p2.11" class="ltx_p">The process of search can be modeled as follows. The query for search has an initial state <math id="S3.p2.1.m1.1" class="ltx_Math" alttext="s_{0}\in\mathcal{S}" display="inline"><semantics id="S3.p2.1.m1.1a"><mrow id="S3.p2.1.m1.1.1" xref="S3.p2.1.m1.1.1.cmml"><msub id="S3.p2.1.m1.1.1.2" xref="S3.p2.1.m1.1.1.2.cmml"><mi id="S3.p2.1.m1.1.1.2.2" xref="S3.p2.1.m1.1.1.2.2.cmml">s</mi><mn id="S3.p2.1.m1.1.1.2.3" xref="S3.p2.1.m1.1.1.2.3.cmml">0</mn></msub><mo id="S3.p2.1.m1.1.1.1" xref="S3.p2.1.m1.1.1.1.cmml">∈</mo><mi class="ltx_font_mathcaligraphic" id="S3.p2.1.m1.1.1.3" xref="S3.p2.1.m1.1.1.3.cmml">𝒮</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.p2.1.m1.1b"><apply id="S3.p2.1.m1.1.1.cmml" xref="S3.p2.1.m1.1.1"><in id="S3.p2.1.m1.1.1.1.cmml" xref="S3.p2.1.m1.1.1.1"></in><apply id="S3.p2.1.m1.1.1.2.cmml" xref="S3.p2.1.m1.1.1.2"><csymbol cd="ambiguous" id="S3.p2.1.m1.1.1.2.1.cmml" xref="S3.p2.1.m1.1.1.2">subscript</csymbol><ci id="S3.p2.1.m1.1.1.2.2.cmml" xref="S3.p2.1.m1.1.1.2.2">𝑠</ci><cn type="integer" id="S3.p2.1.m1.1.1.2.3.cmml" xref="S3.p2.1.m1.1.1.2.3">0</cn></apply><ci id="S3.p2.1.m1.1.1.3.cmml" xref="S3.p2.1.m1.1.1.3">𝒮</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p2.1.m1.1c">s_{0}\in\mathcal{S}</annotation></semantics></math> and a goal state <math id="S3.p2.2.m2.1" class="ltx_Math" alttext="s_{g}\in\mathcal{S}" display="inline"><semantics id="S3.p2.2.m2.1a"><mrow id="S3.p2.2.m2.1.1" xref="S3.p2.2.m2.1.1.cmml"><msub id="S3.p2.2.m2.1.1.2" xref="S3.p2.2.m2.1.1.2.cmml"><mi id="S3.p2.2.m2.1.1.2.2" xref="S3.p2.2.m2.1.1.2.2.cmml">s</mi><mi id="S3.p2.2.m2.1.1.2.3" xref="S3.p2.2.m2.1.1.2.3.cmml">g</mi></msub><mo id="S3.p2.2.m2.1.1.1" xref="S3.p2.2.m2.1.1.1.cmml">∈</mo><mi class="ltx_font_mathcaligraphic" id="S3.p2.2.m2.1.1.3" xref="S3.p2.2.m2.1.1.3.cmml">𝒮</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.p2.2.m2.1b"><apply id="S3.p2.2.m2.1.1.cmml" xref="S3.p2.2.m2.1.1"><in id="S3.p2.2.m2.1.1.1.cmml" xref="S3.p2.2.m2.1.1.1"></in><apply id="S3.p2.2.m2.1.1.2.cmml" xref="S3.p2.2.m2.1.1.2"><csymbol cd="ambiguous" id="S3.p2.2.m2.1.1.2.1.cmml" xref="S3.p2.2.m2.1.1.2">subscript</csymbol><ci id="S3.p2.2.m2.1.1.2.2.cmml" xref="S3.p2.2.m2.1.1.2.2">𝑠</ci><ci id="S3.p2.2.m2.1.1.2.3.cmml" xref="S3.p2.2.m2.1.1.2.3">𝑔</ci></apply><ci id="S3.p2.2.m2.1.1.3.cmml" xref="S3.p2.2.m2.1.1.3">𝒮</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p2.2.m2.1c">s_{g}\in\mathcal{S}</annotation></semantics></math>.
The search tree is defined by the problem itself. It contains all possible explorations of all actions from the initial state <math id="S3.p2.3.m3.1" class="ltx_Math" alttext="s_{0}" display="inline"><semantics id="S3.p2.3.m3.1a"><msub id="S3.p2.3.m3.1.1" xref="S3.p2.3.m3.1.1.cmml"><mi id="S3.p2.3.m3.1.1.2" xref="S3.p2.3.m3.1.1.2.cmml">s</mi><mn id="S3.p2.3.m3.1.1.3" xref="S3.p2.3.m3.1.1.3.cmml">0</mn></msub><annotation-xml encoding="MathML-Content" id="S3.p2.3.m3.1b"><apply id="S3.p2.3.m3.1.1.cmml" xref="S3.p2.3.m3.1.1"><csymbol cd="ambiguous" id="S3.p2.3.m3.1.1.1.cmml" xref="S3.p2.3.m3.1.1">subscript</csymbol><ci id="S3.p2.3.m3.1.1.2.cmml" xref="S3.p2.3.m3.1.1.2">𝑠</ci><cn type="integer" id="S3.p2.3.m3.1.1.3.cmml" xref="S3.p2.3.m3.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p2.3.m3.1c">s_{0}</annotation></semantics></math> (containing the input) to each possible child state of <math id="S3.p2.4.m4.1" class="ltx_Math" alttext="s_{0}" display="inline"><semantics id="S3.p2.4.m4.1a"><msub id="S3.p2.4.m4.1.1" xref="S3.p2.4.m4.1.1.cmml"><mi id="S3.p2.4.m4.1.1.2" xref="S3.p2.4.m4.1.1.2.cmml">s</mi><mn id="S3.p2.4.m4.1.1.3" xref="S3.p2.4.m4.1.1.3.cmml">0</mn></msub><annotation-xml encoding="MathML-Content" id="S3.p2.4.m4.1b"><apply id="S3.p2.4.m4.1.1.cmml" xref="S3.p2.4.m4.1.1"><csymbol cd="ambiguous" id="S3.p2.4.m4.1.1.1.cmml" xref="S3.p2.4.m4.1.1">subscript</csymbol><ci id="S3.p2.4.m4.1.1.2.cmml" xref="S3.p2.4.m4.1.1.2">𝑠</ci><cn type="integer" id="S3.p2.4.m4.1.1.3.cmml" xref="S3.p2.4.m4.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p2.4.m4.1c">s_{0}</annotation></semantics></math>, and so on until a leaf state (a state with no children) is reached. A correct path to the solution <math id="S3.p2.5.m5.1" class="ltx_Math" alttext="\mathcal{P}" display="inline"><semantics id="S3.p2.5.m5.1a"><mi class="ltx_font_mathcaligraphic" id="S3.p2.5.m5.1.1" xref="S3.p2.5.m5.1.1.cmml">𝒫</mi><annotation-xml encoding="MathML-Content" id="S3.p2.5.m5.1b"><ci id="S3.p2.5.m5.1.1.cmml" xref="S3.p2.5.m5.1.1">𝒫</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.p2.5.m5.1c">\mathcal{P}</annotation></semantics></math> is a sequence of states and actions <math id="S3.p2.6.m6.8" class="ltx_Math" alttext="(s_{0},a_{0},s_{1},a_{1},\dots,s_{g-1},a_{g-1},s_{g})" display="inline"><semantics id="S3.p2.6.m6.8a"><mrow id="S3.p2.6.m6.8.8.7" xref="S3.p2.6.m6.8.8.8.cmml"><mo stretchy="false" id="S3.p2.6.m6.8.8.7.8" xref="S3.p2.6.m6.8.8.8.cmml">(</mo><msub id="S3.p2.6.m6.2.2.1.1" xref="S3.p2.6.m6.2.2.1.1.cmml"><mi id="S3.p2.6.m6.2.2.1.1.2" xref="S3.p2.6.m6.2.2.1.1.2.cmml">s</mi><mn id="S3.p2.6.m6.2.2.1.1.3" xref="S3.p2.6.m6.2.2.1.1.3.cmml">0</mn></msub><mo id="S3.p2.6.m6.8.8.7.9" xref="S3.p2.6.m6.8.8.8.cmml">,</mo><msub id="S3.p2.6.m6.3.3.2.2" xref="S3.p2.6.m6.3.3.2.2.cmml"><mi id="S3.p2.6.m6.3.3.2.2.2" xref="S3.p2.6.m6.3.3.2.2.2.cmml">a</mi><mn id="S3.p2.6.m6.3.3.2.2.3" xref="S3.p2.6.m6.3.3.2.2.3.cmml">0</mn></msub><mo id="S3.p2.6.m6.8.8.7.10" xref="S3.p2.6.m6.8.8.8.cmml">,</mo><msub id="S3.p2.6.m6.4.4.3.3" xref="S3.p2.6.m6.4.4.3.3.cmml"><mi id="S3.p2.6.m6.4.4.3.3.2" xref="S3.p2.6.m6.4.4.3.3.2.cmml">s</mi><mn id="S3.p2.6.m6.4.4.3.3.3" xref="S3.p2.6.m6.4.4.3.3.3.cmml">1</mn></msub><mo id="S3.p2.6.m6.8.8.7.11" xref="S3.p2.6.m6.8.8.8.cmml">,</mo><msub id="S3.p2.6.m6.5.5.4.4" xref="S3.p2.6.m6.5.5.4.4.cmml"><mi id="S3.p2.6.m6.5.5.4.4.2" xref="S3.p2.6.m6.5.5.4.4.2.cmml">a</mi><mn id="S3.p2.6.m6.5.5.4.4.3" xref="S3.p2.6.m6.5.5.4.4.3.cmml">1</mn></msub><mo id="S3.p2.6.m6.8.8.7.12" xref="S3.p2.6.m6.8.8.8.cmml">,</mo><mi mathvariant="normal" id="S3.p2.6.m6.1.1" xref="S3.p2.6.m6.1.1.cmml">…</mi><mo id="S3.p2.6.m6.8.8.7.13" xref="S3.p2.6.m6.8.8.8.cmml">,</mo><msub id="S3.p2.6.m6.6.6.5.5" xref="S3.p2.6.m6.6.6.5.5.cmml"><mi id="S3.p2.6.m6.6.6.5.5.2" xref="S3.p2.6.m6.6.6.5.5.2.cmml">s</mi><mrow id="S3.p2.6.m6.6.6.5.5.3" xref="S3.p2.6.m6.6.6.5.5.3.cmml"><mi id="S3.p2.6.m6.6.6.5.5.3.2" xref="S3.p2.6.m6.6.6.5.5.3.2.cmml">g</mi><mo id="S3.p2.6.m6.6.6.5.5.3.1" xref="S3.p2.6.m6.6.6.5.5.3.1.cmml">−</mo><mn id="S3.p2.6.m6.6.6.5.5.3.3" xref="S3.p2.6.m6.6.6.5.5.3.3.cmml">1</mn></mrow></msub><mo id="S3.p2.6.m6.8.8.7.14" xref="S3.p2.6.m6.8.8.8.cmml">,</mo><msub id="S3.p2.6.m6.7.7.6.6" xref="S3.p2.6.m6.7.7.6.6.cmml"><mi id="S3.p2.6.m6.7.7.6.6.2" xref="S3.p2.6.m6.7.7.6.6.2.cmml">a</mi><mrow id="S3.p2.6.m6.7.7.6.6.3" xref="S3.p2.6.m6.7.7.6.6.3.cmml"><mi id="S3.p2.6.m6.7.7.6.6.3.2" xref="S3.p2.6.m6.7.7.6.6.3.2.cmml">g</mi><mo id="S3.p2.6.m6.7.7.6.6.3.1" xref="S3.p2.6.m6.7.7.6.6.3.1.cmml">−</mo><mn id="S3.p2.6.m6.7.7.6.6.3.3" xref="S3.p2.6.m6.7.7.6.6.3.3.cmml">1</mn></mrow></msub><mo id="S3.p2.6.m6.8.8.7.15" xref="S3.p2.6.m6.8.8.8.cmml">,</mo><msub id="S3.p2.6.m6.8.8.7.7" xref="S3.p2.6.m6.8.8.7.7.cmml"><mi id="S3.p2.6.m6.8.8.7.7.2" xref="S3.p2.6.m6.8.8.7.7.2.cmml">s</mi><mi id="S3.p2.6.m6.8.8.7.7.3" xref="S3.p2.6.m6.8.8.7.7.3.cmml">g</mi></msub><mo stretchy="false" id="S3.p2.6.m6.8.8.7.16" xref="S3.p2.6.m6.8.8.8.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.p2.6.m6.8b"><vector id="S3.p2.6.m6.8.8.8.cmml" xref="S3.p2.6.m6.8.8.7"><apply id="S3.p2.6.m6.2.2.1.1.cmml" xref="S3.p2.6.m6.2.2.1.1"><csymbol cd="ambiguous" id="S3.p2.6.m6.2.2.1.1.1.cmml" xref="S3.p2.6.m6.2.2.1.1">subscript</csymbol><ci id="S3.p2.6.m6.2.2.1.1.2.cmml" xref="S3.p2.6.m6.2.2.1.1.2">𝑠</ci><cn type="integer" id="S3.p2.6.m6.2.2.1.1.3.cmml" xref="S3.p2.6.m6.2.2.1.1.3">0</cn></apply><apply id="S3.p2.6.m6.3.3.2.2.cmml" xref="S3.p2.6.m6.3.3.2.2"><csymbol cd="ambiguous" id="S3.p2.6.m6.3.3.2.2.1.cmml" xref="S3.p2.6.m6.3.3.2.2">subscript</csymbol><ci id="S3.p2.6.m6.3.3.2.2.2.cmml" xref="S3.p2.6.m6.3.3.2.2.2">𝑎</ci><cn type="integer" id="S3.p2.6.m6.3.3.2.2.3.cmml" xref="S3.p2.6.m6.3.3.2.2.3">0</cn></apply><apply id="S3.p2.6.m6.4.4.3.3.cmml" xref="S3.p2.6.m6.4.4.3.3"><csymbol cd="ambiguous" id="S3.p2.6.m6.4.4.3.3.1.cmml" xref="S3.p2.6.m6.4.4.3.3">subscript</csymbol><ci id="S3.p2.6.m6.4.4.3.3.2.cmml" xref="S3.p2.6.m6.4.4.3.3.2">𝑠</ci><cn type="integer" id="S3.p2.6.m6.4.4.3.3.3.cmml" xref="S3.p2.6.m6.4.4.3.3.3">1</cn></apply><apply id="S3.p2.6.m6.5.5.4.4.cmml" xref="S3.p2.6.m6.5.5.4.4"><csymbol cd="ambiguous" id="S3.p2.6.m6.5.5.4.4.1.cmml" xref="S3.p2.6.m6.5.5.4.4">subscript</csymbol><ci id="S3.p2.6.m6.5.5.4.4.2.cmml" xref="S3.p2.6.m6.5.5.4.4.2">𝑎</ci><cn type="integer" id="S3.p2.6.m6.5.5.4.4.3.cmml" xref="S3.p2.6.m6.5.5.4.4.3">1</cn></apply><ci id="S3.p2.6.m6.1.1.cmml" xref="S3.p2.6.m6.1.1">…</ci><apply id="S3.p2.6.m6.6.6.5.5.cmml" xref="S3.p2.6.m6.6.6.5.5"><csymbol cd="ambiguous" id="S3.p2.6.m6.6.6.5.5.1.cmml" xref="S3.p2.6.m6.6.6.5.5">subscript</csymbol><ci id="S3.p2.6.m6.6.6.5.5.2.cmml" xref="S3.p2.6.m6.6.6.5.5.2">𝑠</ci><apply id="S3.p2.6.m6.6.6.5.5.3.cmml" xref="S3.p2.6.m6.6.6.5.5.3"><minus id="S3.p2.6.m6.6.6.5.5.3.1.cmml" xref="S3.p2.6.m6.6.6.5.5.3.1"></minus><ci id="S3.p2.6.m6.6.6.5.5.3.2.cmml" xref="S3.p2.6.m6.6.6.5.5.3.2">𝑔</ci><cn type="integer" id="S3.p2.6.m6.6.6.5.5.3.3.cmml" xref="S3.p2.6.m6.6.6.5.5.3.3">1</cn></apply></apply><apply id="S3.p2.6.m6.7.7.6.6.cmml" xref="S3.p2.6.m6.7.7.6.6"><csymbol cd="ambiguous" id="S3.p2.6.m6.7.7.6.6.1.cmml" xref="S3.p2.6.m6.7.7.6.6">subscript</csymbol><ci id="S3.p2.6.m6.7.7.6.6.2.cmml" xref="S3.p2.6.m6.7.7.6.6.2">𝑎</ci><apply id="S3.p2.6.m6.7.7.6.6.3.cmml" xref="S3.p2.6.m6.7.7.6.6.3"><minus id="S3.p2.6.m6.7.7.6.6.3.1.cmml" xref="S3.p2.6.m6.7.7.6.6.3.1"></minus><ci id="S3.p2.6.m6.7.7.6.6.3.2.cmml" xref="S3.p2.6.m6.7.7.6.6.3.2">𝑔</ci><cn type="integer" id="S3.p2.6.m6.7.7.6.6.3.3.cmml" xref="S3.p2.6.m6.7.7.6.6.3.3">1</cn></apply></apply><apply id="S3.p2.6.m6.8.8.7.7.cmml" xref="S3.p2.6.m6.8.8.7.7"><csymbol cd="ambiguous" id="S3.p2.6.m6.8.8.7.7.1.cmml" xref="S3.p2.6.m6.8.8.7.7">subscript</csymbol><ci id="S3.p2.6.m6.8.8.7.7.2.cmml" xref="S3.p2.6.m6.8.8.7.7.2">𝑠</ci><ci id="S3.p2.6.m6.8.8.7.7.3.cmml" xref="S3.p2.6.m6.8.8.7.7.3">𝑔</ci></apply></vector></annotation-xml><annotation encoding="application/x-tex" id="S3.p2.6.m6.8c">(s_{0},a_{0},s_{1},a_{1},\dots,s_{g-1},a_{g-1},s_{g})</annotation></semantics></math> in the search tree, where each successive state <math id="S3.p2.7.m7.1" class="ltx_Math" alttext="s_{i+1}" display="inline"><semantics id="S3.p2.7.m7.1a"><msub id="S3.p2.7.m7.1.1" xref="S3.p2.7.m7.1.1.cmml"><mi id="S3.p2.7.m7.1.1.2" xref="S3.p2.7.m7.1.1.2.cmml">s</mi><mrow id="S3.p2.7.m7.1.1.3" xref="S3.p2.7.m7.1.1.3.cmml"><mi id="S3.p2.7.m7.1.1.3.2" xref="S3.p2.7.m7.1.1.3.2.cmml">i</mi><mo id="S3.p2.7.m7.1.1.3.1" xref="S3.p2.7.m7.1.1.3.1.cmml">+</mo><mn id="S3.p2.7.m7.1.1.3.3" xref="S3.p2.7.m7.1.1.3.3.cmml">1</mn></mrow></msub><annotation-xml encoding="MathML-Content" id="S3.p2.7.m7.1b"><apply id="S3.p2.7.m7.1.1.cmml" xref="S3.p2.7.m7.1.1"><csymbol cd="ambiguous" id="S3.p2.7.m7.1.1.1.cmml" xref="S3.p2.7.m7.1.1">subscript</csymbol><ci id="S3.p2.7.m7.1.1.2.cmml" xref="S3.p2.7.m7.1.1.2">𝑠</ci><apply id="S3.p2.7.m7.1.1.3.cmml" xref="S3.p2.7.m7.1.1.3"><plus id="S3.p2.7.m7.1.1.3.1.cmml" xref="S3.p2.7.m7.1.1.3.1"></plus><ci id="S3.p2.7.m7.1.1.3.2.cmml" xref="S3.p2.7.m7.1.1.3.2">𝑖</ci><cn type="integer" id="S3.p2.7.m7.1.1.3.3.cmml" xref="S3.p2.7.m7.1.1.3.3">1</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p2.7.m7.1c">s_{i+1}</annotation></semantics></math> is obtained by applying a valid action <math id="S3.p2.8.m8.1" class="ltx_Math" alttext="a_{i}\in\mathcal{A}" display="inline"><semantics id="S3.p2.8.m8.1a"><mrow id="S3.p2.8.m8.1.1" xref="S3.p2.8.m8.1.1.cmml"><msub id="S3.p2.8.m8.1.1.2" xref="S3.p2.8.m8.1.1.2.cmml"><mi id="S3.p2.8.m8.1.1.2.2" xref="S3.p2.8.m8.1.1.2.2.cmml">a</mi><mi id="S3.p2.8.m8.1.1.2.3" xref="S3.p2.8.m8.1.1.2.3.cmml">i</mi></msub><mo id="S3.p2.8.m8.1.1.1" xref="S3.p2.8.m8.1.1.1.cmml">∈</mo><mi class="ltx_font_mathcaligraphic" id="S3.p2.8.m8.1.1.3" xref="S3.p2.8.m8.1.1.3.cmml">𝒜</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.p2.8.m8.1b"><apply id="S3.p2.8.m8.1.1.cmml" xref="S3.p2.8.m8.1.1"><in id="S3.p2.8.m8.1.1.1.cmml" xref="S3.p2.8.m8.1.1.1"></in><apply id="S3.p2.8.m8.1.1.2.cmml" xref="S3.p2.8.m8.1.1.2"><csymbol cd="ambiguous" id="S3.p2.8.m8.1.1.2.1.cmml" xref="S3.p2.8.m8.1.1.2">subscript</csymbol><ci id="S3.p2.8.m8.1.1.2.2.cmml" xref="S3.p2.8.m8.1.1.2.2">𝑎</ci><ci id="S3.p2.8.m8.1.1.2.3.cmml" xref="S3.p2.8.m8.1.1.2.3">𝑖</ci></apply><ci id="S3.p2.8.m8.1.1.3.cmml" xref="S3.p2.8.m8.1.1.3">𝒜</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p2.8.m8.1c">a_{i}\in\mathcal{A}</annotation></semantics></math> to the previous state <math id="S3.p2.9.m9.1" class="ltx_Math" alttext="s_{i}" display="inline"><semantics id="S3.p2.9.m9.1a"><msub id="S3.p2.9.m9.1.1" xref="S3.p2.9.m9.1.1.cmml"><mi id="S3.p2.9.m9.1.1.2" xref="S3.p2.9.m9.1.1.2.cmml">s</mi><mi id="S3.p2.9.m9.1.1.3" xref="S3.p2.9.m9.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.p2.9.m9.1b"><apply id="S3.p2.9.m9.1.1.cmml" xref="S3.p2.9.m9.1.1"><csymbol cd="ambiguous" id="S3.p2.9.m9.1.1.1.cmml" xref="S3.p2.9.m9.1.1">subscript</csymbol><ci id="S3.p2.9.m9.1.1.2.cmml" xref="S3.p2.9.m9.1.1.2">𝑠</ci><ci id="S3.p2.9.m9.1.1.3.cmml" xref="S3.p2.9.m9.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p2.9.m9.1c">s_{i}</annotation></semantics></math>, i.e., <math id="S3.p2.10.m10.2" class="ltx_Math" alttext="s_{i+1}=T(s_{i},a_{i})" display="inline"><semantics id="S3.p2.10.m10.2a"><mrow id="S3.p2.10.m10.2.2" xref="S3.p2.10.m10.2.2.cmml"><msub id="S3.p2.10.m10.2.2.4" xref="S3.p2.10.m10.2.2.4.cmml"><mi id="S3.p2.10.m10.2.2.4.2" xref="S3.p2.10.m10.2.2.4.2.cmml">s</mi><mrow id="S3.p2.10.m10.2.2.4.3" xref="S3.p2.10.m10.2.2.4.3.cmml"><mi id="S3.p2.10.m10.2.2.4.3.2" xref="S3.p2.10.m10.2.2.4.3.2.cmml">i</mi><mo id="S3.p2.10.m10.2.2.4.3.1" xref="S3.p2.10.m10.2.2.4.3.1.cmml">+</mo><mn id="S3.p2.10.m10.2.2.4.3.3" xref="S3.p2.10.m10.2.2.4.3.3.cmml">1</mn></mrow></msub><mo id="S3.p2.10.m10.2.2.3" xref="S3.p2.10.m10.2.2.3.cmml">=</mo><mrow id="S3.p2.10.m10.2.2.2" xref="S3.p2.10.m10.2.2.2.cmml"><mi id="S3.p2.10.m10.2.2.2.4" xref="S3.p2.10.m10.2.2.2.4.cmml">T</mi><mo lspace="0em" rspace="0em" id="S3.p2.10.m10.2.2.2.3" xref="S3.p2.10.m10.2.2.2.3.cmml">​</mo><mrow id="S3.p2.10.m10.2.2.2.2.2" xref="S3.p2.10.m10.2.2.2.2.3.cmml"><mo stretchy="false" id="S3.p2.10.m10.2.2.2.2.2.3" xref="S3.p2.10.m10.2.2.2.2.3.cmml">(</mo><msub id="S3.p2.10.m10.1.1.1.1.1.1" xref="S3.p2.10.m10.1.1.1.1.1.1.cmml"><mi id="S3.p2.10.m10.1.1.1.1.1.1.2" xref="S3.p2.10.m10.1.1.1.1.1.1.2.cmml">s</mi><mi id="S3.p2.10.m10.1.1.1.1.1.1.3" xref="S3.p2.10.m10.1.1.1.1.1.1.3.cmml">i</mi></msub><mo id="S3.p2.10.m10.2.2.2.2.2.4" xref="S3.p2.10.m10.2.2.2.2.3.cmml">,</mo><msub id="S3.p2.10.m10.2.2.2.2.2.2" xref="S3.p2.10.m10.2.2.2.2.2.2.cmml"><mi id="S3.p2.10.m10.2.2.2.2.2.2.2" xref="S3.p2.10.m10.2.2.2.2.2.2.2.cmml">a</mi><mi id="S3.p2.10.m10.2.2.2.2.2.2.3" xref="S3.p2.10.m10.2.2.2.2.2.2.3.cmml">i</mi></msub><mo stretchy="false" id="S3.p2.10.m10.2.2.2.2.2.5" xref="S3.p2.10.m10.2.2.2.2.3.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.p2.10.m10.2b"><apply id="S3.p2.10.m10.2.2.cmml" xref="S3.p2.10.m10.2.2"><eq id="S3.p2.10.m10.2.2.3.cmml" xref="S3.p2.10.m10.2.2.3"></eq><apply id="S3.p2.10.m10.2.2.4.cmml" xref="S3.p2.10.m10.2.2.4"><csymbol cd="ambiguous" id="S3.p2.10.m10.2.2.4.1.cmml" xref="S3.p2.10.m10.2.2.4">subscript</csymbol><ci id="S3.p2.10.m10.2.2.4.2.cmml" xref="S3.p2.10.m10.2.2.4.2">𝑠</ci><apply id="S3.p2.10.m10.2.2.4.3.cmml" xref="S3.p2.10.m10.2.2.4.3"><plus id="S3.p2.10.m10.2.2.4.3.1.cmml" xref="S3.p2.10.m10.2.2.4.3.1"></plus><ci id="S3.p2.10.m10.2.2.4.3.2.cmml" xref="S3.p2.10.m10.2.2.4.3.2">𝑖</ci><cn type="integer" id="S3.p2.10.m10.2.2.4.3.3.cmml" xref="S3.p2.10.m10.2.2.4.3.3">1</cn></apply></apply><apply id="S3.p2.10.m10.2.2.2.cmml" xref="S3.p2.10.m10.2.2.2"><times id="S3.p2.10.m10.2.2.2.3.cmml" xref="S3.p2.10.m10.2.2.2.3"></times><ci id="S3.p2.10.m10.2.2.2.4.cmml" xref="S3.p2.10.m10.2.2.2.4">𝑇</ci><interval closure="open" id="S3.p2.10.m10.2.2.2.2.3.cmml" xref="S3.p2.10.m10.2.2.2.2.2"><apply id="S3.p2.10.m10.1.1.1.1.1.1.cmml" xref="S3.p2.10.m10.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.p2.10.m10.1.1.1.1.1.1.1.cmml" xref="S3.p2.10.m10.1.1.1.1.1.1">subscript</csymbol><ci id="S3.p2.10.m10.1.1.1.1.1.1.2.cmml" xref="S3.p2.10.m10.1.1.1.1.1.1.2">𝑠</ci><ci id="S3.p2.10.m10.1.1.1.1.1.1.3.cmml" xref="S3.p2.10.m10.1.1.1.1.1.1.3">𝑖</ci></apply><apply id="S3.p2.10.m10.2.2.2.2.2.2.cmml" xref="S3.p2.10.m10.2.2.2.2.2.2"><csymbol cd="ambiguous" id="S3.p2.10.m10.2.2.2.2.2.2.1.cmml" xref="S3.p2.10.m10.2.2.2.2.2.2">subscript</csymbol><ci id="S3.p2.10.m10.2.2.2.2.2.2.2.cmml" xref="S3.p2.10.m10.2.2.2.2.2.2.2">𝑎</ci><ci id="S3.p2.10.m10.2.2.2.2.2.2.3.cmml" xref="S3.p2.10.m10.2.2.2.2.2.2.3">𝑖</ci></apply></interval></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p2.10.m10.2c">s_{i+1}=T(s_{i},a_{i})</annotation></semantics></math>, and the final state in the sequence is the goal state <math id="S3.p2.11.m11.1" class="ltx_Math" alttext="s_{g}" display="inline"><semantics id="S3.p2.11.m11.1a"><msub id="S3.p2.11.m11.1.1" xref="S3.p2.11.m11.1.1.cmml"><mi id="S3.p2.11.m11.1.1.2" xref="S3.p2.11.m11.1.1.2.cmml">s</mi><mi id="S3.p2.11.m11.1.1.3" xref="S3.p2.11.m11.1.1.3.cmml">g</mi></msub><annotation-xml encoding="MathML-Content" id="S3.p2.11.m11.1b"><apply id="S3.p2.11.m11.1.1.cmml" xref="S3.p2.11.m11.1.1"><csymbol cd="ambiguous" id="S3.p2.11.m11.1.1.1.cmml" xref="S3.p2.11.m11.1.1">subscript</csymbol><ci id="S3.p2.11.m11.1.1.2.cmml" xref="S3.p2.11.m11.1.1.2">𝑠</ci><ci id="S3.p2.11.m11.1.1.3.cmml" xref="S3.p2.11.m11.1.1.3">𝑔</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p2.11.m11.1c">s_{g}</annotation></semantics></math> (see <a href="#S3.F2" title="Figure 2 ‣ 3 A Language for Search ‣ Stream of Search (SoS): Learning to Search in Language" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Fig. 2</span></a>).</p>
</div>
<figure id="S3.F2" class="ltx_figure"><img src="/html/2404.03683/assets/x2.png" id="S3.F2.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="338" height="113" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 2: </span>A visualization of how a search process is translated into a stream of search with a language for search. (left) The search process represented as a tree, with different states and operations. The colored states represent the search trajectory <math id="S3.F2.4.m1.1" class="ltx_Math" alttext="\mathcal{T}" display="inline"><semantics id="S3.F2.4.m1.1b"><mi class="ltx_font_mathcaligraphic" id="S3.F2.4.m1.1.1" xref="S3.F2.4.m1.1.1.cmml">𝒯</mi><annotation-xml encoding="MathML-Content" id="S3.F2.4.m1.1c"><ci id="S3.F2.4.m1.1.1.cmml" xref="S3.F2.4.m1.1.1">𝒯</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.F2.4.m1.1d">\mathcal{T}</annotation></semantics></math>, the green states represent the correct path to the goal <math id="S3.F2.5.m2.1" class="ltx_Math" alttext="\mathcal{P}" display="inline"><semantics id="S3.F2.5.m2.1b"><mi class="ltx_font_mathcaligraphic" id="S3.F2.5.m2.1.1" xref="S3.F2.5.m2.1.1.cmml">𝒫</mi><annotation-xml encoding="MathML-Content" id="S3.F2.5.m2.1c"><ci id="S3.F2.5.m2.1.1.cmml" xref="S3.F2.5.m2.1.1">𝒫</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.F2.5.m2.1d">\mathcal{P}</annotation></semantics></math> and the arrows represent transitions between the states. The black circles represent unexplored states. (center) The search process serialized as text to create a stream of search. The labels specify the different components of the process. See <a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ Stream of Search (SoS): Learning to Search in Language" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Fig. 1</span></a>b for how this is realized in Countdown. (right) The optimal path, <math id="S3.F2.6.m3.1" class="ltx_Math" alttext="\mathcal{P}" display="inline"><semantics id="S3.F2.6.m3.1b"><mi class="ltx_font_mathcaligraphic" id="S3.F2.6.m3.1.1" xref="S3.F2.6.m3.1.1.cmml">𝒫</mi><annotation-xml encoding="MathML-Content" id="S3.F2.6.m3.1c"><ci id="S3.F2.6.m3.1.1.cmml" xref="S3.F2.6.m3.1.1">𝒫</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.F2.6.m3.1d">\mathcal{P}</annotation></semantics></math>, to the goal state. Backtracking, exploration and the messy process of search are excluded.
</figcaption>
</figure>
<div id="S3.p3" class="ltx_para ltx_noindent">
<p id="S3.p3.1" class="ltx_p">We are interested in the process of search, which corresponds to moving around the search tree until a solution is found. To represent this process, we propose a vocabulary of primitive operations<span id="footnote2" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">2</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">2</sup><span class="ltx_tag ltx_tag_note">2</span>This is a non-exhaustive list, we choose to represent the most salient operations here.</span></span></span> that can be used to define components of different search algorithms. This is in spirit similar to defining a domain-specific language for planning <cite class="ltx_cite ltx_citemacro_citep">(Fikes &amp; Nilsson, <a href="#bib.bib7" title="" class="ltx_ref">1971</a>)</cite>.</p>
<ul id="S3.I1" class="ltx_itemize">
<li id="S3.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S3.I1.i1.p1" class="ltx_para">
<p id="S3.I1.i1.p1.2" class="ltx_p"><span id="S3.I1.i1.p1.2.1" class="ltx_text ltx_font_bold">Current State:</span> <math id="S3.I1.i1.p1.1.m1.1" class="ltx_Math" alttext="s_{c}" display="inline"><semantics id="S3.I1.i1.p1.1.m1.1a"><msub id="S3.I1.i1.p1.1.m1.1.1" xref="S3.I1.i1.p1.1.m1.1.1.cmml"><mi id="S3.I1.i1.p1.1.m1.1.1.2" xref="S3.I1.i1.p1.1.m1.1.1.2.cmml">s</mi><mi id="S3.I1.i1.p1.1.m1.1.1.3" xref="S3.I1.i1.p1.1.m1.1.1.3.cmml">c</mi></msub><annotation-xml encoding="MathML-Content" id="S3.I1.i1.p1.1.m1.1b"><apply id="S3.I1.i1.p1.1.m1.1.1.cmml" xref="S3.I1.i1.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S3.I1.i1.p1.1.m1.1.1.1.cmml" xref="S3.I1.i1.p1.1.m1.1.1">subscript</csymbol><ci id="S3.I1.i1.p1.1.m1.1.1.2.cmml" xref="S3.I1.i1.p1.1.m1.1.1.2">𝑠</ci><ci id="S3.I1.i1.p1.1.m1.1.1.3.cmml" xref="S3.I1.i1.p1.1.m1.1.1.3">𝑐</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.I1.i1.p1.1.m1.1c">s_{c}</annotation></semantics></math>, The state <math id="S3.I1.i1.p1.2.m2.1" class="ltx_Math" alttext="s_{c}" display="inline"><semantics id="S3.I1.i1.p1.2.m2.1a"><msub id="S3.I1.i1.p1.2.m2.1.1" xref="S3.I1.i1.p1.2.m2.1.1.cmml"><mi id="S3.I1.i1.p1.2.m2.1.1.2" xref="S3.I1.i1.p1.2.m2.1.1.2.cmml">s</mi><mi id="S3.I1.i1.p1.2.m2.1.1.3" xref="S3.I1.i1.p1.2.m2.1.1.3.cmml">c</mi></msub><annotation-xml encoding="MathML-Content" id="S3.I1.i1.p1.2.m2.1b"><apply id="S3.I1.i1.p1.2.m2.1.1.cmml" xref="S3.I1.i1.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S3.I1.i1.p1.2.m2.1.1.1.cmml" xref="S3.I1.i1.p1.2.m2.1.1">subscript</csymbol><ci id="S3.I1.i1.p1.2.m2.1.1.2.cmml" xref="S3.I1.i1.p1.2.m2.1.1.2">𝑠</ci><ci id="S3.I1.i1.p1.2.m2.1.1.3.cmml" xref="S3.I1.i1.p1.2.m2.1.1.3">𝑐</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.I1.i1.p1.2.m2.1c">s_{c}</annotation></semantics></math> that is being explored.</p>
</div>
</li>
<li id="S3.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S3.I1.i2.p1" class="ltx_para">
<p id="S3.I1.i2.p1.2" class="ltx_p"><span id="S3.I1.i2.p1.2.1" class="ltx_text ltx_font_bold">Goal State:</span> <math id="S3.I1.i2.p1.1.m1.1" class="ltx_Math" alttext="s_{g}" display="inline"><semantics id="S3.I1.i2.p1.1.m1.1a"><msub id="S3.I1.i2.p1.1.m1.1.1" xref="S3.I1.i2.p1.1.m1.1.1.cmml"><mi id="S3.I1.i2.p1.1.m1.1.1.2" xref="S3.I1.i2.p1.1.m1.1.1.2.cmml">s</mi><mi id="S3.I1.i2.p1.1.m1.1.1.3" xref="S3.I1.i2.p1.1.m1.1.1.3.cmml">g</mi></msub><annotation-xml encoding="MathML-Content" id="S3.I1.i2.p1.1.m1.1b"><apply id="S3.I1.i2.p1.1.m1.1.1.cmml" xref="S3.I1.i2.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S3.I1.i2.p1.1.m1.1.1.1.cmml" xref="S3.I1.i2.p1.1.m1.1.1">subscript</csymbol><ci id="S3.I1.i2.p1.1.m1.1.1.2.cmml" xref="S3.I1.i2.p1.1.m1.1.1.2">𝑠</ci><ci id="S3.I1.i2.p1.1.m1.1.1.3.cmml" xref="S3.I1.i2.p1.1.m1.1.1.3">𝑔</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.I1.i2.p1.1.m1.1c">s_{g}</annotation></semantics></math>, The state <math id="S3.I1.i2.p1.2.m2.1" class="ltx_Math" alttext="s_{g}" display="inline"><semantics id="S3.I1.i2.p1.2.m2.1a"><msub id="S3.I1.i2.p1.2.m2.1.1" xref="S3.I1.i2.p1.2.m2.1.1.cmml"><mi id="S3.I1.i2.p1.2.m2.1.1.2" xref="S3.I1.i2.p1.2.m2.1.1.2.cmml">s</mi><mi id="S3.I1.i2.p1.2.m2.1.1.3" xref="S3.I1.i2.p1.2.m2.1.1.3.cmml">g</mi></msub><annotation-xml encoding="MathML-Content" id="S3.I1.i2.p1.2.m2.1b"><apply id="S3.I1.i2.p1.2.m2.1.1.cmml" xref="S3.I1.i2.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S3.I1.i2.p1.2.m2.1.1.1.cmml" xref="S3.I1.i2.p1.2.m2.1.1">subscript</csymbol><ci id="S3.I1.i2.p1.2.m2.1.1.2.cmml" xref="S3.I1.i2.p1.2.m2.1.1.2">𝑠</ci><ci id="S3.I1.i2.p1.2.m2.1.1.3.cmml" xref="S3.I1.i2.p1.2.m2.1.1.3">𝑔</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.I1.i2.p1.2.m2.1c">s_{g}</annotation></semantics></math> that is the target.</p>
</div>
</li>
<li id="S3.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S3.I1.i3.p1" class="ltx_para">
<p id="S3.I1.i3.p1.1" class="ltx_p"><span id="S3.I1.i3.p1.1.1" class="ltx_text ltx_font_bold">State Queue:</span> <math id="S3.I1.i3.p1.1.m1.1" class="ltx_Math" alttext="S_{q}" display="inline"><semantics id="S3.I1.i3.p1.1.m1.1a"><msub id="S3.I1.i3.p1.1.m1.1.1" xref="S3.I1.i3.p1.1.m1.1.1.cmml"><mi id="S3.I1.i3.p1.1.m1.1.1.2" xref="S3.I1.i3.p1.1.m1.1.1.2.cmml">S</mi><mi id="S3.I1.i3.p1.1.m1.1.1.3" xref="S3.I1.i3.p1.1.m1.1.1.3.cmml">q</mi></msub><annotation-xml encoding="MathML-Content" id="S3.I1.i3.p1.1.m1.1b"><apply id="S3.I1.i3.p1.1.m1.1.1.cmml" xref="S3.I1.i3.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S3.I1.i3.p1.1.m1.1.1.1.cmml" xref="S3.I1.i3.p1.1.m1.1.1">subscript</csymbol><ci id="S3.I1.i3.p1.1.m1.1.1.2.cmml" xref="S3.I1.i3.p1.1.m1.1.1.2">𝑆</ci><ci id="S3.I1.i3.p1.1.m1.1.1.3.cmml" xref="S3.I1.i3.p1.1.m1.1.1.3">𝑞</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.I1.i3.p1.1.m1.1c">S_{q}</annotation></semantics></math> The states at the ‘frontier’ of the the the trajectory that haven’t been explored yet.</p>
</div>
</li>
<li id="S3.I1.i4" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S3.I1.i4.p1" class="ltx_para">
<p id="S3.I1.i4.p1.3" class="ltx_p"><span id="S3.I1.i4.p1.3.1" class="ltx_text ltx_font_bold">State Expansion Function:</span> <math id="S3.I1.i4.p1.1.m1.1" class="ltx_Math" alttext="SE:\mathcal{S}\rightarrow\mathcal{S}" display="inline"><semantics id="S3.I1.i4.p1.1.m1.1a"><mrow id="S3.I1.i4.p1.1.m1.1.1" xref="S3.I1.i4.p1.1.m1.1.1.cmml"><mrow id="S3.I1.i4.p1.1.m1.1.1.2" xref="S3.I1.i4.p1.1.m1.1.1.2.cmml"><mi id="S3.I1.i4.p1.1.m1.1.1.2.2" xref="S3.I1.i4.p1.1.m1.1.1.2.2.cmml">S</mi><mo lspace="0em" rspace="0em" id="S3.I1.i4.p1.1.m1.1.1.2.1" xref="S3.I1.i4.p1.1.m1.1.1.2.1.cmml">​</mo><mi id="S3.I1.i4.p1.1.m1.1.1.2.3" xref="S3.I1.i4.p1.1.m1.1.1.2.3.cmml">E</mi></mrow><mo lspace="0.278em" rspace="0.278em" id="S3.I1.i4.p1.1.m1.1.1.1" xref="S3.I1.i4.p1.1.m1.1.1.1.cmml">:</mo><mrow id="S3.I1.i4.p1.1.m1.1.1.3" xref="S3.I1.i4.p1.1.m1.1.1.3.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.I1.i4.p1.1.m1.1.1.3.2" xref="S3.I1.i4.p1.1.m1.1.1.3.2.cmml">𝒮</mi><mo stretchy="false" id="S3.I1.i4.p1.1.m1.1.1.3.1" xref="S3.I1.i4.p1.1.m1.1.1.3.1.cmml">→</mo><mi class="ltx_font_mathcaligraphic" id="S3.I1.i4.p1.1.m1.1.1.3.3" xref="S3.I1.i4.p1.1.m1.1.1.3.3.cmml">𝒮</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.I1.i4.p1.1.m1.1b"><apply id="S3.I1.i4.p1.1.m1.1.1.cmml" xref="S3.I1.i4.p1.1.m1.1.1"><ci id="S3.I1.i4.p1.1.m1.1.1.1.cmml" xref="S3.I1.i4.p1.1.m1.1.1.1">:</ci><apply id="S3.I1.i4.p1.1.m1.1.1.2.cmml" xref="S3.I1.i4.p1.1.m1.1.1.2"><times id="S3.I1.i4.p1.1.m1.1.1.2.1.cmml" xref="S3.I1.i4.p1.1.m1.1.1.2.1"></times><ci id="S3.I1.i4.p1.1.m1.1.1.2.2.cmml" xref="S3.I1.i4.p1.1.m1.1.1.2.2">𝑆</ci><ci id="S3.I1.i4.p1.1.m1.1.1.2.3.cmml" xref="S3.I1.i4.p1.1.m1.1.1.2.3">𝐸</ci></apply><apply id="S3.I1.i4.p1.1.m1.1.1.3.cmml" xref="S3.I1.i4.p1.1.m1.1.1.3"><ci id="S3.I1.i4.p1.1.m1.1.1.3.1.cmml" xref="S3.I1.i4.p1.1.m1.1.1.3.1">→</ci><ci id="S3.I1.i4.p1.1.m1.1.1.3.2.cmml" xref="S3.I1.i4.p1.1.m1.1.1.3.2">𝒮</ci><ci id="S3.I1.i4.p1.1.m1.1.1.3.3.cmml" xref="S3.I1.i4.p1.1.m1.1.1.3.3">𝒮</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.I1.i4.p1.1.m1.1c">SE:\mathcal{S}\rightarrow\mathcal{S}</annotation></semantics></math> , Explore a state adjacent to the current state <math id="S3.I1.i4.p1.2.m2.1" class="ltx_Math" alttext="s_{c}" display="inline"><semantics id="S3.I1.i4.p1.2.m2.1a"><msub id="S3.I1.i4.p1.2.m2.1.1" xref="S3.I1.i4.p1.2.m2.1.1.cmml"><mi id="S3.I1.i4.p1.2.m2.1.1.2" xref="S3.I1.i4.p1.2.m2.1.1.2.cmml">s</mi><mi id="S3.I1.i4.p1.2.m2.1.1.3" xref="S3.I1.i4.p1.2.m2.1.1.3.cmml">c</mi></msub><annotation-xml encoding="MathML-Content" id="S3.I1.i4.p1.2.m2.1b"><apply id="S3.I1.i4.p1.2.m2.1.1.cmml" xref="S3.I1.i4.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S3.I1.i4.p1.2.m2.1.1.1.cmml" xref="S3.I1.i4.p1.2.m2.1.1">subscript</csymbol><ci id="S3.I1.i4.p1.2.m2.1.1.2.cmml" xref="S3.I1.i4.p1.2.m2.1.1.2">𝑠</ci><ci id="S3.I1.i4.p1.2.m2.1.1.3.cmml" xref="S3.I1.i4.p1.2.m2.1.1.3">𝑐</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.I1.i4.p1.2.m2.1c">s_{c}</annotation></semantics></math> based on a transition function <math id="S3.I1.i4.p1.3.m3.1" class="ltx_Math" alttext="T" display="inline"><semantics id="S3.I1.i4.p1.3.m3.1a"><mi id="S3.I1.i4.p1.3.m3.1.1" xref="S3.I1.i4.p1.3.m3.1.1.cmml">T</mi><annotation-xml encoding="MathML-Content" id="S3.I1.i4.p1.3.m3.1b"><ci id="S3.I1.i4.p1.3.m3.1.1.cmml" xref="S3.I1.i4.p1.3.m3.1.1">𝑇</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.I1.i4.p1.3.m3.1c">T</annotation></semantics></math>.</p>
</div>
</li>
<li id="S3.I1.i5" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S3.I1.i5.p1" class="ltx_para">
<p id="S3.I1.i5.p1.1" class="ltx_p"><span id="S3.I1.i5.p1.1.1" class="ltx_text ltx_font_bold">Exploration Choice:</span> Choosing the order of states to explore following the state expansion. Eg: Should the breadth be explored first (BFS) or the depth (DFS) or any of the frontier states (<math id="S3.I1.i5.p1.1.m1.1" class="ltx_Math" alttext="A^{*}" display="inline"><semantics id="S3.I1.i5.p1.1.m1.1a"><msup id="S3.I1.i5.p1.1.m1.1.1" xref="S3.I1.i5.p1.1.m1.1.1.cmml"><mi id="S3.I1.i5.p1.1.m1.1.1.2" xref="S3.I1.i5.p1.1.m1.1.1.2.cmml">A</mi><mo id="S3.I1.i5.p1.1.m1.1.1.3" xref="S3.I1.i5.p1.1.m1.1.1.3.cmml">∗</mo></msup><annotation-xml encoding="MathML-Content" id="S3.I1.i5.p1.1.m1.1b"><apply id="S3.I1.i5.p1.1.m1.1.1.cmml" xref="S3.I1.i5.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S3.I1.i5.p1.1.m1.1.1.1.cmml" xref="S3.I1.i5.p1.1.m1.1.1">superscript</csymbol><ci id="S3.I1.i5.p1.1.m1.1.1.2.cmml" xref="S3.I1.i5.p1.1.m1.1.1.2">𝐴</ci><times id="S3.I1.i5.p1.1.m1.1.1.3.cmml" xref="S3.I1.i5.p1.1.m1.1.1.3"></times></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.I1.i5.p1.1.m1.1c">A^{*}</annotation></semantics></math>).</p>
</div>
</li>
<li id="S3.I1.i6" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S3.I1.i6.p1" class="ltx_para">
<p id="S3.I1.i6.p1.1" class="ltx_p"><span id="S3.I1.i6.p1.1.1" class="ltx_text ltx_font_bold">Pruning:</span> An action discarding states or subtrees that are unlikely to lead to a solution.</p>
</div>
</li>
<li id="S3.I1.i7" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S3.I1.i7.p1" class="ltx_para">
<p id="S3.I1.i7.p1.1" class="ltx_p"><span id="S3.I1.i7.p1.1.1" class="ltx_text ltx_font_bold">Backtracking:</span> An action to move between nodes that have been explored. This allows the algorithm to choose which state should be expanded next.</p>
</div>
</li>
<li id="S3.I1.i8" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S3.I1.i8.p1" class="ltx_para">
<p id="S3.I1.i8.p1.1" class="ltx_p"><span id="S3.I1.i8.p1.1.1" class="ltx_text ltx_font_bold">Goal Check:</span> An action to check if the current state is the goal state <math id="S3.I1.i8.p1.1.m1.1" class="ltx_math_unparsed" alttext="(s_{c}==s_{g})" display="inline"><semantics id="S3.I1.i8.p1.1.m1.1a"><mrow id="S3.I1.i8.p1.1.m1.1b"><mo stretchy="false" id="S3.I1.i8.p1.1.m1.1.1">(</mo><msub id="S3.I1.i8.p1.1.m1.1.2"><mi id="S3.I1.i8.p1.1.m1.1.2.2">s</mi><mi id="S3.I1.i8.p1.1.m1.1.2.3">c</mi></msub><mo rspace="0em" id="S3.I1.i8.p1.1.m1.1.3">=</mo><mo lspace="0em" id="S3.I1.i8.p1.1.m1.1.4">=</mo><msub id="S3.I1.i8.p1.1.m1.1.5"><mi id="S3.I1.i8.p1.1.m1.1.5.2">s</mi><mi id="S3.I1.i8.p1.1.m1.1.5.3">g</mi></msub><mo stretchy="false" id="S3.I1.i8.p1.1.m1.1.6">)</mo></mrow><annotation encoding="application/x-tex" id="S3.I1.i8.p1.1.m1.1c">(s_{c}==s_{g})</annotation></semantics></math>.</p>
</div>
</li>
<li id="S3.I1.i9" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S3.I1.i9.p1" class="ltx_para ltx_noindent">
<p id="S3.I1.i9.p1.3" class="ltx_p"><span id="S3.I1.i9.p1.3.1" class="ltx_text ltx_font_bold">Heuristic:</span> A function <math id="S3.I1.i9.p1.1.m1.1" class="ltx_Math" alttext="h\in\mathcal{H}:\mathcal{S}\times\mathcal{S}\rightarrow\mathbb{R}" display="inline"><semantics id="S3.I1.i9.p1.1.m1.1a"><mrow id="S3.I1.i9.p1.1.m1.1.1" xref="S3.I1.i9.p1.1.m1.1.1.cmml"><mrow id="S3.I1.i9.p1.1.m1.1.1.2" xref="S3.I1.i9.p1.1.m1.1.1.2.cmml"><mi id="S3.I1.i9.p1.1.m1.1.1.2.2" xref="S3.I1.i9.p1.1.m1.1.1.2.2.cmml">h</mi><mo id="S3.I1.i9.p1.1.m1.1.1.2.1" xref="S3.I1.i9.p1.1.m1.1.1.2.1.cmml">∈</mo><mi class="ltx_font_mathcaligraphic" id="S3.I1.i9.p1.1.m1.1.1.2.3" xref="S3.I1.i9.p1.1.m1.1.1.2.3.cmml">ℋ</mi></mrow><mo lspace="0.278em" rspace="0.278em" id="S3.I1.i9.p1.1.m1.1.1.1" xref="S3.I1.i9.p1.1.m1.1.1.1.cmml">:</mo><mrow id="S3.I1.i9.p1.1.m1.1.1.3" xref="S3.I1.i9.p1.1.m1.1.1.3.cmml"><mrow id="S3.I1.i9.p1.1.m1.1.1.3.2" xref="S3.I1.i9.p1.1.m1.1.1.3.2.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.I1.i9.p1.1.m1.1.1.3.2.2" xref="S3.I1.i9.p1.1.m1.1.1.3.2.2.cmml">𝒮</mi><mo lspace="0.222em" rspace="0.222em" id="S3.I1.i9.p1.1.m1.1.1.3.2.1" xref="S3.I1.i9.p1.1.m1.1.1.3.2.1.cmml">×</mo><mi class="ltx_font_mathcaligraphic" id="S3.I1.i9.p1.1.m1.1.1.3.2.3" xref="S3.I1.i9.p1.1.m1.1.1.3.2.3.cmml">𝒮</mi></mrow><mo stretchy="false" id="S3.I1.i9.p1.1.m1.1.1.3.1" xref="S3.I1.i9.p1.1.m1.1.1.3.1.cmml">→</mo><mi id="S3.I1.i9.p1.1.m1.1.1.3.3" xref="S3.I1.i9.p1.1.m1.1.1.3.3.cmml">ℝ</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.I1.i9.p1.1.m1.1b"><apply id="S3.I1.i9.p1.1.m1.1.1.cmml" xref="S3.I1.i9.p1.1.m1.1.1"><ci id="S3.I1.i9.p1.1.m1.1.1.1.cmml" xref="S3.I1.i9.p1.1.m1.1.1.1">:</ci><apply id="S3.I1.i9.p1.1.m1.1.1.2.cmml" xref="S3.I1.i9.p1.1.m1.1.1.2"><in id="S3.I1.i9.p1.1.m1.1.1.2.1.cmml" xref="S3.I1.i9.p1.1.m1.1.1.2.1"></in><ci id="S3.I1.i9.p1.1.m1.1.1.2.2.cmml" xref="S3.I1.i9.p1.1.m1.1.1.2.2">ℎ</ci><ci id="S3.I1.i9.p1.1.m1.1.1.2.3.cmml" xref="S3.I1.i9.p1.1.m1.1.1.2.3">ℋ</ci></apply><apply id="S3.I1.i9.p1.1.m1.1.1.3.cmml" xref="S3.I1.i9.p1.1.m1.1.1.3"><ci id="S3.I1.i9.p1.1.m1.1.1.3.1.cmml" xref="S3.I1.i9.p1.1.m1.1.1.3.1">→</ci><apply id="S3.I1.i9.p1.1.m1.1.1.3.2.cmml" xref="S3.I1.i9.p1.1.m1.1.1.3.2"><times id="S3.I1.i9.p1.1.m1.1.1.3.2.1.cmml" xref="S3.I1.i9.p1.1.m1.1.1.3.2.1"></times><ci id="S3.I1.i9.p1.1.m1.1.1.3.2.2.cmml" xref="S3.I1.i9.p1.1.m1.1.1.3.2.2">𝒮</ci><ci id="S3.I1.i9.p1.1.m1.1.1.3.2.3.cmml" xref="S3.I1.i9.p1.1.m1.1.1.3.2.3">𝒮</ci></apply><ci id="S3.I1.i9.p1.1.m1.1.1.3.3.cmml" xref="S3.I1.i9.p1.1.m1.1.1.3.3">ℝ</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.I1.i9.p1.1.m1.1c">h\in\mathcal{H}:\mathcal{S}\times\mathcal{S}\rightarrow\mathbb{R}</annotation></semantics></math> that approximates the distance of the current state <math id="S3.I1.i9.p1.2.m2.1" class="ltx_Math" alttext="s_{c}" display="inline"><semantics id="S3.I1.i9.p1.2.m2.1a"><msub id="S3.I1.i9.p1.2.m2.1.1" xref="S3.I1.i9.p1.2.m2.1.1.cmml"><mi id="S3.I1.i9.p1.2.m2.1.1.2" xref="S3.I1.i9.p1.2.m2.1.1.2.cmml">s</mi><mi id="S3.I1.i9.p1.2.m2.1.1.3" xref="S3.I1.i9.p1.2.m2.1.1.3.cmml">c</mi></msub><annotation-xml encoding="MathML-Content" id="S3.I1.i9.p1.2.m2.1b"><apply id="S3.I1.i9.p1.2.m2.1.1.cmml" xref="S3.I1.i9.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S3.I1.i9.p1.2.m2.1.1.1.cmml" xref="S3.I1.i9.p1.2.m2.1.1">subscript</csymbol><ci id="S3.I1.i9.p1.2.m2.1.1.2.cmml" xref="S3.I1.i9.p1.2.m2.1.1.2">𝑠</ci><ci id="S3.I1.i9.p1.2.m2.1.1.3.cmml" xref="S3.I1.i9.p1.2.m2.1.1.3">𝑐</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.I1.i9.p1.2.m2.1c">s_{c}</annotation></semantics></math> from the goal <math id="S3.I1.i9.p1.3.m3.1" class="ltx_Math" alttext="s_{g}" display="inline"><semantics id="S3.I1.i9.p1.3.m3.1a"><msub id="S3.I1.i9.p1.3.m3.1.1" xref="S3.I1.i9.p1.3.m3.1.1.cmml"><mi id="S3.I1.i9.p1.3.m3.1.1.2" xref="S3.I1.i9.p1.3.m3.1.1.2.cmml">s</mi><mi id="S3.I1.i9.p1.3.m3.1.1.3" xref="S3.I1.i9.p1.3.m3.1.1.3.cmml">g</mi></msub><annotation-xml encoding="MathML-Content" id="S3.I1.i9.p1.3.m3.1b"><apply id="S3.I1.i9.p1.3.m3.1.1.cmml" xref="S3.I1.i9.p1.3.m3.1.1"><csymbol cd="ambiguous" id="S3.I1.i9.p1.3.m3.1.1.1.cmml" xref="S3.I1.i9.p1.3.m3.1.1">subscript</csymbol><ci id="S3.I1.i9.p1.3.m3.1.1.2.cmml" xref="S3.I1.i9.p1.3.m3.1.1.2">𝑠</ci><ci id="S3.I1.i9.p1.3.m3.1.1.3.cmml" xref="S3.I1.i9.p1.3.m3.1.1.3">𝑔</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.I1.i9.p1.3.m3.1c">s_{g}</annotation></semantics></math>. This can be used to decide which states should be explored or pruned. The heuristic function serves as an approximation of the value function, guiding the search or decision-making process by estimating the desirability of states in terms of their distance to the goal.</p>
</div>
</li>
</ul>
</div>
<div id="S3.p4" class="ltx_para ltx_noindent">
<p id="S3.p4.1" class="ltx_p">Each of these operations can be left implicit, affecting how the trajectory unfolds, or made explicit in language as part of the search trajectory <math id="S3.p4.1.m1.1" class="ltx_Math" alttext="\mathcal{T}" display="inline"><semantics id="S3.p4.1.m1.1a"><mi class="ltx_font_mathcaligraphic" id="S3.p4.1.m1.1.1" xref="S3.p4.1.m1.1.1.cmml">𝒯</mi><annotation-xml encoding="MathML-Content" id="S3.p4.1.m1.1b"><ci id="S3.p4.1.m1.1.1.cmml" xref="S3.p4.1.m1.1.1">𝒯</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.p4.1.m1.1c">\mathcal{T}</annotation></semantics></math>. When operations are implicit, a model is more likely to internalize abstract representations for them that can be improved with training. Explicit operations will turn into explicit reasoning moves made by the LM. We choose to represent the current state, the goal state, the backtracking operations, the goal checks and the exploration choices explicitly in the trajectory (written using language, <a href="#S3.F2" title="Figure 2 ‣ 3 A Language for Search ‣ Stream of Search (SoS): Learning to Search in Language" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Fig. 2</span></a>). We choose to keep the heuristic functions, values of states and the pruning strategy implicit.</p>
</div>
</section>
<section id="S4" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">4 </span>Problem Setup</h2>

<section id="S4.SS0.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Task Description: Countdown</h4>

<div id="S4.SS0.SSS0.Px1.p1" class="ltx_para ltx_noindent">
<p id="S4.SS0.SSS0.Px1.p1.1" class="ltx_p">To show the utility of using streams of search, we look at a generalization of the 24 Game <cite class="ltx_cite ltx_citemacro_citep">(Yang et al., <a href="#bib.bib25" title="" class="ltx_ref">2022</a>)</cite> called Countdown <cite class="ltx_cite ltx_citemacro_citep">(Countdown, <a href="#bib.bib5" title="" class="ltx_ref">2024</a>)</cite>. Countdown is a game where a set of input numbers need to be combined with simple arithmetic operations to reach a target number (see <a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ Stream of Search (SoS): Learning to Search in Language" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Fig. 1</span></a>a). We choose this task, since it has a high branching factor (<math id="S4.SS0.SSS0.Px1.p1.1.m1.3" class="ltx_Math" alttext="{N\choose 2}*4" display="inline"><semantics id="S4.SS0.SSS0.Px1.p1.1.m1.3a"><mrow id="S4.SS0.SSS0.Px1.p1.1.m1.3.4" xref="S4.SS0.SSS0.Px1.p1.1.m1.3.4.cmml"><mrow id="S4.SS0.SSS0.Px1.p1.1.m1.3.3.5" xref="S4.SS0.SSS0.Px1.p1.1.m1.3.3.4.cmml"><mo id="S4.SS0.SSS0.Px1.p1.1.m1.3.3.5.1" xref="S4.SS0.SSS0.Px1.p1.1.m1.1.1.1.1.1.cmml">(</mo><mfrac linethickness="0pt" id="S4.SS0.SSS0.Px1.p1.1.m1.3.3.3.3" xref="S4.SS0.SSS0.Px1.p1.1.m1.3.3.4.cmml"><mi id="S4.SS0.SSS0.Px1.p1.1.m1.2.2.2.2.2" xref="S4.SS0.SSS0.Px1.p1.1.m1.2.2.2.2.2.cmml">N</mi><mn id="S4.SS0.SSS0.Px1.p1.1.m1.3.3.3.3.3" xref="S4.SS0.SSS0.Px1.p1.1.m1.3.3.3.3.3.cmml">2</mn></mfrac><mo rspace="0.055em" id="S4.SS0.SSS0.Px1.p1.1.m1.3.3.5.2" xref="S4.SS0.SSS0.Px1.p1.1.m1.1.1.1.1.1.cmml">)</mo></mrow><mo rspace="0.222em" id="S4.SS0.SSS0.Px1.p1.1.m1.3.4.1" xref="S4.SS0.SSS0.Px1.p1.1.m1.3.4.1.cmml">∗</mo><mn id="S4.SS0.SSS0.Px1.p1.1.m1.3.4.2" xref="S4.SS0.SSS0.Px1.p1.1.m1.3.4.2.cmml">4</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.SS0.SSS0.Px1.p1.1.m1.3b"><apply id="S4.SS0.SSS0.Px1.p1.1.m1.3.4.cmml" xref="S4.SS0.SSS0.Px1.p1.1.m1.3.4"><times id="S4.SS0.SSS0.Px1.p1.1.m1.3.4.1.cmml" xref="S4.SS0.SSS0.Px1.p1.1.m1.3.4.1"></times><apply id="S4.SS0.SSS0.Px1.p1.1.m1.3.3.4.cmml" xref="S4.SS0.SSS0.Px1.p1.1.m1.3.3.5"><csymbol cd="latexml" id="S4.SS0.SSS0.Px1.p1.1.m1.1.1.1.1.1.cmml" xref="S4.SS0.SSS0.Px1.p1.1.m1.3.3.5.1">binomial</csymbol><ci id="S4.SS0.SSS0.Px1.p1.1.m1.2.2.2.2.2.cmml" xref="S4.SS0.SSS0.Px1.p1.1.m1.2.2.2.2.2">𝑁</ci><cn type="integer" id="S4.SS0.SSS0.Px1.p1.1.m1.3.3.3.3.3.cmml" xref="S4.SS0.SSS0.Px1.p1.1.m1.3.3.3.3.3">2</cn></apply><cn type="integer" id="S4.SS0.SSS0.Px1.p1.1.m1.3.4.2.cmml" xref="S4.SS0.SSS0.Px1.p1.1.m1.3.4.2">4</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS0.SSS0.Px1.p1.1.m1.3c">{N\choose 2}*4</annotation></semantics></math> for a depth that has N inputs) and thus requires planning, search and backtracking to be solved. We consider problems with 4 input numbers since these problems are challenging enough to have long search traces without the search traces exceeding a standard LM context window (for example, the search trajectories for games with 5 input numbers can be 60,000 tokens long). The range of target numbers in our problems is from 10 to 100. We randomly hold out 10% of the targets for an ‘out-of-distribution’ evaluation.</p>
</div>
</section>
<section id="S4.SS0.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Training data</h4>

<div id="S4.SS0.SSS0.Px2.p1" class="ltx_para ltx_noindent">
<p id="S4.SS0.SSS0.Px2.p1.1" class="ltx_p">To train a model on streams of search for Countdown, we construct a synthetic dataset using set of diverse, and suboptimal symbolic search strategies.
To construct our Stream of Search dataset, we define 12 search strategies based on breadth first search (BFS) and depth first search (DFS) (see App. <a href="#alg4" title="Algorithm 4 ‣ Appendix B Creation of the SoS Dataset ‣ Stream of Search (SoS): Learning to Search in Language" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Alg. 4</span></a>, <a href="#alg3" title="Algorithm 3 ‣ Appendix B Creation of the SoS Dataset ‣ Stream of Search (SoS): Learning to Search in Language" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Alg. 3</span></a>) that rely on two simple and interpretable heuristic functions. The heuristics we use to guide search are 1) the absolute difference between the sum of the remaining options and the target and 2) distance to the factors of the target (see <a href="#A2" title="Appendix B Creation of the SoS Dataset ‣ Stream of Search (SoS): Learning to Search in Language" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">App. B</span></a>). We generate a stream of search dataset of 500,000 search trajectories and corresponding trajectories with just the optimal solution. Of these 500,000 trajectories, only about 57% of the trajectories lead to the solution.
In our dataset, a search trajectory or a stream of search is serialized as a string that represents a list of tree nodes / states in the order of traversal (either generation or exploration).
We hold out two kinds of generalization cases: 1) seen targets with new sets of inputs, and 2) new targets with new sets of inputs.</p>
</div>
</section>
<section id="S4.SS0.SSS0.Px3" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Metrics</h4>

<div id="S4.SS0.SSS0.Px3.p1" class="ltx_para ltx_noindent">
<p id="S4.SS0.SSS0.Px3.p1.3" class="ltx_p">To measure accuracy in Countdown, we evaluate the percentage of problems for which the model is able to generate a correct solution trajectory. More formally, we define correctness as a binary function, <math id="S4.SS0.SSS0.Px3.p1.1.m1.1" class="ltx_Math" alttext="\mathbf{1}" display="inline"><semantics id="S4.SS0.SSS0.Px3.p1.1.m1.1a"><mn id="S4.SS0.SSS0.Px3.p1.1.m1.1.1" xref="S4.SS0.SSS0.Px3.p1.1.m1.1.1.cmml">𝟏</mn><annotation-xml encoding="MathML-Content" id="S4.SS0.SSS0.Px3.p1.1.m1.1b"><cn type="integer" id="S4.SS0.SSS0.Px3.p1.1.m1.1.1.cmml" xref="S4.SS0.SSS0.Px3.p1.1.m1.1.1">1</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS0.SSS0.Px3.p1.1.m1.1c">\mathbf{1}</annotation></semantics></math>, if the correct path to the solution, <math id="S4.SS0.SSS0.Px3.p1.2.m2.1" class="ltx_Math" alttext="\mathcal{P}" display="inline"><semantics id="S4.SS0.SSS0.Px3.p1.2.m2.1a"><mi class="ltx_font_mathcaligraphic" id="S4.SS0.SSS0.Px3.p1.2.m2.1.1" xref="S4.SS0.SSS0.Px3.p1.2.m2.1.1.cmml">𝒫</mi><annotation-xml encoding="MathML-Content" id="S4.SS0.SSS0.Px3.p1.2.m2.1b"><ci id="S4.SS0.SSS0.Px3.p1.2.m2.1.1.cmml" xref="S4.SS0.SSS0.Px3.p1.2.m2.1.1">𝒫</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS0.SSS0.Px3.p1.2.m2.1c">\mathcal{P}</annotation></semantics></math>, is present in the generated trajectory, <math id="S4.SS0.SSS0.Px3.p1.3.m3.1" class="ltx_Math" alttext="\mathcal{T}" display="inline"><semantics id="S4.SS0.SSS0.Px3.p1.3.m3.1a"><mi class="ltx_font_mathcaligraphic" id="S4.SS0.SSS0.Px3.p1.3.m3.1.1" xref="S4.SS0.SSS0.Px3.p1.3.m3.1.1.cmml">𝒯</mi><annotation-xml encoding="MathML-Content" id="S4.SS0.SSS0.Px3.p1.3.m3.1b"><ci id="S4.SS0.SSS0.Px3.p1.3.m3.1.1.cmml" xref="S4.SS0.SSS0.Px3.p1.3.m3.1.1">𝒯</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS0.SSS0.Px3.p1.3.m3.1c">\mathcal{T}</annotation></semantics></math>.</p>
</div>
<div id="S4.SS0.SSS0.Px3.p2" class="ltx_para ltx_noindent">
<p id="S4.SS0.SSS0.Px3.p2.1" class="ltx_p">To quantitatively understand the search strategies used by the trained models, we define two ways to measure alignment between different search strategies:</p>
<ol id="S4.I1" class="ltx_enumerate">
<li id="S4.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">1.</span> 
<div id="S4.I1.i1.p1" class="ltx_para">
<p id="S4.I1.i1.p1.1" class="ltx_p">Alignment of Correctness: This measures whether two search strategies solve the same set of problems correctly and incorrectly. We calculate this as the Pearson correlation between the solved and unsolved problems for the two search strategies.</p>
</div>
</li>
<li id="S4.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">2.</span> 
<div id="S4.I1.i2.p1" class="ltx_para ltx_noindent">
<p id="S4.I1.i2.p1.7" class="ltx_p">Alignment of States Visited: This measures the overlap in the states visited by two search strategies. To calculate this, we parse the search trajectories, <math id="S4.I1.i2.p1.1.m1.1" class="ltx_Math" alttext="\mathcal{T}_{1}" display="inline"><semantics id="S4.I1.i2.p1.1.m1.1a"><msub id="S4.I1.i2.p1.1.m1.1.1" xref="S4.I1.i2.p1.1.m1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S4.I1.i2.p1.1.m1.1.1.2" xref="S4.I1.i2.p1.1.m1.1.1.2.cmml">𝒯</mi><mn id="S4.I1.i2.p1.1.m1.1.1.3" xref="S4.I1.i2.p1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S4.I1.i2.p1.1.m1.1b"><apply id="S4.I1.i2.p1.1.m1.1.1.cmml" xref="S4.I1.i2.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S4.I1.i2.p1.1.m1.1.1.1.cmml" xref="S4.I1.i2.p1.1.m1.1.1">subscript</csymbol><ci id="S4.I1.i2.p1.1.m1.1.1.2.cmml" xref="S4.I1.i2.p1.1.m1.1.1.2">𝒯</ci><cn type="integer" id="S4.I1.i2.p1.1.m1.1.1.3.cmml" xref="S4.I1.i2.p1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.I1.i2.p1.1.m1.1c">\mathcal{T}_{1}</annotation></semantics></math> and <math id="S4.I1.i2.p1.2.m2.1" class="ltx_Math" alttext="\mathcal{T}_{2}" display="inline"><semantics id="S4.I1.i2.p1.2.m2.1a"><msub id="S4.I1.i2.p1.2.m2.1.1" xref="S4.I1.i2.p1.2.m2.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S4.I1.i2.p1.2.m2.1.1.2" xref="S4.I1.i2.p1.2.m2.1.1.2.cmml">𝒯</mi><mn id="S4.I1.i2.p1.2.m2.1.1.3" xref="S4.I1.i2.p1.2.m2.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="S4.I1.i2.p1.2.m2.1b"><apply id="S4.I1.i2.p1.2.m2.1.1.cmml" xref="S4.I1.i2.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S4.I1.i2.p1.2.m2.1.1.1.cmml" xref="S4.I1.i2.p1.2.m2.1.1">subscript</csymbol><ci id="S4.I1.i2.p1.2.m2.1.1.2.cmml" xref="S4.I1.i2.p1.2.m2.1.1.2">𝒯</ci><cn type="integer" id="S4.I1.i2.p1.2.m2.1.1.3.cmml" xref="S4.I1.i2.p1.2.m2.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.I1.i2.p1.2.m2.1c">\mathcal{T}_{2}</annotation></semantics></math>, into their constituent states and count the number of states that are common between them. We normalize this count by dividing it by the maximum number of states in the two trajectories. More formally: <math id="S4.I1.i2.p1.3.m3.6" class="ltx_Math" alttext="\text{State Alignment}(\mathcal{T}_{1},\mathcal{T}_{2})=\frac{|\mathcal{T}_{1}\cap\mathcal{T}_{2}|}{\max(|\mathcal{T}_{1}|,|\mathcal{T}_{2}|)}" display="inline"><semantics id="S4.I1.i2.p1.3.m3.6a"><mrow id="S4.I1.i2.p1.3.m3.6.6" xref="S4.I1.i2.p1.3.m3.6.6.cmml"><mrow id="S4.I1.i2.p1.3.m3.6.6.2" xref="S4.I1.i2.p1.3.m3.6.6.2.cmml"><mtext id="S4.I1.i2.p1.3.m3.6.6.2.4" xref="S4.I1.i2.p1.3.m3.6.6.2.4a.cmml">State Alignment</mtext><mo lspace="0em" rspace="0em" id="S4.I1.i2.p1.3.m3.6.6.2.3" xref="S4.I1.i2.p1.3.m3.6.6.2.3.cmml">​</mo><mrow id="S4.I1.i2.p1.3.m3.6.6.2.2.2" xref="S4.I1.i2.p1.3.m3.6.6.2.2.3.cmml"><mo stretchy="false" id="S4.I1.i2.p1.3.m3.6.6.2.2.2.3" xref="S4.I1.i2.p1.3.m3.6.6.2.2.3.cmml">(</mo><msub id="S4.I1.i2.p1.3.m3.5.5.1.1.1.1" xref="S4.I1.i2.p1.3.m3.5.5.1.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S4.I1.i2.p1.3.m3.5.5.1.1.1.1.2" xref="S4.I1.i2.p1.3.m3.5.5.1.1.1.1.2.cmml">𝒯</mi><mn id="S4.I1.i2.p1.3.m3.5.5.1.1.1.1.3" xref="S4.I1.i2.p1.3.m3.5.5.1.1.1.1.3.cmml">1</mn></msub><mo id="S4.I1.i2.p1.3.m3.6.6.2.2.2.4" xref="S4.I1.i2.p1.3.m3.6.6.2.2.3.cmml">,</mo><msub id="S4.I1.i2.p1.3.m3.6.6.2.2.2.2" xref="S4.I1.i2.p1.3.m3.6.6.2.2.2.2.cmml"><mi class="ltx_font_mathcaligraphic" id="S4.I1.i2.p1.3.m3.6.6.2.2.2.2.2" xref="S4.I1.i2.p1.3.m3.6.6.2.2.2.2.2.cmml">𝒯</mi><mn id="S4.I1.i2.p1.3.m3.6.6.2.2.2.2.3" xref="S4.I1.i2.p1.3.m3.6.6.2.2.2.2.3.cmml">2</mn></msub><mo stretchy="false" id="S4.I1.i2.p1.3.m3.6.6.2.2.2.5" xref="S4.I1.i2.p1.3.m3.6.6.2.2.3.cmml">)</mo></mrow></mrow><mo id="S4.I1.i2.p1.3.m3.6.6.3" xref="S4.I1.i2.p1.3.m3.6.6.3.cmml">=</mo><mfrac id="S4.I1.i2.p1.3.m3.4.4" xref="S4.I1.i2.p1.3.m3.4.4.cmml"><mrow id="S4.I1.i2.p1.3.m3.1.1.1.1" xref="S4.I1.i2.p1.3.m3.1.1.1.2.cmml"><mo stretchy="false" id="S4.I1.i2.p1.3.m3.1.1.1.1.2" xref="S4.I1.i2.p1.3.m3.1.1.1.2.1.cmml">|</mo><mrow id="S4.I1.i2.p1.3.m3.1.1.1.1.1" xref="S4.I1.i2.p1.3.m3.1.1.1.1.1.cmml"><msub id="S4.I1.i2.p1.3.m3.1.1.1.1.1.2" xref="S4.I1.i2.p1.3.m3.1.1.1.1.1.2.cmml"><mi class="ltx_font_mathcaligraphic" id="S4.I1.i2.p1.3.m3.1.1.1.1.1.2.2" xref="S4.I1.i2.p1.3.m3.1.1.1.1.1.2.2.cmml">𝒯</mi><mn id="S4.I1.i2.p1.3.m3.1.1.1.1.1.2.3" xref="S4.I1.i2.p1.3.m3.1.1.1.1.1.2.3.cmml">1</mn></msub><mo id="S4.I1.i2.p1.3.m3.1.1.1.1.1.1" xref="S4.I1.i2.p1.3.m3.1.1.1.1.1.1.cmml">∩</mo><msub id="S4.I1.i2.p1.3.m3.1.1.1.1.1.3" xref="S4.I1.i2.p1.3.m3.1.1.1.1.1.3.cmml"><mi class="ltx_font_mathcaligraphic" id="S4.I1.i2.p1.3.m3.1.1.1.1.1.3.2" xref="S4.I1.i2.p1.3.m3.1.1.1.1.1.3.2.cmml">𝒯</mi><mn id="S4.I1.i2.p1.3.m3.1.1.1.1.1.3.3" xref="S4.I1.i2.p1.3.m3.1.1.1.1.1.3.3.cmml">2</mn></msub></mrow><mo stretchy="false" id="S4.I1.i2.p1.3.m3.1.1.1.1.3" xref="S4.I1.i2.p1.3.m3.1.1.1.2.1.cmml">|</mo></mrow><mrow id="S4.I1.i2.p1.3.m3.4.4.4.3" xref="S4.I1.i2.p1.3.m3.4.4.4.4.cmml"><mi id="S4.I1.i2.p1.3.m3.2.2.2.1" xref="S4.I1.i2.p1.3.m3.2.2.2.1.cmml">max</mi><mo id="S4.I1.i2.p1.3.m3.4.4.4.3a" xref="S4.I1.i2.p1.3.m3.4.4.4.4.cmml">⁡</mo><mrow id="S4.I1.i2.p1.3.m3.4.4.4.3.2" xref="S4.I1.i2.p1.3.m3.4.4.4.4.cmml"><mo stretchy="false" id="S4.I1.i2.p1.3.m3.4.4.4.3.2.3" xref="S4.I1.i2.p1.3.m3.4.4.4.4.cmml">(</mo><mrow id="S4.I1.i2.p1.3.m3.3.3.3.2.1.1.1" xref="S4.I1.i2.p1.3.m3.3.3.3.2.1.1.2.cmml"><mo stretchy="false" id="S4.I1.i2.p1.3.m3.3.3.3.2.1.1.1.2" xref="S4.I1.i2.p1.3.m3.3.3.3.2.1.1.2.1.cmml">|</mo><msub id="S4.I1.i2.p1.3.m3.3.3.3.2.1.1.1.1" xref="S4.I1.i2.p1.3.m3.3.3.3.2.1.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S4.I1.i2.p1.3.m3.3.3.3.2.1.1.1.1.2" xref="S4.I1.i2.p1.3.m3.3.3.3.2.1.1.1.1.2.cmml">𝒯</mi><mn id="S4.I1.i2.p1.3.m3.3.3.3.2.1.1.1.1.3" xref="S4.I1.i2.p1.3.m3.3.3.3.2.1.1.1.1.3.cmml">1</mn></msub><mo stretchy="false" id="S4.I1.i2.p1.3.m3.3.3.3.2.1.1.1.3" xref="S4.I1.i2.p1.3.m3.3.3.3.2.1.1.2.1.cmml">|</mo></mrow><mo id="S4.I1.i2.p1.3.m3.4.4.4.3.2.4" xref="S4.I1.i2.p1.3.m3.4.4.4.4.cmml">,</mo><mrow id="S4.I1.i2.p1.3.m3.4.4.4.3.2.2.1" xref="S4.I1.i2.p1.3.m3.4.4.4.3.2.2.2.cmml"><mo stretchy="false" id="S4.I1.i2.p1.3.m3.4.4.4.3.2.2.1.2" xref="S4.I1.i2.p1.3.m3.4.4.4.3.2.2.2.1.cmml">|</mo><msub id="S4.I1.i2.p1.3.m3.4.4.4.3.2.2.1.1" xref="S4.I1.i2.p1.3.m3.4.4.4.3.2.2.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S4.I1.i2.p1.3.m3.4.4.4.3.2.2.1.1.2" xref="S4.I1.i2.p1.3.m3.4.4.4.3.2.2.1.1.2.cmml">𝒯</mi><mn id="S4.I1.i2.p1.3.m3.4.4.4.3.2.2.1.1.3" xref="S4.I1.i2.p1.3.m3.4.4.4.3.2.2.1.1.3.cmml">2</mn></msub><mo stretchy="false" id="S4.I1.i2.p1.3.m3.4.4.4.3.2.2.1.3" xref="S4.I1.i2.p1.3.m3.4.4.4.3.2.2.2.1.cmml">|</mo></mrow><mo stretchy="false" id="S4.I1.i2.p1.3.m3.4.4.4.3.2.5" xref="S4.I1.i2.p1.3.m3.4.4.4.4.cmml">)</mo></mrow></mrow></mfrac></mrow><annotation-xml encoding="MathML-Content" id="S4.I1.i2.p1.3.m3.6b"><apply id="S4.I1.i2.p1.3.m3.6.6.cmml" xref="S4.I1.i2.p1.3.m3.6.6"><eq id="S4.I1.i2.p1.3.m3.6.6.3.cmml" xref="S4.I1.i2.p1.3.m3.6.6.3"></eq><apply id="S4.I1.i2.p1.3.m3.6.6.2.cmml" xref="S4.I1.i2.p1.3.m3.6.6.2"><times id="S4.I1.i2.p1.3.m3.6.6.2.3.cmml" xref="S4.I1.i2.p1.3.m3.6.6.2.3"></times><ci id="S4.I1.i2.p1.3.m3.6.6.2.4a.cmml" xref="S4.I1.i2.p1.3.m3.6.6.2.4"><mtext id="S4.I1.i2.p1.3.m3.6.6.2.4.cmml" xref="S4.I1.i2.p1.3.m3.6.6.2.4">State Alignment</mtext></ci><interval closure="open" id="S4.I1.i2.p1.3.m3.6.6.2.2.3.cmml" xref="S4.I1.i2.p1.3.m3.6.6.2.2.2"><apply id="S4.I1.i2.p1.3.m3.5.5.1.1.1.1.cmml" xref="S4.I1.i2.p1.3.m3.5.5.1.1.1.1"><csymbol cd="ambiguous" id="S4.I1.i2.p1.3.m3.5.5.1.1.1.1.1.cmml" xref="S4.I1.i2.p1.3.m3.5.5.1.1.1.1">subscript</csymbol><ci id="S4.I1.i2.p1.3.m3.5.5.1.1.1.1.2.cmml" xref="S4.I1.i2.p1.3.m3.5.5.1.1.1.1.2">𝒯</ci><cn type="integer" id="S4.I1.i2.p1.3.m3.5.5.1.1.1.1.3.cmml" xref="S4.I1.i2.p1.3.m3.5.5.1.1.1.1.3">1</cn></apply><apply id="S4.I1.i2.p1.3.m3.6.6.2.2.2.2.cmml" xref="S4.I1.i2.p1.3.m3.6.6.2.2.2.2"><csymbol cd="ambiguous" id="S4.I1.i2.p1.3.m3.6.6.2.2.2.2.1.cmml" xref="S4.I1.i2.p1.3.m3.6.6.2.2.2.2">subscript</csymbol><ci id="S4.I1.i2.p1.3.m3.6.6.2.2.2.2.2.cmml" xref="S4.I1.i2.p1.3.m3.6.6.2.2.2.2.2">𝒯</ci><cn type="integer" id="S4.I1.i2.p1.3.m3.6.6.2.2.2.2.3.cmml" xref="S4.I1.i2.p1.3.m3.6.6.2.2.2.2.3">2</cn></apply></interval></apply><apply id="S4.I1.i2.p1.3.m3.4.4.cmml" xref="S4.I1.i2.p1.3.m3.4.4"><divide id="S4.I1.i2.p1.3.m3.4.4.5.cmml" xref="S4.I1.i2.p1.3.m3.4.4"></divide><apply id="S4.I1.i2.p1.3.m3.1.1.1.2.cmml" xref="S4.I1.i2.p1.3.m3.1.1.1.1"><abs id="S4.I1.i2.p1.3.m3.1.1.1.2.1.cmml" xref="S4.I1.i2.p1.3.m3.1.1.1.1.2"></abs><apply id="S4.I1.i2.p1.3.m3.1.1.1.1.1.cmml" xref="S4.I1.i2.p1.3.m3.1.1.1.1.1"><intersect id="S4.I1.i2.p1.3.m3.1.1.1.1.1.1.cmml" xref="S4.I1.i2.p1.3.m3.1.1.1.1.1.1"></intersect><apply id="S4.I1.i2.p1.3.m3.1.1.1.1.1.2.cmml" xref="S4.I1.i2.p1.3.m3.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S4.I1.i2.p1.3.m3.1.1.1.1.1.2.1.cmml" xref="S4.I1.i2.p1.3.m3.1.1.1.1.1.2">subscript</csymbol><ci id="S4.I1.i2.p1.3.m3.1.1.1.1.1.2.2.cmml" xref="S4.I1.i2.p1.3.m3.1.1.1.1.1.2.2">𝒯</ci><cn type="integer" id="S4.I1.i2.p1.3.m3.1.1.1.1.1.2.3.cmml" xref="S4.I1.i2.p1.3.m3.1.1.1.1.1.2.3">1</cn></apply><apply id="S4.I1.i2.p1.3.m3.1.1.1.1.1.3.cmml" xref="S4.I1.i2.p1.3.m3.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S4.I1.i2.p1.3.m3.1.1.1.1.1.3.1.cmml" xref="S4.I1.i2.p1.3.m3.1.1.1.1.1.3">subscript</csymbol><ci id="S4.I1.i2.p1.3.m3.1.1.1.1.1.3.2.cmml" xref="S4.I1.i2.p1.3.m3.1.1.1.1.1.3.2">𝒯</ci><cn type="integer" id="S4.I1.i2.p1.3.m3.1.1.1.1.1.3.3.cmml" xref="S4.I1.i2.p1.3.m3.1.1.1.1.1.3.3">2</cn></apply></apply></apply><apply id="S4.I1.i2.p1.3.m3.4.4.4.4.cmml" xref="S4.I1.i2.p1.3.m3.4.4.4.3"><max id="S4.I1.i2.p1.3.m3.2.2.2.1.cmml" xref="S4.I1.i2.p1.3.m3.2.2.2.1"></max><apply id="S4.I1.i2.p1.3.m3.3.3.3.2.1.1.2.cmml" xref="S4.I1.i2.p1.3.m3.3.3.3.2.1.1.1"><abs id="S4.I1.i2.p1.3.m3.3.3.3.2.1.1.2.1.cmml" xref="S4.I1.i2.p1.3.m3.3.3.3.2.1.1.1.2"></abs><apply id="S4.I1.i2.p1.3.m3.3.3.3.2.1.1.1.1.cmml" xref="S4.I1.i2.p1.3.m3.3.3.3.2.1.1.1.1"><csymbol cd="ambiguous" id="S4.I1.i2.p1.3.m3.3.3.3.2.1.1.1.1.1.cmml" xref="S4.I1.i2.p1.3.m3.3.3.3.2.1.1.1.1">subscript</csymbol><ci id="S4.I1.i2.p1.3.m3.3.3.3.2.1.1.1.1.2.cmml" xref="S4.I1.i2.p1.3.m3.3.3.3.2.1.1.1.1.2">𝒯</ci><cn type="integer" id="S4.I1.i2.p1.3.m3.3.3.3.2.1.1.1.1.3.cmml" xref="S4.I1.i2.p1.3.m3.3.3.3.2.1.1.1.1.3">1</cn></apply></apply><apply id="S4.I1.i2.p1.3.m3.4.4.4.3.2.2.2.cmml" xref="S4.I1.i2.p1.3.m3.4.4.4.3.2.2.1"><abs id="S4.I1.i2.p1.3.m3.4.4.4.3.2.2.2.1.cmml" xref="S4.I1.i2.p1.3.m3.4.4.4.3.2.2.1.2"></abs><apply id="S4.I1.i2.p1.3.m3.4.4.4.3.2.2.1.1.cmml" xref="S4.I1.i2.p1.3.m3.4.4.4.3.2.2.1.1"><csymbol cd="ambiguous" id="S4.I1.i2.p1.3.m3.4.4.4.3.2.2.1.1.1.cmml" xref="S4.I1.i2.p1.3.m3.4.4.4.3.2.2.1.1">subscript</csymbol><ci id="S4.I1.i2.p1.3.m3.4.4.4.3.2.2.1.1.2.cmml" xref="S4.I1.i2.p1.3.m3.4.4.4.3.2.2.1.1.2">𝒯</ci><cn type="integer" id="S4.I1.i2.p1.3.m3.4.4.4.3.2.2.1.1.3.cmml" xref="S4.I1.i2.p1.3.m3.4.4.4.3.2.2.1.1.3">2</cn></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.I1.i2.p1.3.m3.6c">\text{State Alignment}(\mathcal{T}_{1},\mathcal{T}_{2})=\frac{|\mathcal{T}_{1}\cap\mathcal{T}_{2}|}{\max(|\mathcal{T}_{1}|,|\mathcal{T}_{2}|)}</annotation></semantics></math> where <math id="S4.I1.i2.p1.4.m4.1" class="ltx_Math" alttext="|\mathcal{T}_{1}|" display="inline"><semantics id="S4.I1.i2.p1.4.m4.1a"><mrow id="S4.I1.i2.p1.4.m4.1.1.1" xref="S4.I1.i2.p1.4.m4.1.1.2.cmml"><mo stretchy="false" id="S4.I1.i2.p1.4.m4.1.1.1.2" xref="S4.I1.i2.p1.4.m4.1.1.2.1.cmml">|</mo><msub id="S4.I1.i2.p1.4.m4.1.1.1.1" xref="S4.I1.i2.p1.4.m4.1.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S4.I1.i2.p1.4.m4.1.1.1.1.2" xref="S4.I1.i2.p1.4.m4.1.1.1.1.2.cmml">𝒯</mi><mn id="S4.I1.i2.p1.4.m4.1.1.1.1.3" xref="S4.I1.i2.p1.4.m4.1.1.1.1.3.cmml">1</mn></msub><mo stretchy="false" id="S4.I1.i2.p1.4.m4.1.1.1.3" xref="S4.I1.i2.p1.4.m4.1.1.2.1.cmml">|</mo></mrow><annotation-xml encoding="MathML-Content" id="S4.I1.i2.p1.4.m4.1b"><apply id="S4.I1.i2.p1.4.m4.1.1.2.cmml" xref="S4.I1.i2.p1.4.m4.1.1.1"><abs id="S4.I1.i2.p1.4.m4.1.1.2.1.cmml" xref="S4.I1.i2.p1.4.m4.1.1.1.2"></abs><apply id="S4.I1.i2.p1.4.m4.1.1.1.1.cmml" xref="S4.I1.i2.p1.4.m4.1.1.1.1"><csymbol cd="ambiguous" id="S4.I1.i2.p1.4.m4.1.1.1.1.1.cmml" xref="S4.I1.i2.p1.4.m4.1.1.1.1">subscript</csymbol><ci id="S4.I1.i2.p1.4.m4.1.1.1.1.2.cmml" xref="S4.I1.i2.p1.4.m4.1.1.1.1.2">𝒯</ci><cn type="integer" id="S4.I1.i2.p1.4.m4.1.1.1.1.3.cmml" xref="S4.I1.i2.p1.4.m4.1.1.1.1.3">1</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.I1.i2.p1.4.m4.1c">|\mathcal{T}_{1}|</annotation></semantics></math> and <math id="S4.I1.i2.p1.5.m5.1" class="ltx_Math" alttext="|\mathcal{T}_{2}|" display="inline"><semantics id="S4.I1.i2.p1.5.m5.1a"><mrow id="S4.I1.i2.p1.5.m5.1.1.1" xref="S4.I1.i2.p1.5.m5.1.1.2.cmml"><mo stretchy="false" id="S4.I1.i2.p1.5.m5.1.1.1.2" xref="S4.I1.i2.p1.5.m5.1.1.2.1.cmml">|</mo><msub id="S4.I1.i2.p1.5.m5.1.1.1.1" xref="S4.I1.i2.p1.5.m5.1.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S4.I1.i2.p1.5.m5.1.1.1.1.2" xref="S4.I1.i2.p1.5.m5.1.1.1.1.2.cmml">𝒯</mi><mn id="S4.I1.i2.p1.5.m5.1.1.1.1.3" xref="S4.I1.i2.p1.5.m5.1.1.1.1.3.cmml">2</mn></msub><mo stretchy="false" id="S4.I1.i2.p1.5.m5.1.1.1.3" xref="S4.I1.i2.p1.5.m5.1.1.2.1.cmml">|</mo></mrow><annotation-xml encoding="MathML-Content" id="S4.I1.i2.p1.5.m5.1b"><apply id="S4.I1.i2.p1.5.m5.1.1.2.cmml" xref="S4.I1.i2.p1.5.m5.1.1.1"><abs id="S4.I1.i2.p1.5.m5.1.1.2.1.cmml" xref="S4.I1.i2.p1.5.m5.1.1.1.2"></abs><apply id="S4.I1.i2.p1.5.m5.1.1.1.1.cmml" xref="S4.I1.i2.p1.5.m5.1.1.1.1"><csymbol cd="ambiguous" id="S4.I1.i2.p1.5.m5.1.1.1.1.1.cmml" xref="S4.I1.i2.p1.5.m5.1.1.1.1">subscript</csymbol><ci id="S4.I1.i2.p1.5.m5.1.1.1.1.2.cmml" xref="S4.I1.i2.p1.5.m5.1.1.1.1.2">𝒯</ci><cn type="integer" id="S4.I1.i2.p1.5.m5.1.1.1.1.3.cmml" xref="S4.I1.i2.p1.5.m5.1.1.1.1.3">2</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.I1.i2.p1.5.m5.1c">|\mathcal{T}_{2}|</annotation></semantics></math> denote the number of states in trajectories <math id="S4.I1.i2.p1.6.m6.1" class="ltx_Math" alttext="\mathcal{T}_{1}" display="inline"><semantics id="S4.I1.i2.p1.6.m6.1a"><msub id="S4.I1.i2.p1.6.m6.1.1" xref="S4.I1.i2.p1.6.m6.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S4.I1.i2.p1.6.m6.1.1.2" xref="S4.I1.i2.p1.6.m6.1.1.2.cmml">𝒯</mi><mn id="S4.I1.i2.p1.6.m6.1.1.3" xref="S4.I1.i2.p1.6.m6.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S4.I1.i2.p1.6.m6.1b"><apply id="S4.I1.i2.p1.6.m6.1.1.cmml" xref="S4.I1.i2.p1.6.m6.1.1"><csymbol cd="ambiguous" id="S4.I1.i2.p1.6.m6.1.1.1.cmml" xref="S4.I1.i2.p1.6.m6.1.1">subscript</csymbol><ci id="S4.I1.i2.p1.6.m6.1.1.2.cmml" xref="S4.I1.i2.p1.6.m6.1.1.2">𝒯</ci><cn type="integer" id="S4.I1.i2.p1.6.m6.1.1.3.cmml" xref="S4.I1.i2.p1.6.m6.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.I1.i2.p1.6.m6.1c">\mathcal{T}_{1}</annotation></semantics></math> and <math id="S4.I1.i2.p1.7.m7.1" class="ltx_Math" alttext="\mathcal{T}_{2}" display="inline"><semantics id="S4.I1.i2.p1.7.m7.1a"><msub id="S4.I1.i2.p1.7.m7.1.1" xref="S4.I1.i2.p1.7.m7.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S4.I1.i2.p1.7.m7.1.1.2" xref="S4.I1.i2.p1.7.m7.1.1.2.cmml">𝒯</mi><mn id="S4.I1.i2.p1.7.m7.1.1.3" xref="S4.I1.i2.p1.7.m7.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="S4.I1.i2.p1.7.m7.1b"><apply id="S4.I1.i2.p1.7.m7.1.1.cmml" xref="S4.I1.i2.p1.7.m7.1.1"><csymbol cd="ambiguous" id="S4.I1.i2.p1.7.m7.1.1.1.cmml" xref="S4.I1.i2.p1.7.m7.1.1">subscript</csymbol><ci id="S4.I1.i2.p1.7.m7.1.1.2.cmml" xref="S4.I1.i2.p1.7.m7.1.1.2">𝒯</ci><cn type="integer" id="S4.I1.i2.p1.7.m7.1.1.3.cmml" xref="S4.I1.i2.p1.7.m7.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.I1.i2.p1.7.m7.1c">\mathcal{T}_{2}</annotation></semantics></math>, respectively. To measure alignment between two models, we calculate the state alignment score for each problem and then compute the mean of these scores.</p>
</div>
</li>
</ol>
</div>
</section>
</section>
<section id="S5" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">5 </span>Learning from Suboptimal Search Strategies</h2>

<div id="S5.p1" class="ltx_para ltx_noindent">
<p id="S5.p1.1" class="ltx_p">Is it more useful to learn from clean, optimal solutions, or from messy, and sometimes unsuccessful, search trajectories? In this section we train LMs from scratch on one or the other and evaluate there performance at solving held out Countdown problems.</p>
</div>
<figure id="S5.F3" class="ltx_figure"><img src="/html/2404.03683/assets/x3.png" id="S5.F3.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="422" height="130" alt="Refer to caption">
<br class="ltx_break ltx_centering">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 3: </span>
(a) Average accuracy of the symbolic search strategies used to construct the Stream of Search dataset compared to the model trained with optimal paths and the Stream of Search model. Error bars represent 95% binomial confidence intervals. (b) Alignment of Correctness: The correlations of performance on different problems for the Stream of Search model and different symbolic search strategies. (c) Alignment of States Visited: Number of times each search strategy visited the same states, normalized by the maximum number of states in the two trajectories. We find that the alignment scores for states visited for the SoS model are not highly correlated with any single symbolic strategy.
</figcaption>
</figure>
<div id="S5.p2" class="ltx_para ltx_noindent">
<p id="S5.p2.2" class="ltx_p"><span id="S5.p2.2.1" class="ltx_text ltx_font_bold">Experiment Setup.</span>
For all experiments, we train a GPT-Neo model <cite class="ltx_cite ltx_citemacro_citep">(Gao et al., <a href="#bib.bib9" title="" class="ltx_ref">2020</a>)</cite> with 250M parameters and a context length of 4096 tokens (see appendix for more details).
We train models with a causal language modeling objective on the 500,000 Countdown problems in two conditions: 1) Optimal Paths (OP): the model is trained to predict the correct, and optimal path <math id="S5.p2.1.m1.1" class="ltx_Math" alttext="\mathcal{P}" display="inline"><semantics id="S5.p2.1.m1.1a"><mi class="ltx_font_mathcaligraphic" id="S5.p2.1.m1.1.1" xref="S5.p2.1.m1.1.1.cmml">𝒫</mi><annotation-xml encoding="MathML-Content" id="S5.p2.1.m1.1b"><ci id="S5.p2.1.m1.1.1.cmml" xref="S5.p2.1.m1.1.1">𝒫</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.p2.1.m1.1c">\mathcal{P}</annotation></semantics></math> for all problems in the dataset. 2) Stream of Search (SoS): The model is trained on search trajectories <math id="S5.p2.2.m2.1" class="ltx_Math" alttext="\mathcal{T}" display="inline"><semantics id="S5.p2.2.m2.1a"><mi class="ltx_font_mathcaligraphic" id="S5.p2.2.m2.1.1" xref="S5.p2.2.m2.1.1.cmml">𝒯</mi><annotation-xml encoding="MathML-Content" id="S5.p2.2.m2.1b"><ci id="S5.p2.2.m2.1.1.cmml" xref="S5.p2.2.m2.1.1">𝒯</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.p2.2.m2.1c">\mathcal{T}</annotation></semantics></math> sampled from different search strategies. The SoS dataset has 285,000 correct solutions while the OP dataset has 500,000 correct solutions. Both models are trained up to the same number of gradient steps.</p>
</div>
<div id="S5.p3" class="ltx_para ltx_noindent">
<p id="S5.p3.1" class="ltx_p"><span id="S5.p3.1.1" class="ltx_text ltx_font_bold">Results.</span> We find that the model trained on streams of search outperforms the model trained on the optimal solutions (see <a href="#S5.F3" title="Figure 3 ‣ 5 Learning from Suboptimal Search Strategies ‣ Stream of Search (SoS): Learning to Search in Language" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Fig. 3</span></a>a). The SoS model achieves an accuracy of 51.27% on held-out inputs compared to 25.73% of the OP model. A similar pattern is seen for the test set with held-out targets. This shows that despite having fewer ‘correct’ examples for training, the SoS model outperforms the OP model.
The SoS model has a slightly lower accuracy compared to the average accuracy of the symbolic search methods used to construct the SoS training dataset. However, it is worth noting that SoS is solving a harder problem: symbolic search relies on an environment model to expose state transitions, while the SoS LM simulates those transitions itself, effectively learning to search with an internal ‘world model’; the language model must also learn how to perform arithmetic operations. Despite these challenges, the SoS model generates valid trajectories with a low error rate in state exploration (0.8%) and only about 2 arithmetic errors per trajectory on average (see App. <a href="#A4.T2" title="Table 2 ‣ Appendix D Analysis of generated trajectories ‣ Stream of Search (SoS): Learning to Search in Language" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Tab. 2</span></a>).</p>
</div>
<div id="S5.p4" class="ltx_para ltx_noindent">
<p id="S5.p4.1" class="ltx_p">To understand the strategies that the trained SoS model uses, we measure the alignment of the model generated search trajectories with symbolic strategies. We find that the alignment scores for states visited for the SoS model are not highly correlated with any single symbolic strategy (see <a href="#S5.F3" title="Figure 3 ‣ 5 Learning from Suboptimal Search Strategies ‣ Stream of Search (SoS): Learning to Search in Language" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Fig. 3</span></a>c). The highest correlation of SoS is with DFS using the sum heuristic (0.57), and the lowest correlation is with BFS using a breadth size of 5 and the sum heuristic (0.27). A similar pattern is observed when we measure the alignments of correctness (see <a href="#S5.F3" title="Figure 3 ‣ 5 Learning from Suboptimal Search Strategies ‣ Stream of Search (SoS): Learning to Search in Language" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Fig. 3</span></a>b) for the SoS model and the symbolic search strategies. Overall, the SoS model has higher scores for alignment with strategies that use the sum heuristic, but does not seem to predominantly use any one strategy from its training data.</p>
</div>
</section>
<section id="S6" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">6 </span>Policy Improvement with Stream of Search</h2>

<div id="S6.p1" class="ltx_para ltx_noindent">
<p id="S6.p1.1" class="ltx_p">The SoS LM has learned to use search to solve new problems.
Can it learn to improve upon the symbolic strategies seen in its training data? In this section, we explore if the model can self-improve with feedback based on correctness and efficiency. We measure ability to solve previously unsolved problems (by the symbolic search strategies) from the training dataset and solve difficult problems from the training set that none of the symbolic search strategies can solve. To improve the model, we use two RL strategies, expert iteration using STaR <cite class="ltx_cite ltx_citemacro_citep">(Zelikman et al., <a href="#bib.bib27" title="" class="ltx_ref">2022</a>)</cite>, and Advantage-Induced Policy Alignment <cite class="ltx_cite ltx_citemacro_citep">(Zhu et al., <a href="#bib.bib28" title="" class="ltx_ref">2023</a>)</cite>.</p>
</div>
<div id="S6.p2" class="ltx_para ltx_noindent">
<p id="S6.p2.1" class="ltx_p"><span id="S6.p2.1.1" class="ltx_text ltx_font_bold">Experiment Setup.</span> To improve the models with STaR (<a href="#alg1" title="Algorithm 1 ‣ 6 Policy Improvement with Stream of Search ‣ Stream of Search (SoS): Learning to Search in Language" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Alg. 1</span></a>), we use problems in the training dataset to generate 100,000 correct trajectories. We sample with a temperature of <math id="S6.p2.1.m1.1" class="ltx_Math" alttext="0.8" display="inline"><semantics id="S6.p2.1.m1.1a"><mn id="S6.p2.1.m1.1.1" xref="S6.p2.1.m1.1.1.cmml">0.8</mn><annotation-xml encoding="MathML-Content" id="S6.p2.1.m1.1b"><cn type="float" id="S6.p2.1.m1.1.1.cmml" xref="S6.p2.1.m1.1.1">0.8</cn></annotation-xml><annotation encoding="application/x-tex" id="S6.p2.1.m1.1c">0.8</annotation></semantics></math>. These trajectories are then used to finetune the model. We repeat this process until we see a convergence in performance on the validation set.</p>
</div>
<figure id="alg1" class="ltx_float ltx_float_algorithm ltx_framed ltx_framed_top">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_float"><span id="alg1.2.1.1" class="ltx_text ltx_font_bold">Algorithm 1</span> </span> Expert Iteration with STaR <cite class="ltx_cite ltx_citemacro_citep">(Zelikman et al., <a href="#bib.bib27" title="" class="ltx_ref">2022</a>)</cite></figcaption>
<div id="alg1.3" class="ltx_listing ltx_listing">
<div id="alg1.l1" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">1:</span><span id="alg1.l1.1" class="ltx_text ltx_font_bold">Input <span id="alg1.l1.1.1" class="ltx_text ltx_font_medium">The SoS model trained on Stream of Search Dataset </span></span> <math id="alg1.l1.m1.1" class="ltx_Math" alttext="D=\{(x_{i},\mathcal{T}_{i})\}_{i=1}^{m}" display="inline"><semantics id="alg1.l1.m1.1a"><mrow id="alg1.l1.m1.1.1" xref="alg1.l1.m1.1.1.cmml"><mi id="alg1.l1.m1.1.1.3" xref="alg1.l1.m1.1.1.3.cmml">D</mi><mo id="alg1.l1.m1.1.1.2" xref="alg1.l1.m1.1.1.2.cmml">=</mo><msubsup id="alg1.l1.m1.1.1.1" xref="alg1.l1.m1.1.1.1.cmml"><mrow id="alg1.l1.m1.1.1.1.1.1.1" xref="alg1.l1.m1.1.1.1.1.1.2.cmml"><mo stretchy="false" id="alg1.l1.m1.1.1.1.1.1.1.2" xref="alg1.l1.m1.1.1.1.1.1.2.cmml">{</mo><mrow id="alg1.l1.m1.1.1.1.1.1.1.1.2" xref="alg1.l1.m1.1.1.1.1.1.1.1.3.cmml"><mo stretchy="false" id="alg1.l1.m1.1.1.1.1.1.1.1.2.3" xref="alg1.l1.m1.1.1.1.1.1.1.1.3.cmml">(</mo><msub id="alg1.l1.m1.1.1.1.1.1.1.1.1.1" xref="alg1.l1.m1.1.1.1.1.1.1.1.1.1.cmml"><mi id="alg1.l1.m1.1.1.1.1.1.1.1.1.1.2" xref="alg1.l1.m1.1.1.1.1.1.1.1.1.1.2.cmml">x</mi><mi id="alg1.l1.m1.1.1.1.1.1.1.1.1.1.3" xref="alg1.l1.m1.1.1.1.1.1.1.1.1.1.3.cmml">i</mi></msub><mo id="alg1.l1.m1.1.1.1.1.1.1.1.2.4" xref="alg1.l1.m1.1.1.1.1.1.1.1.3.cmml">,</mo><msub id="alg1.l1.m1.1.1.1.1.1.1.1.2.2" xref="alg1.l1.m1.1.1.1.1.1.1.1.2.2.cmml"><mi class="ltx_font_mathcaligraphic" id="alg1.l1.m1.1.1.1.1.1.1.1.2.2.2" xref="alg1.l1.m1.1.1.1.1.1.1.1.2.2.2.cmml">𝒯</mi><mi id="alg1.l1.m1.1.1.1.1.1.1.1.2.2.3" xref="alg1.l1.m1.1.1.1.1.1.1.1.2.2.3.cmml">i</mi></msub><mo stretchy="false" id="alg1.l1.m1.1.1.1.1.1.1.1.2.5" xref="alg1.l1.m1.1.1.1.1.1.1.1.3.cmml">)</mo></mrow><mo stretchy="false" id="alg1.l1.m1.1.1.1.1.1.1.3" xref="alg1.l1.m1.1.1.1.1.1.2.cmml">}</mo></mrow><mrow id="alg1.l1.m1.1.1.1.1.3" xref="alg1.l1.m1.1.1.1.1.3.cmml"><mi id="alg1.l1.m1.1.1.1.1.3.2" xref="alg1.l1.m1.1.1.1.1.3.2.cmml">i</mi><mo id="alg1.l1.m1.1.1.1.1.3.1" xref="alg1.l1.m1.1.1.1.1.3.1.cmml">=</mo><mn id="alg1.l1.m1.1.1.1.1.3.3" xref="alg1.l1.m1.1.1.1.1.3.3.cmml">1</mn></mrow><mi id="alg1.l1.m1.1.1.1.3" xref="alg1.l1.m1.1.1.1.3.cmml">m</mi></msubsup></mrow><annotation-xml encoding="MathML-Content" id="alg1.l1.m1.1b"><apply id="alg1.l1.m1.1.1.cmml" xref="alg1.l1.m1.1.1"><eq id="alg1.l1.m1.1.1.2.cmml" xref="alg1.l1.m1.1.1.2"></eq><ci id="alg1.l1.m1.1.1.3.cmml" xref="alg1.l1.m1.1.1.3">𝐷</ci><apply id="alg1.l1.m1.1.1.1.cmml" xref="alg1.l1.m1.1.1.1"><csymbol cd="ambiguous" id="alg1.l1.m1.1.1.1.2.cmml" xref="alg1.l1.m1.1.1.1">superscript</csymbol><apply id="alg1.l1.m1.1.1.1.1.cmml" xref="alg1.l1.m1.1.1.1"><csymbol cd="ambiguous" id="alg1.l1.m1.1.1.1.1.2.cmml" xref="alg1.l1.m1.1.1.1">subscript</csymbol><set id="alg1.l1.m1.1.1.1.1.1.2.cmml" xref="alg1.l1.m1.1.1.1.1.1.1"><interval closure="open" id="alg1.l1.m1.1.1.1.1.1.1.1.3.cmml" xref="alg1.l1.m1.1.1.1.1.1.1.1.2"><apply id="alg1.l1.m1.1.1.1.1.1.1.1.1.1.cmml" xref="alg1.l1.m1.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="alg1.l1.m1.1.1.1.1.1.1.1.1.1.1.cmml" xref="alg1.l1.m1.1.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="alg1.l1.m1.1.1.1.1.1.1.1.1.1.2.cmml" xref="alg1.l1.m1.1.1.1.1.1.1.1.1.1.2">𝑥</ci><ci id="alg1.l1.m1.1.1.1.1.1.1.1.1.1.3.cmml" xref="alg1.l1.m1.1.1.1.1.1.1.1.1.1.3">𝑖</ci></apply><apply id="alg1.l1.m1.1.1.1.1.1.1.1.2.2.cmml" xref="alg1.l1.m1.1.1.1.1.1.1.1.2.2"><csymbol cd="ambiguous" id="alg1.l1.m1.1.1.1.1.1.1.1.2.2.1.cmml" xref="alg1.l1.m1.1.1.1.1.1.1.1.2.2">subscript</csymbol><ci id="alg1.l1.m1.1.1.1.1.1.1.1.2.2.2.cmml" xref="alg1.l1.m1.1.1.1.1.1.1.1.2.2.2">𝒯</ci><ci id="alg1.l1.m1.1.1.1.1.1.1.1.2.2.3.cmml" xref="alg1.l1.m1.1.1.1.1.1.1.1.2.2.3">𝑖</ci></apply></interval></set><apply id="alg1.l1.m1.1.1.1.1.3.cmml" xref="alg1.l1.m1.1.1.1.1.3"><eq id="alg1.l1.m1.1.1.1.1.3.1.cmml" xref="alg1.l1.m1.1.1.1.1.3.1"></eq><ci id="alg1.l1.m1.1.1.1.1.3.2.cmml" xref="alg1.l1.m1.1.1.1.1.3.2">𝑖</ci><cn type="integer" id="alg1.l1.m1.1.1.1.1.3.3.cmml" xref="alg1.l1.m1.1.1.1.1.3.3">1</cn></apply></apply><ci id="alg1.l1.m1.1.1.1.3.cmml" xref="alg1.l1.m1.1.1.1.3">𝑚</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l1.m1.1c">D=\{(x_{i},\mathcal{T}_{i})\}_{i=1}^{m}</annotation></semantics></math>

</div>
<div id="alg1.l2" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">2:</span><math id="alg1.l2.m1.1" class="ltx_Math" alttext="M_{0}\leftarrow SoS" display="inline"><semantics id="alg1.l2.m1.1a"><mrow id="alg1.l2.m1.1.1" xref="alg1.l2.m1.1.1.cmml"><msub id="alg1.l2.m1.1.1.2" xref="alg1.l2.m1.1.1.2.cmml"><mi id="alg1.l2.m1.1.1.2.2" xref="alg1.l2.m1.1.1.2.2.cmml">M</mi><mn id="alg1.l2.m1.1.1.2.3" xref="alg1.l2.m1.1.1.2.3.cmml">0</mn></msub><mo stretchy="false" id="alg1.l2.m1.1.1.1" xref="alg1.l2.m1.1.1.1.cmml">←</mo><mrow id="alg1.l2.m1.1.1.3" xref="alg1.l2.m1.1.1.3.cmml"><mi id="alg1.l2.m1.1.1.3.2" xref="alg1.l2.m1.1.1.3.2.cmml">S</mi><mo lspace="0em" rspace="0em" id="alg1.l2.m1.1.1.3.1" xref="alg1.l2.m1.1.1.3.1.cmml">​</mo><mi id="alg1.l2.m1.1.1.3.3" xref="alg1.l2.m1.1.1.3.3.cmml">o</mi><mo lspace="0em" rspace="0em" id="alg1.l2.m1.1.1.3.1a" xref="alg1.l2.m1.1.1.3.1.cmml">​</mo><mi id="alg1.l2.m1.1.1.3.4" xref="alg1.l2.m1.1.1.3.4.cmml">S</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l2.m1.1b"><apply id="alg1.l2.m1.1.1.cmml" xref="alg1.l2.m1.1.1"><ci id="alg1.l2.m1.1.1.1.cmml" xref="alg1.l2.m1.1.1.1">←</ci><apply id="alg1.l2.m1.1.1.2.cmml" xref="alg1.l2.m1.1.1.2"><csymbol cd="ambiguous" id="alg1.l2.m1.1.1.2.1.cmml" xref="alg1.l2.m1.1.1.2">subscript</csymbol><ci id="alg1.l2.m1.1.1.2.2.cmml" xref="alg1.l2.m1.1.1.2.2">𝑀</ci><cn type="integer" id="alg1.l2.m1.1.1.2.3.cmml" xref="alg1.l2.m1.1.1.2.3">0</cn></apply><apply id="alg1.l2.m1.1.1.3.cmml" xref="alg1.l2.m1.1.1.3"><times id="alg1.l2.m1.1.1.3.1.cmml" xref="alg1.l2.m1.1.1.3.1"></times><ci id="alg1.l2.m1.1.1.3.2.cmml" xref="alg1.l2.m1.1.1.3.2">𝑆</ci><ci id="alg1.l2.m1.1.1.3.3.cmml" xref="alg1.l2.m1.1.1.3.3">𝑜</ci><ci id="alg1.l2.m1.1.1.3.4.cmml" xref="alg1.l2.m1.1.1.3.4">𝑆</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l2.m1.1c">M_{0}\leftarrow SoS</annotation></semantics></math> <span id="alg1.l2.1" class="ltx_text" style="float:right;"><math id="alg1.l2.1.m1.1" class="ltx_Math" alttext="\triangleright" display="inline"><semantics id="alg1.l2.1.m1.1a"><mo id="alg1.l2.1.m1.1.1" xref="alg1.l2.1.m1.1.1.cmml">▷</mo><annotation-xml encoding="MathML-Content" id="alg1.l2.1.m1.1b"><ci id="alg1.l2.1.m1.1.1.cmml" xref="alg1.l2.1.m1.1.1">▷</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l2.1.m1.1c">\triangleright</annotation></semantics></math> Initialize the SoS model
</span>
</div>
<div id="alg1.l3" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">3:</span><span id="alg1.l3.1" class="ltx_text ltx_font_bold">for</span> <math id="alg1.l3.m1.1" class="ltx_Math" alttext="n" display="inline"><semantics id="alg1.l3.m1.1a"><mi id="alg1.l3.m1.1.1" xref="alg1.l3.m1.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="alg1.l3.m1.1b"><ci id="alg1.l3.m1.1.1.cmml" xref="alg1.l3.m1.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l3.m1.1c">n</annotation></semantics></math> in <math id="alg1.l3.m2.1" class="ltx_Math" alttext="1\ldots N" display="inline"><semantics id="alg1.l3.m2.1a"><mrow id="alg1.l3.m2.1.1" xref="alg1.l3.m2.1.1.cmml"><mn id="alg1.l3.m2.1.1.2" xref="alg1.l3.m2.1.1.2.cmml">1</mn><mo lspace="0em" rspace="0em" id="alg1.l3.m2.1.1.1" xref="alg1.l3.m2.1.1.1.cmml">​</mo><mi mathvariant="normal" id="alg1.l3.m2.1.1.3" xref="alg1.l3.m2.1.1.3.cmml">…</mi><mo lspace="0em" rspace="0em" id="alg1.l3.m2.1.1.1a" xref="alg1.l3.m2.1.1.1.cmml">​</mo><mi id="alg1.l3.m2.1.1.4" xref="alg1.l3.m2.1.1.4.cmml">N</mi></mrow><annotation-xml encoding="MathML-Content" id="alg1.l3.m2.1b"><apply id="alg1.l3.m2.1.1.cmml" xref="alg1.l3.m2.1.1"><times id="alg1.l3.m2.1.1.1.cmml" xref="alg1.l3.m2.1.1.1"></times><cn type="integer" id="alg1.l3.m2.1.1.2.cmml" xref="alg1.l3.m2.1.1.2">1</cn><ci id="alg1.l3.m2.1.1.3.cmml" xref="alg1.l3.m2.1.1.3">…</ci><ci id="alg1.l3.m2.1.1.4.cmml" xref="alg1.l3.m2.1.1.4">𝑁</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l3.m2.1c">1\ldots N</annotation></semantics></math> <span id="alg1.l3.2" class="ltx_text ltx_font_bold">do</span>

</div>
<div id="alg1.l4" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">4:</span>     <math id="alg1.l4.m1.3" class="ltx_Math" alttext="\mathcal{T}_{i}\leftarrow M_{n-1}(x_{i})\text{ }\forall i\in[1,m]" display="inline"><semantics id="alg1.l4.m1.3a"><mrow id="alg1.l4.m1.3.3" xref="alg1.l4.m1.3.3.cmml"><msub id="alg1.l4.m1.3.3.3" xref="alg1.l4.m1.3.3.3.cmml"><mi class="ltx_font_mathcaligraphic" id="alg1.l4.m1.3.3.3.2" xref="alg1.l4.m1.3.3.3.2.cmml">𝒯</mi><mi id="alg1.l4.m1.3.3.3.3" xref="alg1.l4.m1.3.3.3.3.cmml">i</mi></msub><mo stretchy="false" id="alg1.l4.m1.3.3.4" xref="alg1.l4.m1.3.3.4.cmml">←</mo><mrow id="alg1.l4.m1.3.3.1" xref="alg1.l4.m1.3.3.1.cmml"><msub id="alg1.l4.m1.3.3.1.3" xref="alg1.l4.m1.3.3.1.3.cmml"><mi id="alg1.l4.m1.3.3.1.3.2" xref="alg1.l4.m1.3.3.1.3.2.cmml">M</mi><mrow id="alg1.l4.m1.3.3.1.3.3" xref="alg1.l4.m1.3.3.1.3.3.cmml"><mi id="alg1.l4.m1.3.3.1.3.3.2" xref="alg1.l4.m1.3.3.1.3.3.2.cmml">n</mi><mo id="alg1.l4.m1.3.3.1.3.3.1" xref="alg1.l4.m1.3.3.1.3.3.1.cmml">−</mo><mn id="alg1.l4.m1.3.3.1.3.3.3" xref="alg1.l4.m1.3.3.1.3.3.3.cmml">1</mn></mrow></msub><mo lspace="0em" rspace="0em" id="alg1.l4.m1.3.3.1.2" xref="alg1.l4.m1.3.3.1.2.cmml">​</mo><mrow id="alg1.l4.m1.3.3.1.1.1" xref="alg1.l4.m1.3.3.1.1.1.1.cmml"><mo stretchy="false" id="alg1.l4.m1.3.3.1.1.1.2" xref="alg1.l4.m1.3.3.1.1.1.1.cmml">(</mo><msub id="alg1.l4.m1.3.3.1.1.1.1" xref="alg1.l4.m1.3.3.1.1.1.1.cmml"><mi id="alg1.l4.m1.3.3.1.1.1.1.2" xref="alg1.l4.m1.3.3.1.1.1.1.2.cmml">x</mi><mi id="alg1.l4.m1.3.3.1.1.1.1.3" xref="alg1.l4.m1.3.3.1.1.1.1.3.cmml">i</mi></msub><mo stretchy="false" id="alg1.l4.m1.3.3.1.1.1.3" xref="alg1.l4.m1.3.3.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="alg1.l4.m1.3.3.1.2a" xref="alg1.l4.m1.3.3.1.2.cmml">​</mo><mtext id="alg1.l4.m1.3.3.1.4" xref="alg1.l4.m1.3.3.1.4a.cmml"> </mtext><mo lspace="0.167em" rspace="0em" id="alg1.l4.m1.3.3.1.2b" xref="alg1.l4.m1.3.3.1.2.cmml">​</mo><mrow id="alg1.l4.m1.3.3.1.5" xref="alg1.l4.m1.3.3.1.5.cmml"><mo rspace="0.167em" id="alg1.l4.m1.3.3.1.5.1" xref="alg1.l4.m1.3.3.1.5.1.cmml">∀</mo><mi id="alg1.l4.m1.3.3.1.5.2" xref="alg1.l4.m1.3.3.1.5.2.cmml">i</mi></mrow></mrow><mo id="alg1.l4.m1.3.3.5" xref="alg1.l4.m1.3.3.5.cmml">∈</mo><mrow id="alg1.l4.m1.3.3.6.2" xref="alg1.l4.m1.3.3.6.1.cmml"><mo stretchy="false" id="alg1.l4.m1.3.3.6.2.1" xref="alg1.l4.m1.3.3.6.1.cmml">[</mo><mn id="alg1.l4.m1.1.1" xref="alg1.l4.m1.1.1.cmml">1</mn><mo id="alg1.l4.m1.3.3.6.2.2" xref="alg1.l4.m1.3.3.6.1.cmml">,</mo><mi id="alg1.l4.m1.2.2" xref="alg1.l4.m1.2.2.cmml">m</mi><mo stretchy="false" id="alg1.l4.m1.3.3.6.2.3" xref="alg1.l4.m1.3.3.6.1.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l4.m1.3b"><apply id="alg1.l4.m1.3.3.cmml" xref="alg1.l4.m1.3.3"><and id="alg1.l4.m1.3.3a.cmml" xref="alg1.l4.m1.3.3"></and><apply id="alg1.l4.m1.3.3b.cmml" xref="alg1.l4.m1.3.3"><ci id="alg1.l4.m1.3.3.4.cmml" xref="alg1.l4.m1.3.3.4">←</ci><apply id="alg1.l4.m1.3.3.3.cmml" xref="alg1.l4.m1.3.3.3"><csymbol cd="ambiguous" id="alg1.l4.m1.3.3.3.1.cmml" xref="alg1.l4.m1.3.3.3">subscript</csymbol><ci id="alg1.l4.m1.3.3.3.2.cmml" xref="alg1.l4.m1.3.3.3.2">𝒯</ci><ci id="alg1.l4.m1.3.3.3.3.cmml" xref="alg1.l4.m1.3.3.3.3">𝑖</ci></apply><apply id="alg1.l4.m1.3.3.1.cmml" xref="alg1.l4.m1.3.3.1"><times id="alg1.l4.m1.3.3.1.2.cmml" xref="alg1.l4.m1.3.3.1.2"></times><apply id="alg1.l4.m1.3.3.1.3.cmml" xref="alg1.l4.m1.3.3.1.3"><csymbol cd="ambiguous" id="alg1.l4.m1.3.3.1.3.1.cmml" xref="alg1.l4.m1.3.3.1.3">subscript</csymbol><ci id="alg1.l4.m1.3.3.1.3.2.cmml" xref="alg1.l4.m1.3.3.1.3.2">𝑀</ci><apply id="alg1.l4.m1.3.3.1.3.3.cmml" xref="alg1.l4.m1.3.3.1.3.3"><minus id="alg1.l4.m1.3.3.1.3.3.1.cmml" xref="alg1.l4.m1.3.3.1.3.3.1"></minus><ci id="alg1.l4.m1.3.3.1.3.3.2.cmml" xref="alg1.l4.m1.3.3.1.3.3.2">𝑛</ci><cn type="integer" id="alg1.l4.m1.3.3.1.3.3.3.cmml" xref="alg1.l4.m1.3.3.1.3.3.3">1</cn></apply></apply><apply id="alg1.l4.m1.3.3.1.1.1.1.cmml" xref="alg1.l4.m1.3.3.1.1.1"><csymbol cd="ambiguous" id="alg1.l4.m1.3.3.1.1.1.1.1.cmml" xref="alg1.l4.m1.3.3.1.1.1">subscript</csymbol><ci id="alg1.l4.m1.3.3.1.1.1.1.2.cmml" xref="alg1.l4.m1.3.3.1.1.1.1.2">𝑥</ci><ci id="alg1.l4.m1.3.3.1.1.1.1.3.cmml" xref="alg1.l4.m1.3.3.1.1.1.1.3">𝑖</ci></apply><ci id="alg1.l4.m1.3.3.1.4a.cmml" xref="alg1.l4.m1.3.3.1.4"><mtext id="alg1.l4.m1.3.3.1.4.cmml" xref="alg1.l4.m1.3.3.1.4"> </mtext></ci><apply id="alg1.l4.m1.3.3.1.5.cmml" xref="alg1.l4.m1.3.3.1.5"><csymbol cd="latexml" id="alg1.l4.m1.3.3.1.5.1.cmml" xref="alg1.l4.m1.3.3.1.5.1">for-all</csymbol><ci id="alg1.l4.m1.3.3.1.5.2.cmml" xref="alg1.l4.m1.3.3.1.5.2">𝑖</ci></apply></apply></apply><apply id="alg1.l4.m1.3.3c.cmml" xref="alg1.l4.m1.3.3"><in id="alg1.l4.m1.3.3.5.cmml" xref="alg1.l4.m1.3.3.5"></in><share href="#alg1.l4.m1.3.3.1.cmml" id="alg1.l4.m1.3.3d.cmml" xref="alg1.l4.m1.3.3"></share><interval closure="closed" id="alg1.l4.m1.3.3.6.1.cmml" xref="alg1.l4.m1.3.3.6.2"><cn type="integer" id="alg1.l4.m1.1.1.cmml" xref="alg1.l4.m1.1.1">1</cn><ci id="alg1.l4.m1.2.2.cmml" xref="alg1.l4.m1.2.2">𝑚</ci></interval></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l4.m1.3c">\mathcal{T}_{i}\leftarrow M_{n-1}(x_{i})\text{ }\forall i\in[1,m]</annotation></semantics></math> <span id="alg1.l4.1" class="ltx_text" style="float:right;"><math id="alg1.l4.1.m1.1" class="ltx_Math" alttext="\triangleright" display="inline"><semantics id="alg1.l4.1.m1.1a"><mo id="alg1.l4.1.m1.1.1" xref="alg1.l4.1.m1.1.1.cmml">▷</mo><annotation-xml encoding="MathML-Content" id="alg1.l4.1.m1.1b"><ci id="alg1.l4.1.m1.1.1.cmml" xref="alg1.l4.1.m1.1.1">▷</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l4.1.m1.1c">\triangleright</annotation></semantics></math> Perform trajectory generation
</span>
</div>
<div id="alg1.l5" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">5:</span>     <math id="alg1.l5.m1.2" class="ltx_math_unparsed" alttext="D_{n}\leftarrow\{(x_{i},\mathcal{T}_{i})|i\in[1,m]\text{ s.t. }\mathcal{P}_{i}\in\mathcal{T}_{i}" display="inline"><semantics id="alg1.l5.m1.2a"><mrow id="alg1.l5.m1.2b"><msub id="alg1.l5.m1.2.3"><mi id="alg1.l5.m1.2.3.2">D</mi><mi id="alg1.l5.m1.2.3.3">n</mi></msub><mo stretchy="false" id="alg1.l5.m1.2.4">←</mo><mrow id="alg1.l5.m1.2.5"><mo stretchy="false" id="alg1.l5.m1.2.5.1">{</mo><mrow id="alg1.l5.m1.2.5.2"><mo stretchy="false" id="alg1.l5.m1.2.5.2.1">(</mo><msub id="alg1.l5.m1.2.5.2.2"><mi id="alg1.l5.m1.2.5.2.2.2">x</mi><mi id="alg1.l5.m1.2.5.2.2.3">i</mi></msub><mo id="alg1.l5.m1.2.5.2.3">,</mo><msub id="alg1.l5.m1.2.5.2.4"><mi class="ltx_font_mathcaligraphic" id="alg1.l5.m1.2.5.2.4.2">𝒯</mi><mi id="alg1.l5.m1.2.5.2.4.3">i</mi></msub><mo stretchy="false" id="alg1.l5.m1.2.5.2.5">)</mo></mrow><mo fence="false" rspace="0.167em" stretchy="false" id="alg1.l5.m1.2.5.3">|</mo><mi id="alg1.l5.m1.2.5.4">i</mi><mo id="alg1.l5.m1.2.5.5">∈</mo><mrow id="alg1.l5.m1.2.5.6"><mo stretchy="false" id="alg1.l5.m1.2.5.6.1">[</mo><mn id="alg1.l5.m1.1.1">1</mn><mo id="alg1.l5.m1.2.5.6.2">,</mo><mi id="alg1.l5.m1.2.2">m</mi><mo stretchy="false" id="alg1.l5.m1.2.5.6.3">]</mo></mrow><mtext id="alg1.l5.m1.2.5.7"> s.t. </mtext><msub id="alg1.l5.m1.2.5.8"><mi class="ltx_font_mathcaligraphic" id="alg1.l5.m1.2.5.8.2">𝒫</mi><mi id="alg1.l5.m1.2.5.8.3">i</mi></msub><mo id="alg1.l5.m1.2.5.9">∈</mo><msub id="alg1.l5.m1.2.5.10"><mi class="ltx_font_mathcaligraphic" id="alg1.l5.m1.2.5.10.2">𝒯</mi><mi id="alg1.l5.m1.2.5.10.3">i</mi></msub></mrow></mrow><annotation encoding="application/x-tex" id="alg1.l5.m1.2c">D_{n}\leftarrow\{(x_{i},\mathcal{T}_{i})|i\in[1,m]\text{ s.t. }\mathcal{P}_{i}\in\mathcal{T}_{i}</annotation></semantics></math> <span id="alg1.l5.1" class="ltx_text" style="float:right;"><math id="alg1.l5.1.m1.1" class="ltx_Math" alttext="\triangleright" display="inline"><semantics id="alg1.l5.1.m1.1a"><mo id="alg1.l5.1.m1.1.1" xref="alg1.l5.1.m1.1.1.cmml">▷</mo><annotation-xml encoding="MathML-Content" id="alg1.l5.1.m1.1b"><ci id="alg1.l5.1.m1.1.1.cmml" xref="alg1.l5.1.m1.1.1">▷</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l5.1.m1.1c">\triangleright</annotation></semantics></math> Filter trajectories based on correctness
</span>
</div>
<div id="alg1.l6" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">6:</span>     <math id="alg1.l6.m1.2" class="ltx_Math" alttext="M_{n}\leftarrow\text{train}(M_{0},D_{n})" display="inline"><semantics id="alg1.l6.m1.2a"><mrow id="alg1.l6.m1.2.2" xref="alg1.l6.m1.2.2.cmml"><msub id="alg1.l6.m1.2.2.4" xref="alg1.l6.m1.2.2.4.cmml"><mi id="alg1.l6.m1.2.2.4.2" xref="alg1.l6.m1.2.2.4.2.cmml">M</mi><mi id="alg1.l6.m1.2.2.4.3" xref="alg1.l6.m1.2.2.4.3.cmml">n</mi></msub><mo stretchy="false" id="alg1.l6.m1.2.2.3" xref="alg1.l6.m1.2.2.3.cmml">←</mo><mrow id="alg1.l6.m1.2.2.2" xref="alg1.l6.m1.2.2.2.cmml"><mtext id="alg1.l6.m1.2.2.2.4" xref="alg1.l6.m1.2.2.2.4a.cmml">train</mtext><mo lspace="0em" rspace="0em" id="alg1.l6.m1.2.2.2.3" xref="alg1.l6.m1.2.2.2.3.cmml">​</mo><mrow id="alg1.l6.m1.2.2.2.2.2" xref="alg1.l6.m1.2.2.2.2.3.cmml"><mo stretchy="false" id="alg1.l6.m1.2.2.2.2.2.3" xref="alg1.l6.m1.2.2.2.2.3.cmml">(</mo><msub id="alg1.l6.m1.1.1.1.1.1.1" xref="alg1.l6.m1.1.1.1.1.1.1.cmml"><mi id="alg1.l6.m1.1.1.1.1.1.1.2" xref="alg1.l6.m1.1.1.1.1.1.1.2.cmml">M</mi><mn id="alg1.l6.m1.1.1.1.1.1.1.3" xref="alg1.l6.m1.1.1.1.1.1.1.3.cmml">0</mn></msub><mo id="alg1.l6.m1.2.2.2.2.2.4" xref="alg1.l6.m1.2.2.2.2.3.cmml">,</mo><msub id="alg1.l6.m1.2.2.2.2.2.2" xref="alg1.l6.m1.2.2.2.2.2.2.cmml"><mi id="alg1.l6.m1.2.2.2.2.2.2.2" xref="alg1.l6.m1.2.2.2.2.2.2.2.cmml">D</mi><mi id="alg1.l6.m1.2.2.2.2.2.2.3" xref="alg1.l6.m1.2.2.2.2.2.2.3.cmml">n</mi></msub><mo stretchy="false" id="alg1.l6.m1.2.2.2.2.2.5" xref="alg1.l6.m1.2.2.2.2.3.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l6.m1.2b"><apply id="alg1.l6.m1.2.2.cmml" xref="alg1.l6.m1.2.2"><ci id="alg1.l6.m1.2.2.3.cmml" xref="alg1.l6.m1.2.2.3">←</ci><apply id="alg1.l6.m1.2.2.4.cmml" xref="alg1.l6.m1.2.2.4"><csymbol cd="ambiguous" id="alg1.l6.m1.2.2.4.1.cmml" xref="alg1.l6.m1.2.2.4">subscript</csymbol><ci id="alg1.l6.m1.2.2.4.2.cmml" xref="alg1.l6.m1.2.2.4.2">𝑀</ci><ci id="alg1.l6.m1.2.2.4.3.cmml" xref="alg1.l6.m1.2.2.4.3">𝑛</ci></apply><apply id="alg1.l6.m1.2.2.2.cmml" xref="alg1.l6.m1.2.2.2"><times id="alg1.l6.m1.2.2.2.3.cmml" xref="alg1.l6.m1.2.2.2.3"></times><ci id="alg1.l6.m1.2.2.2.4a.cmml" xref="alg1.l6.m1.2.2.2.4"><mtext id="alg1.l6.m1.2.2.2.4.cmml" xref="alg1.l6.m1.2.2.2.4">train</mtext></ci><interval closure="open" id="alg1.l6.m1.2.2.2.2.3.cmml" xref="alg1.l6.m1.2.2.2.2.2"><apply id="alg1.l6.m1.1.1.1.1.1.1.cmml" xref="alg1.l6.m1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="alg1.l6.m1.1.1.1.1.1.1.1.cmml" xref="alg1.l6.m1.1.1.1.1.1.1">subscript</csymbol><ci id="alg1.l6.m1.1.1.1.1.1.1.2.cmml" xref="alg1.l6.m1.1.1.1.1.1.1.2">𝑀</ci><cn type="integer" id="alg1.l6.m1.1.1.1.1.1.1.3.cmml" xref="alg1.l6.m1.1.1.1.1.1.1.3">0</cn></apply><apply id="alg1.l6.m1.2.2.2.2.2.2.cmml" xref="alg1.l6.m1.2.2.2.2.2.2"><csymbol cd="ambiguous" id="alg1.l6.m1.2.2.2.2.2.2.1.cmml" xref="alg1.l6.m1.2.2.2.2.2.2">subscript</csymbol><ci id="alg1.l6.m1.2.2.2.2.2.2.2.cmml" xref="alg1.l6.m1.2.2.2.2.2.2.2">𝐷</ci><ci id="alg1.l6.m1.2.2.2.2.2.2.3.cmml" xref="alg1.l6.m1.2.2.2.2.2.2.3">𝑛</ci></apply></interval></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l6.m1.2c">M_{n}\leftarrow\text{train}(M_{0},D_{n})</annotation></semantics></math> <span id="alg1.l6.1" class="ltx_text" style="float:right;"><math id="alg1.l6.1.m1.1" class="ltx_Math" alttext="\triangleright" display="inline"><semantics id="alg1.l6.1.m1.1a"><mo id="alg1.l6.1.m1.1.1" xref="alg1.l6.1.m1.1.1.cmml">▷</mo><annotation-xml encoding="MathML-Content" id="alg1.l6.1.m1.1b"><ci id="alg1.l6.1.m1.1.1.cmml" xref="alg1.l6.1.m1.1.1">▷</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l6.1.m1.1c">\triangleright</annotation></semantics></math> Finetune the model on correct solutions
</span>
</div>
<div id="alg1.l7" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">7:</span><span id="alg1.l7.1" class="ltx_text ltx_font_bold">end</span> <span id="alg1.l7.2" class="ltx_text ltx_font_bold">for</span>
</div>
</div>
</figure>
<div id="S6.p3" class="ltx_para ltx_noindent">
<p id="S6.p3.1" class="ltx_p">Alternatively, we can use advantage-induced policy alignment (APA; <a href="#alg2" title="Algorithm 2 ‣ 6 Policy Improvement with Stream of Search ‣ Stream of Search (SoS): Learning to Search in Language" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Alg. 2</span></a>). APA is an Actor-Critic reinforcement learning technique that involves creating a copy of the language model to serve as a value network that then used to enhance the policy, the original language model. We define a straightforward reward function that takes into account the correctness and length of the generated trajectory. We try APA, as we wanted to see if using a separate value network would improve exploration and the path-stitching abilities of the language model. We chose APA over other methods like Proximal Policy Optimization (PPO) due to its stability and robustness to changes in hyperparameters.</p>
</div>
<div id="S6.p4" class="ltx_para ltx_noindent">
<p id="S6.p4.3" class="ltx_p">APA uses a reference policy, <math id="S6.p4.1.m1.1" class="ltx_Math" alttext="\pi_{ref}" display="inline"><semantics id="S6.p4.1.m1.1a"><msub id="S6.p4.1.m1.1.1" xref="S6.p4.1.m1.1.1.cmml"><mi id="S6.p4.1.m1.1.1.2" xref="S6.p4.1.m1.1.1.2.cmml">π</mi><mrow id="S6.p4.1.m1.1.1.3" xref="S6.p4.1.m1.1.1.3.cmml"><mi id="S6.p4.1.m1.1.1.3.2" xref="S6.p4.1.m1.1.1.3.2.cmml">r</mi><mo lspace="0em" rspace="0em" id="S6.p4.1.m1.1.1.3.1" xref="S6.p4.1.m1.1.1.3.1.cmml">​</mo><mi id="S6.p4.1.m1.1.1.3.3" xref="S6.p4.1.m1.1.1.3.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="S6.p4.1.m1.1.1.3.1a" xref="S6.p4.1.m1.1.1.3.1.cmml">​</mo><mi id="S6.p4.1.m1.1.1.3.4" xref="S6.p4.1.m1.1.1.3.4.cmml">f</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S6.p4.1.m1.1b"><apply id="S6.p4.1.m1.1.1.cmml" xref="S6.p4.1.m1.1.1"><csymbol cd="ambiguous" id="S6.p4.1.m1.1.1.1.cmml" xref="S6.p4.1.m1.1.1">subscript</csymbol><ci id="S6.p4.1.m1.1.1.2.cmml" xref="S6.p4.1.m1.1.1.2">𝜋</ci><apply id="S6.p4.1.m1.1.1.3.cmml" xref="S6.p4.1.m1.1.1.3"><times id="S6.p4.1.m1.1.1.3.1.cmml" xref="S6.p4.1.m1.1.1.3.1"></times><ci id="S6.p4.1.m1.1.1.3.2.cmml" xref="S6.p4.1.m1.1.1.3.2">𝑟</ci><ci id="S6.p4.1.m1.1.1.3.3.cmml" xref="S6.p4.1.m1.1.1.3.3">𝑒</ci><ci id="S6.p4.1.m1.1.1.3.4.cmml" xref="S6.p4.1.m1.1.1.3.4">𝑓</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.p4.1.m1.1c">\pi_{ref}</annotation></semantics></math>, to prevent the policy from drifting from its initial state. We observe that updating the reference policy whenever the validation reward converges results in further policy improvement (see <a href="#S6.F4" title="Figure 4 ‣ 6 Policy Improvement with Stream of Search ‣ Stream of Search (SoS): Learning to Search in Language" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Fig. 4</span></a>b). This shifting of the reference distribution can be interpreted as a means to reduce the weight assigned to staying close to the reference distribution (the <math id="S6.p4.2.m2.1" class="ltx_Math" alttext="\lambda" display="inline"><semantics id="S6.p4.2.m2.1a"><mi id="S6.p4.2.m2.1.1" xref="S6.p4.2.m2.1.1.cmml">λ</mi><annotation-xml encoding="MathML-Content" id="S6.p4.2.m2.1b"><ci id="S6.p4.2.m2.1.1.cmml" xref="S6.p4.2.m2.1.1">𝜆</ci></annotation-xml><annotation encoding="application/x-tex" id="S6.p4.2.m2.1c">\lambda</annotation></semantics></math> parameter in the APA objective). In practice, we found the strategy of shifting the reference distribution to be more stable for training when compared to designing a schedule for reducing <math id="S6.p4.3.m3.1" class="ltx_Math" alttext="\lambda" display="inline"><semantics id="S6.p4.3.m3.1a"><mi id="S6.p4.3.m3.1.1" xref="S6.p4.3.m3.1.1.cmml">λ</mi><annotation-xml encoding="MathML-Content" id="S6.p4.3.m3.1b"><ci id="S6.p4.3.m3.1.1.cmml" xref="S6.p4.3.m3.1.1">𝜆</ci></annotation-xml><annotation encoding="application/x-tex" id="S6.p4.3.m3.1c">\lambda</annotation></semantics></math> over training.</p>
</div>
<figure id="alg2" class="ltx_float ltx_float_algorithm ltx_framed ltx_framed_top">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_float"><span id="alg2.2.1.1" class="ltx_text ltx_font_bold">Algorithm 2</span> </span> Advantage Indued Policy-Alignment (APA) <cite class="ltx_cite ltx_citemacro_citep">(Zhu et al., <a href="#bib.bib28" title="" class="ltx_ref">2023</a>)</cite></figcaption>
<div id="alg2.3" class="ltx_listing ltx_listing">
<div id="alg2.l1" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">1:</span><span id="alg2.l1.1" class="ltx_text ltx_font_bold">Input: </span> An initial policy parameter <math id="alg2.l1.m1.1" class="ltx_Math" alttext="\pi_{init}" display="inline"><semantics id="alg2.l1.m1.1a"><msub id="alg2.l1.m1.1.1" xref="alg2.l1.m1.1.1.cmml"><mi id="alg2.l1.m1.1.1.2" xref="alg2.l1.m1.1.1.2.cmml">π</mi><mrow id="alg2.l1.m1.1.1.3" xref="alg2.l1.m1.1.1.3.cmml"><mi id="alg2.l1.m1.1.1.3.2" xref="alg2.l1.m1.1.1.3.2.cmml">i</mi><mo lspace="0em" rspace="0em" id="alg2.l1.m1.1.1.3.1" xref="alg2.l1.m1.1.1.3.1.cmml">​</mo><mi id="alg2.l1.m1.1.1.3.3" xref="alg2.l1.m1.1.1.3.3.cmml">n</mi><mo lspace="0em" rspace="0em" id="alg2.l1.m1.1.1.3.1a" xref="alg2.l1.m1.1.1.3.1.cmml">​</mo><mi id="alg2.l1.m1.1.1.3.4" xref="alg2.l1.m1.1.1.3.4.cmml">i</mi><mo lspace="0em" rspace="0em" id="alg2.l1.m1.1.1.3.1b" xref="alg2.l1.m1.1.1.3.1.cmml">​</mo><mi id="alg2.l1.m1.1.1.3.5" xref="alg2.l1.m1.1.1.3.5.cmml">t</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="alg2.l1.m1.1b"><apply id="alg2.l1.m1.1.1.cmml" xref="alg2.l1.m1.1.1"><csymbol cd="ambiguous" id="alg2.l1.m1.1.1.1.cmml" xref="alg2.l1.m1.1.1">subscript</csymbol><ci id="alg2.l1.m1.1.1.2.cmml" xref="alg2.l1.m1.1.1.2">𝜋</ci><apply id="alg2.l1.m1.1.1.3.cmml" xref="alg2.l1.m1.1.1.3"><times id="alg2.l1.m1.1.1.3.1.cmml" xref="alg2.l1.m1.1.1.3.1"></times><ci id="alg2.l1.m1.1.1.3.2.cmml" xref="alg2.l1.m1.1.1.3.2">𝑖</ci><ci id="alg2.l1.m1.1.1.3.3.cmml" xref="alg2.l1.m1.1.1.3.3">𝑛</ci><ci id="alg2.l1.m1.1.1.3.4.cmml" xref="alg2.l1.m1.1.1.3.4">𝑖</ci><ci id="alg2.l1.m1.1.1.3.5.cmml" xref="alg2.l1.m1.1.1.3.5">𝑡</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l1.m1.1c">\pi_{init}</annotation></semantics></math>, a given reward function <math id="alg2.l1.m2.1" class="ltx_Math" alttext="R" display="inline"><semantics id="alg2.l1.m2.1a"><mi id="alg2.l1.m2.1.1" xref="alg2.l1.m2.1.1.cmml">R</mi><annotation-xml encoding="MathML-Content" id="alg2.l1.m2.1b"><ci id="alg2.l1.m2.1.1.cmml" xref="alg2.l1.m2.1.1">𝑅</ci></annotation-xml><annotation encoding="application/x-tex" id="alg2.l1.m2.1c">R</annotation></semantics></math>, Advantage coefficient <math id="alg2.l1.m3.1" class="ltx_Math" alttext="\lambda" display="inline"><semantics id="alg2.l1.m3.1a"><mi id="alg2.l1.m3.1.1" xref="alg2.l1.m3.1.1.cmml">λ</mi><annotation-xml encoding="MathML-Content" id="alg2.l1.m3.1b"><ci id="alg2.l1.m3.1.1.cmml" xref="alg2.l1.m3.1.1">𝜆</ci></annotation-xml><annotation encoding="application/x-tex" id="alg2.l1.m3.1c">\lambda</annotation></semantics></math>.

</div>
<div id="alg2.l2" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">2:</span><math id="alg2.l2.m1.1" class="ltx_Math" alttext="\pi_{0}\leftarrow\pi_{init}" display="inline"><semantics id="alg2.l2.m1.1a"><mrow id="alg2.l2.m1.1.1" xref="alg2.l2.m1.1.1.cmml"><msub id="alg2.l2.m1.1.1.2" xref="alg2.l2.m1.1.1.2.cmml"><mi id="alg2.l2.m1.1.1.2.2" xref="alg2.l2.m1.1.1.2.2.cmml">π</mi><mn id="alg2.l2.m1.1.1.2.3" xref="alg2.l2.m1.1.1.2.3.cmml">0</mn></msub><mo stretchy="false" id="alg2.l2.m1.1.1.1" xref="alg2.l2.m1.1.1.1.cmml">←</mo><msub id="alg2.l2.m1.1.1.3" xref="alg2.l2.m1.1.1.3.cmml"><mi id="alg2.l2.m1.1.1.3.2" xref="alg2.l2.m1.1.1.3.2.cmml">π</mi><mrow id="alg2.l2.m1.1.1.3.3" xref="alg2.l2.m1.1.1.3.3.cmml"><mi id="alg2.l2.m1.1.1.3.3.2" xref="alg2.l2.m1.1.1.3.3.2.cmml">i</mi><mo lspace="0em" rspace="0em" id="alg2.l2.m1.1.1.3.3.1" xref="alg2.l2.m1.1.1.3.3.1.cmml">​</mo><mi id="alg2.l2.m1.1.1.3.3.3" xref="alg2.l2.m1.1.1.3.3.3.cmml">n</mi><mo lspace="0em" rspace="0em" id="alg2.l2.m1.1.1.3.3.1a" xref="alg2.l2.m1.1.1.3.3.1.cmml">​</mo><mi id="alg2.l2.m1.1.1.3.3.4" xref="alg2.l2.m1.1.1.3.3.4.cmml">i</mi><mo lspace="0em" rspace="0em" id="alg2.l2.m1.1.1.3.3.1b" xref="alg2.l2.m1.1.1.3.3.1.cmml">​</mo><mi id="alg2.l2.m1.1.1.3.3.5" xref="alg2.l2.m1.1.1.3.3.5.cmml">t</mi></mrow></msub></mrow><annotation-xml encoding="MathML-Content" id="alg2.l2.m1.1b"><apply id="alg2.l2.m1.1.1.cmml" xref="alg2.l2.m1.1.1"><ci id="alg2.l2.m1.1.1.1.cmml" xref="alg2.l2.m1.1.1.1">←</ci><apply id="alg2.l2.m1.1.1.2.cmml" xref="alg2.l2.m1.1.1.2"><csymbol cd="ambiguous" id="alg2.l2.m1.1.1.2.1.cmml" xref="alg2.l2.m1.1.1.2">subscript</csymbol><ci id="alg2.l2.m1.1.1.2.2.cmml" xref="alg2.l2.m1.1.1.2.2">𝜋</ci><cn type="integer" id="alg2.l2.m1.1.1.2.3.cmml" xref="alg2.l2.m1.1.1.2.3">0</cn></apply><apply id="alg2.l2.m1.1.1.3.cmml" xref="alg2.l2.m1.1.1.3"><csymbol cd="ambiguous" id="alg2.l2.m1.1.1.3.1.cmml" xref="alg2.l2.m1.1.1.3">subscript</csymbol><ci id="alg2.l2.m1.1.1.3.2.cmml" xref="alg2.l2.m1.1.1.3.2">𝜋</ci><apply id="alg2.l2.m1.1.1.3.3.cmml" xref="alg2.l2.m1.1.1.3.3"><times id="alg2.l2.m1.1.1.3.3.1.cmml" xref="alg2.l2.m1.1.1.3.3.1"></times><ci id="alg2.l2.m1.1.1.3.3.2.cmml" xref="alg2.l2.m1.1.1.3.3.2">𝑖</ci><ci id="alg2.l2.m1.1.1.3.3.3.cmml" xref="alg2.l2.m1.1.1.3.3.3">𝑛</ci><ci id="alg2.l2.m1.1.1.3.3.4.cmml" xref="alg2.l2.m1.1.1.3.3.4">𝑖</ci><ci id="alg2.l2.m1.1.1.3.3.5.cmml" xref="alg2.l2.m1.1.1.3.3.5">𝑡</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l2.m1.1c">\pi_{0}\leftarrow\pi_{init}</annotation></semantics></math>

</div>
<div id="alg2.l3" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">3:</span><math id="alg2.l3.m1.1" class="ltx_Math" alttext="\pi_{ref}\leftarrow\pi_{init}" display="inline"><semantics id="alg2.l3.m1.1a"><mrow id="alg2.l3.m1.1.1" xref="alg2.l3.m1.1.1.cmml"><msub id="alg2.l3.m1.1.1.2" xref="alg2.l3.m1.1.1.2.cmml"><mi id="alg2.l3.m1.1.1.2.2" xref="alg2.l3.m1.1.1.2.2.cmml">π</mi><mrow id="alg2.l3.m1.1.1.2.3" xref="alg2.l3.m1.1.1.2.3.cmml"><mi id="alg2.l3.m1.1.1.2.3.2" xref="alg2.l3.m1.1.1.2.3.2.cmml">r</mi><mo lspace="0em" rspace="0em" id="alg2.l3.m1.1.1.2.3.1" xref="alg2.l3.m1.1.1.2.3.1.cmml">​</mo><mi id="alg2.l3.m1.1.1.2.3.3" xref="alg2.l3.m1.1.1.2.3.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg2.l3.m1.1.1.2.3.1a" xref="alg2.l3.m1.1.1.2.3.1.cmml">​</mo><mi id="alg2.l3.m1.1.1.2.3.4" xref="alg2.l3.m1.1.1.2.3.4.cmml">f</mi></mrow></msub><mo stretchy="false" id="alg2.l3.m1.1.1.1" xref="alg2.l3.m1.1.1.1.cmml">←</mo><msub id="alg2.l3.m1.1.1.3" xref="alg2.l3.m1.1.1.3.cmml"><mi id="alg2.l3.m1.1.1.3.2" xref="alg2.l3.m1.1.1.3.2.cmml">π</mi><mrow id="alg2.l3.m1.1.1.3.3" xref="alg2.l3.m1.1.1.3.3.cmml"><mi id="alg2.l3.m1.1.1.3.3.2" xref="alg2.l3.m1.1.1.3.3.2.cmml">i</mi><mo lspace="0em" rspace="0em" id="alg2.l3.m1.1.1.3.3.1" xref="alg2.l3.m1.1.1.3.3.1.cmml">​</mo><mi id="alg2.l3.m1.1.1.3.3.3" xref="alg2.l3.m1.1.1.3.3.3.cmml">n</mi><mo lspace="0em" rspace="0em" id="alg2.l3.m1.1.1.3.3.1a" xref="alg2.l3.m1.1.1.3.3.1.cmml">​</mo><mi id="alg2.l3.m1.1.1.3.3.4" xref="alg2.l3.m1.1.1.3.3.4.cmml">i</mi><mo lspace="0em" rspace="0em" id="alg2.l3.m1.1.1.3.3.1b" xref="alg2.l3.m1.1.1.3.3.1.cmml">​</mo><mi id="alg2.l3.m1.1.1.3.3.5" xref="alg2.l3.m1.1.1.3.3.5.cmml">t</mi></mrow></msub></mrow><annotation-xml encoding="MathML-Content" id="alg2.l3.m1.1b"><apply id="alg2.l3.m1.1.1.cmml" xref="alg2.l3.m1.1.1"><ci id="alg2.l3.m1.1.1.1.cmml" xref="alg2.l3.m1.1.1.1">←</ci><apply id="alg2.l3.m1.1.1.2.cmml" xref="alg2.l3.m1.1.1.2"><csymbol cd="ambiguous" id="alg2.l3.m1.1.1.2.1.cmml" xref="alg2.l3.m1.1.1.2">subscript</csymbol><ci id="alg2.l3.m1.1.1.2.2.cmml" xref="alg2.l3.m1.1.1.2.2">𝜋</ci><apply id="alg2.l3.m1.1.1.2.3.cmml" xref="alg2.l3.m1.1.1.2.3"><times id="alg2.l3.m1.1.1.2.3.1.cmml" xref="alg2.l3.m1.1.1.2.3.1"></times><ci id="alg2.l3.m1.1.1.2.3.2.cmml" xref="alg2.l3.m1.1.1.2.3.2">𝑟</ci><ci id="alg2.l3.m1.1.1.2.3.3.cmml" xref="alg2.l3.m1.1.1.2.3.3">𝑒</ci><ci id="alg2.l3.m1.1.1.2.3.4.cmml" xref="alg2.l3.m1.1.1.2.3.4">𝑓</ci></apply></apply><apply id="alg2.l3.m1.1.1.3.cmml" xref="alg2.l3.m1.1.1.3"><csymbol cd="ambiguous" id="alg2.l3.m1.1.1.3.1.cmml" xref="alg2.l3.m1.1.1.3">subscript</csymbol><ci id="alg2.l3.m1.1.1.3.2.cmml" xref="alg2.l3.m1.1.1.3.2">𝜋</ci><apply id="alg2.l3.m1.1.1.3.3.cmml" xref="alg2.l3.m1.1.1.3.3"><times id="alg2.l3.m1.1.1.3.3.1.cmml" xref="alg2.l3.m1.1.1.3.3.1"></times><ci id="alg2.l3.m1.1.1.3.3.2.cmml" xref="alg2.l3.m1.1.1.3.3.2">𝑖</ci><ci id="alg2.l3.m1.1.1.3.3.3.cmml" xref="alg2.l3.m1.1.1.3.3.3">𝑛</ci><ci id="alg2.l3.m1.1.1.3.3.4.cmml" xref="alg2.l3.m1.1.1.3.3.4">𝑖</ci><ci id="alg2.l3.m1.1.1.3.3.5.cmml" xref="alg2.l3.m1.1.1.3.3.5">𝑡</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l3.m1.1c">\pi_{ref}\leftarrow\pi_{init}</annotation></semantics></math> <span id="alg2.l3.1" class="ltx_text" style="float:right;"><math id="alg2.l3.1.m1.1" class="ltx_Math" alttext="\triangleright" display="inline"><semantics id="alg2.l3.1.m1.1a"><mo id="alg2.l3.1.m1.1.1" xref="alg2.l3.1.m1.1.1.cmml">▷</mo><annotation-xml encoding="MathML-Content" id="alg2.l3.1.m1.1b"><ci id="alg2.l3.1.m1.1.1.cmml" xref="alg2.l3.1.m1.1.1">▷</ci></annotation-xml><annotation encoding="application/x-tex" id="alg2.l3.1.m1.1c">\triangleright</annotation></semantics></math> Copy the SoS model to create a reference network.
</span>
</div>
<div id="alg2.l4" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">4:</span><math id="alg2.l4.m1.1" class="ltx_Math" alttext="\pi_{value}\leftarrow\pi_{init}" display="inline"><semantics id="alg2.l4.m1.1a"><mrow id="alg2.l4.m1.1.1" xref="alg2.l4.m1.1.1.cmml"><msub id="alg2.l4.m1.1.1.2" xref="alg2.l4.m1.1.1.2.cmml"><mi id="alg2.l4.m1.1.1.2.2" xref="alg2.l4.m1.1.1.2.2.cmml">π</mi><mrow id="alg2.l4.m1.1.1.2.3" xref="alg2.l4.m1.1.1.2.3.cmml"><mi id="alg2.l4.m1.1.1.2.3.2" xref="alg2.l4.m1.1.1.2.3.2.cmml">v</mi><mo lspace="0em" rspace="0em" id="alg2.l4.m1.1.1.2.3.1" xref="alg2.l4.m1.1.1.2.3.1.cmml">​</mo><mi id="alg2.l4.m1.1.1.2.3.3" xref="alg2.l4.m1.1.1.2.3.3.cmml">a</mi><mo lspace="0em" rspace="0em" id="alg2.l4.m1.1.1.2.3.1a" xref="alg2.l4.m1.1.1.2.3.1.cmml">​</mo><mi id="alg2.l4.m1.1.1.2.3.4" xref="alg2.l4.m1.1.1.2.3.4.cmml">l</mi><mo lspace="0em" rspace="0em" id="alg2.l4.m1.1.1.2.3.1b" xref="alg2.l4.m1.1.1.2.3.1.cmml">​</mo><mi id="alg2.l4.m1.1.1.2.3.5" xref="alg2.l4.m1.1.1.2.3.5.cmml">u</mi><mo lspace="0em" rspace="0em" id="alg2.l4.m1.1.1.2.3.1c" xref="alg2.l4.m1.1.1.2.3.1.cmml">​</mo><mi id="alg2.l4.m1.1.1.2.3.6" xref="alg2.l4.m1.1.1.2.3.6.cmml">e</mi></mrow></msub><mo stretchy="false" id="alg2.l4.m1.1.1.1" xref="alg2.l4.m1.1.1.1.cmml">←</mo><msub id="alg2.l4.m1.1.1.3" xref="alg2.l4.m1.1.1.3.cmml"><mi id="alg2.l4.m1.1.1.3.2" xref="alg2.l4.m1.1.1.3.2.cmml">π</mi><mrow id="alg2.l4.m1.1.1.3.3" xref="alg2.l4.m1.1.1.3.3.cmml"><mi id="alg2.l4.m1.1.1.3.3.2" xref="alg2.l4.m1.1.1.3.3.2.cmml">i</mi><mo lspace="0em" rspace="0em" id="alg2.l4.m1.1.1.3.3.1" xref="alg2.l4.m1.1.1.3.3.1.cmml">​</mo><mi id="alg2.l4.m1.1.1.3.3.3" xref="alg2.l4.m1.1.1.3.3.3.cmml">n</mi><mo lspace="0em" rspace="0em" id="alg2.l4.m1.1.1.3.3.1a" xref="alg2.l4.m1.1.1.3.3.1.cmml">​</mo><mi id="alg2.l4.m1.1.1.3.3.4" xref="alg2.l4.m1.1.1.3.3.4.cmml">i</mi><mo lspace="0em" rspace="0em" id="alg2.l4.m1.1.1.3.3.1b" xref="alg2.l4.m1.1.1.3.3.1.cmml">​</mo><mi id="alg2.l4.m1.1.1.3.3.5" xref="alg2.l4.m1.1.1.3.3.5.cmml">t</mi></mrow></msub></mrow><annotation-xml encoding="MathML-Content" id="alg2.l4.m1.1b"><apply id="alg2.l4.m1.1.1.cmml" xref="alg2.l4.m1.1.1"><ci id="alg2.l4.m1.1.1.1.cmml" xref="alg2.l4.m1.1.1.1">←</ci><apply id="alg2.l4.m1.1.1.2.cmml" xref="alg2.l4.m1.1.1.2"><csymbol cd="ambiguous" id="alg2.l4.m1.1.1.2.1.cmml" xref="alg2.l4.m1.1.1.2">subscript</csymbol><ci id="alg2.l4.m1.1.1.2.2.cmml" xref="alg2.l4.m1.1.1.2.2">𝜋</ci><apply id="alg2.l4.m1.1.1.2.3.cmml" xref="alg2.l4.m1.1.1.2.3"><times id="alg2.l4.m1.1.1.2.3.1.cmml" xref="alg2.l4.m1.1.1.2.3.1"></times><ci id="alg2.l4.m1.1.1.2.3.2.cmml" xref="alg2.l4.m1.1.1.2.3.2">𝑣</ci><ci id="alg2.l4.m1.1.1.2.3.3.cmml" xref="alg2.l4.m1.1.1.2.3.3">𝑎</ci><ci id="alg2.l4.m1.1.1.2.3.4.cmml" xref="alg2.l4.m1.1.1.2.3.4">𝑙</ci><ci id="alg2.l4.m1.1.1.2.3.5.cmml" xref="alg2.l4.m1.1.1.2.3.5">𝑢</ci><ci id="alg2.l4.m1.1.1.2.3.6.cmml" xref="alg2.l4.m1.1.1.2.3.6">𝑒</ci></apply></apply><apply id="alg2.l4.m1.1.1.3.cmml" xref="alg2.l4.m1.1.1.3"><csymbol cd="ambiguous" id="alg2.l4.m1.1.1.3.1.cmml" xref="alg2.l4.m1.1.1.3">subscript</csymbol><ci id="alg2.l4.m1.1.1.3.2.cmml" xref="alg2.l4.m1.1.1.3.2">𝜋</ci><apply id="alg2.l4.m1.1.1.3.3.cmml" xref="alg2.l4.m1.1.1.3.3"><times id="alg2.l4.m1.1.1.3.3.1.cmml" xref="alg2.l4.m1.1.1.3.3.1"></times><ci id="alg2.l4.m1.1.1.3.3.2.cmml" xref="alg2.l4.m1.1.1.3.3.2">𝑖</ci><ci id="alg2.l4.m1.1.1.3.3.3.cmml" xref="alg2.l4.m1.1.1.3.3.3">𝑛</ci><ci id="alg2.l4.m1.1.1.3.3.4.cmml" xref="alg2.l4.m1.1.1.3.3.4">𝑖</ci><ci id="alg2.l4.m1.1.1.3.3.5.cmml" xref="alg2.l4.m1.1.1.3.3.5">𝑡</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l4.m1.1c">\pi_{value}\leftarrow\pi_{init}</annotation></semantics></math> <span id="alg2.l4.1" class="ltx_text" style="float:right;"><math id="alg2.l4.1.m1.1" class="ltx_Math" alttext="\triangleright" display="inline"><semantics id="alg2.l4.1.m1.1a"><mo id="alg2.l4.1.m1.1.1" xref="alg2.l4.1.m1.1.1.cmml">▷</mo><annotation-xml encoding="MathML-Content" id="alg2.l4.1.m1.1b"><ci id="alg2.l4.1.m1.1.1.cmml" xref="alg2.l4.1.m1.1.1">▷</ci></annotation-xml><annotation encoding="application/x-tex" id="alg2.l4.1.m1.1c">\triangleright</annotation></semantics></math> Copy the SoS model to create a value network.
</span>
</div>
<div id="alg2.l5" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">5:</span><span id="alg2.l5.1" class="ltx_text ltx_font_bold">for</span> <math id="alg2.l5.m1.1" class="ltx_Math" alttext="t" display="inline"><semantics id="alg2.l5.m1.1a"><mi id="alg2.l5.m1.1.1" xref="alg2.l5.m1.1.1.cmml">t</mi><annotation-xml encoding="MathML-Content" id="alg2.l5.m1.1b"><ci id="alg2.l5.m1.1.1.cmml" xref="alg2.l5.m1.1.1">𝑡</ci></annotation-xml><annotation encoding="application/x-tex" id="alg2.l5.m1.1c">t</annotation></semantics></math> in <math id="alg2.l5.m2.1" class="ltx_Math" alttext="1\ldots T" display="inline"><semantics id="alg2.l5.m2.1a"><mrow id="alg2.l5.m2.1.1" xref="alg2.l5.m2.1.1.cmml"><mn id="alg2.l5.m2.1.1.2" xref="alg2.l5.m2.1.1.2.cmml">1</mn><mo lspace="0em" rspace="0em" id="alg2.l5.m2.1.1.1" xref="alg2.l5.m2.1.1.1.cmml">​</mo><mi mathvariant="normal" id="alg2.l5.m2.1.1.3" xref="alg2.l5.m2.1.1.3.cmml">…</mi><mo lspace="0em" rspace="0em" id="alg2.l5.m2.1.1.1a" xref="alg2.l5.m2.1.1.1.cmml">​</mo><mi id="alg2.l5.m2.1.1.4" xref="alg2.l5.m2.1.1.4.cmml">T</mi></mrow><annotation-xml encoding="MathML-Content" id="alg2.l5.m2.1b"><apply id="alg2.l5.m2.1.1.cmml" xref="alg2.l5.m2.1.1"><times id="alg2.l5.m2.1.1.1.cmml" xref="alg2.l5.m2.1.1.1"></times><cn type="integer" id="alg2.l5.m2.1.1.2.cmml" xref="alg2.l5.m2.1.1.2">1</cn><ci id="alg2.l5.m2.1.1.3.cmml" xref="alg2.l5.m2.1.1.3">…</ci><ci id="alg2.l5.m2.1.1.4.cmml" xref="alg2.l5.m2.1.1.4">𝑇</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l5.m2.1c">1\ldots T</annotation></semantics></math> <span id="alg2.l5.2" class="ltx_text ltx_font_bold">do</span>

</div>
<div id="alg2.l6" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">6:</span>     Roll out <math id="alg2.l6.m1.1" class="ltx_Math" alttext="\pi_{\theta_{t-1}}" display="inline"><semantics id="alg2.l6.m1.1a"><msub id="alg2.l6.m1.1.1" xref="alg2.l6.m1.1.1.cmml"><mi id="alg2.l6.m1.1.1.2" xref="alg2.l6.m1.1.1.2.cmml">π</mi><msub id="alg2.l6.m1.1.1.3" xref="alg2.l6.m1.1.1.3.cmml"><mi id="alg2.l6.m1.1.1.3.2" xref="alg2.l6.m1.1.1.3.2.cmml">θ</mi><mrow id="alg2.l6.m1.1.1.3.3" xref="alg2.l6.m1.1.1.3.3.cmml"><mi id="alg2.l6.m1.1.1.3.3.2" xref="alg2.l6.m1.1.1.3.3.2.cmml">t</mi><mo id="alg2.l6.m1.1.1.3.3.1" xref="alg2.l6.m1.1.1.3.3.1.cmml">−</mo><mn id="alg2.l6.m1.1.1.3.3.3" xref="alg2.l6.m1.1.1.3.3.3.cmml">1</mn></mrow></msub></msub><annotation-xml encoding="MathML-Content" id="alg2.l6.m1.1b"><apply id="alg2.l6.m1.1.1.cmml" xref="alg2.l6.m1.1.1"><csymbol cd="ambiguous" id="alg2.l6.m1.1.1.1.cmml" xref="alg2.l6.m1.1.1">subscript</csymbol><ci id="alg2.l6.m1.1.1.2.cmml" xref="alg2.l6.m1.1.1.2">𝜋</ci><apply id="alg2.l6.m1.1.1.3.cmml" xref="alg2.l6.m1.1.1.3"><csymbol cd="ambiguous" id="alg2.l6.m1.1.1.3.1.cmml" xref="alg2.l6.m1.1.1.3">subscript</csymbol><ci id="alg2.l6.m1.1.1.3.2.cmml" xref="alg2.l6.m1.1.1.3.2">𝜃</ci><apply id="alg2.l6.m1.1.1.3.3.cmml" xref="alg2.l6.m1.1.1.3.3"><minus id="alg2.l6.m1.1.1.3.3.1.cmml" xref="alg2.l6.m1.1.1.3.3.1"></minus><ci id="alg2.l6.m1.1.1.3.3.2.cmml" xref="alg2.l6.m1.1.1.3.3.2">𝑡</ci><cn type="integer" id="alg2.l6.m1.1.1.3.3.3.cmml" xref="alg2.l6.m1.1.1.3.3.3">1</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l6.m1.1c">\pi_{\theta_{t-1}}</annotation></semantics></math> to produce dataset <math id="alg2.l6.m2.9" class="ltx_Math" alttext="D_{t}=\{(s^{(t)}_{1},a^{(t)}_{1},r^{(t)}_{1}),\cdots,(s^{(t)}_{n},a^{(t)}_{n},r^{(t)}_{n})\}" display="inline"><semantics id="alg2.l6.m2.9a"><mrow id="alg2.l6.m2.9.9" xref="alg2.l6.m2.9.9.cmml"><msub id="alg2.l6.m2.9.9.4" xref="alg2.l6.m2.9.9.4.cmml"><mi id="alg2.l6.m2.9.9.4.2" xref="alg2.l6.m2.9.9.4.2.cmml">D</mi><mi id="alg2.l6.m2.9.9.4.3" xref="alg2.l6.m2.9.9.4.3.cmml">t</mi></msub><mo id="alg2.l6.m2.9.9.3" xref="alg2.l6.m2.9.9.3.cmml">=</mo><mrow id="alg2.l6.m2.9.9.2.2" xref="alg2.l6.m2.9.9.2.3.cmml"><mo stretchy="false" id="alg2.l6.m2.9.9.2.2.3" xref="alg2.l6.m2.9.9.2.3.cmml">{</mo><mrow id="alg2.l6.m2.8.8.1.1.1.3" xref="alg2.l6.m2.8.8.1.1.1.4.cmml"><mo stretchy="false" id="alg2.l6.m2.8.8.1.1.1.3.4" xref="alg2.l6.m2.8.8.1.1.1.4.cmml">(</mo><msubsup id="alg2.l6.m2.8.8.1.1.1.1.1" xref="alg2.l6.m2.8.8.1.1.1.1.1.cmml"><mi id="alg2.l6.m2.8.8.1.1.1.1.1.2.2" xref="alg2.l6.m2.8.8.1.1.1.1.1.2.2.cmml">s</mi><mn id="alg2.l6.m2.8.8.1.1.1.1.1.3" xref="alg2.l6.m2.8.8.1.1.1.1.1.3.cmml">1</mn><mrow id="alg2.l6.m2.1.1.1.3" xref="alg2.l6.m2.8.8.1.1.1.1.1.cmml"><mo stretchy="false" id="alg2.l6.m2.1.1.1.3.1" xref="alg2.l6.m2.8.8.1.1.1.1.1.cmml">(</mo><mi id="alg2.l6.m2.1.1.1.1" xref="alg2.l6.m2.1.1.1.1.cmml">t</mi><mo stretchy="false" id="alg2.l6.m2.1.1.1.3.2" xref="alg2.l6.m2.8.8.1.1.1.1.1.cmml">)</mo></mrow></msubsup><mo id="alg2.l6.m2.8.8.1.1.1.3.5" xref="alg2.l6.m2.8.8.1.1.1.4.cmml">,</mo><msubsup id="alg2.l6.m2.8.8.1.1.1.2.2" xref="alg2.l6.m2.8.8.1.1.1.2.2.cmml"><mi id="alg2.l6.m2.8.8.1.1.1.2.2.2.2" xref="alg2.l6.m2.8.8.1.1.1.2.2.2.2.cmml">a</mi><mn id="alg2.l6.m2.8.8.1.1.1.2.2.3" xref="alg2.l6.m2.8.8.1.1.1.2.2.3.cmml">1</mn><mrow id="alg2.l6.m2.2.2.1.3" xref="alg2.l6.m2.8.8.1.1.1.2.2.cmml"><mo stretchy="false" id="alg2.l6.m2.2.2.1.3.1" xref="alg2.l6.m2.8.8.1.1.1.2.2.cmml">(</mo><mi id="alg2.l6.m2.2.2.1.1" xref="alg2.l6.m2.2.2.1.1.cmml">t</mi><mo stretchy="false" id="alg2.l6.m2.2.2.1.3.2" xref="alg2.l6.m2.8.8.1.1.1.2.2.cmml">)</mo></mrow></msubsup><mo id="alg2.l6.m2.8.8.1.1.1.3.6" xref="alg2.l6.m2.8.8.1.1.1.4.cmml">,</mo><msubsup id="alg2.l6.m2.8.8.1.1.1.3.3" xref="alg2.l6.m2.8.8.1.1.1.3.3.cmml"><mi id="alg2.l6.m2.8.8.1.1.1.3.3.2.2" xref="alg2.l6.m2.8.8.1.1.1.3.3.2.2.cmml">r</mi><mn id="alg2.l6.m2.8.8.1.1.1.3.3.3" xref="alg2.l6.m2.8.8.1.1.1.3.3.3.cmml">1</mn><mrow id="alg2.l6.m2.3.3.1.3" xref="alg2.l6.m2.8.8.1.1.1.3.3.cmml"><mo stretchy="false" id="alg2.l6.m2.3.3.1.3.1" xref="alg2.l6.m2.8.8.1.1.1.3.3.cmml">(</mo><mi id="alg2.l6.m2.3.3.1.1" xref="alg2.l6.m2.3.3.1.1.cmml">t</mi><mo stretchy="false" id="alg2.l6.m2.3.3.1.3.2" xref="alg2.l6.m2.8.8.1.1.1.3.3.cmml">)</mo></mrow></msubsup><mo stretchy="false" id="alg2.l6.m2.8.8.1.1.1.3.7" xref="alg2.l6.m2.8.8.1.1.1.4.cmml">)</mo></mrow><mo id="alg2.l6.m2.9.9.2.2.4" xref="alg2.l6.m2.9.9.2.3.cmml">,</mo><mi mathvariant="normal" id="alg2.l6.m2.7.7" xref="alg2.l6.m2.7.7.cmml">⋯</mi><mo id="alg2.l6.m2.9.9.2.2.5" xref="alg2.l6.m2.9.9.2.3.cmml">,</mo><mrow id="alg2.l6.m2.9.9.2.2.2.3" xref="alg2.l6.m2.9.9.2.2.2.4.cmml"><mo stretchy="false" id="alg2.l6.m2.9.9.2.2.2.3.4" xref="alg2.l6.m2.9.9.2.2.2.4.cmml">(</mo><msubsup id="alg2.l6.m2.9.9.2.2.2.1.1" xref="alg2.l6.m2.9.9.2.2.2.1.1.cmml"><mi id="alg2.l6.m2.9.9.2.2.2.1.1.2.2" xref="alg2.l6.m2.9.9.2.2.2.1.1.2.2.cmml">s</mi><mi id="alg2.l6.m2.9.9.2.2.2.1.1.3" xref="alg2.l6.m2.9.9.2.2.2.1.1.3.cmml">n</mi><mrow id="alg2.l6.m2.4.4.1.3" xref="alg2.l6.m2.9.9.2.2.2.1.1.cmml"><mo stretchy="false" id="alg2.l6.m2.4.4.1.3.1" xref="alg2.l6.m2.9.9.2.2.2.1.1.cmml">(</mo><mi id="alg2.l6.m2.4.4.1.1" xref="alg2.l6.m2.4.4.1.1.cmml">t</mi><mo stretchy="false" id="alg2.l6.m2.4.4.1.3.2" xref="alg2.l6.m2.9.9.2.2.2.1.1.cmml">)</mo></mrow></msubsup><mo id="alg2.l6.m2.9.9.2.2.2.3.5" xref="alg2.l6.m2.9.9.2.2.2.4.cmml">,</mo><msubsup id="alg2.l6.m2.9.9.2.2.2.2.2" xref="alg2.l6.m2.9.9.2.2.2.2.2.cmml"><mi id="alg2.l6.m2.9.9.2.2.2.2.2.2.2" xref="alg2.l6.m2.9.9.2.2.2.2.2.2.2.cmml">a</mi><mi id="alg2.l6.m2.9.9.2.2.2.2.2.3" xref="alg2.l6.m2.9.9.2.2.2.2.2.3.cmml">n</mi><mrow id="alg2.l6.m2.5.5.1.3" xref="alg2.l6.m2.9.9.2.2.2.2.2.cmml"><mo stretchy="false" id="alg2.l6.m2.5.5.1.3.1" xref="alg2.l6.m2.9.9.2.2.2.2.2.cmml">(</mo><mi id="alg2.l6.m2.5.5.1.1" xref="alg2.l6.m2.5.5.1.1.cmml">t</mi><mo stretchy="false" id="alg2.l6.m2.5.5.1.3.2" xref="alg2.l6.m2.9.9.2.2.2.2.2.cmml">)</mo></mrow></msubsup><mo id="alg2.l6.m2.9.9.2.2.2.3.6" xref="alg2.l6.m2.9.9.2.2.2.4.cmml">,</mo><msubsup id="alg2.l6.m2.9.9.2.2.2.3.3" xref="alg2.l6.m2.9.9.2.2.2.3.3.cmml"><mi id="alg2.l6.m2.9.9.2.2.2.3.3.2.2" xref="alg2.l6.m2.9.9.2.2.2.3.3.2.2.cmml">r</mi><mi id="alg2.l6.m2.9.9.2.2.2.3.3.3" xref="alg2.l6.m2.9.9.2.2.2.3.3.3.cmml">n</mi><mrow id="alg2.l6.m2.6.6.1.3" xref="alg2.l6.m2.9.9.2.2.2.3.3.cmml"><mo stretchy="false" id="alg2.l6.m2.6.6.1.3.1" xref="alg2.l6.m2.9.9.2.2.2.3.3.cmml">(</mo><mi id="alg2.l6.m2.6.6.1.1" xref="alg2.l6.m2.6.6.1.1.cmml">t</mi><mo stretchy="false" id="alg2.l6.m2.6.6.1.3.2" xref="alg2.l6.m2.9.9.2.2.2.3.3.cmml">)</mo></mrow></msubsup><mo stretchy="false" id="alg2.l6.m2.9.9.2.2.2.3.7" xref="alg2.l6.m2.9.9.2.2.2.4.cmml">)</mo></mrow><mo stretchy="false" id="alg2.l6.m2.9.9.2.2.6" xref="alg2.l6.m2.9.9.2.3.cmml">}</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg2.l6.m2.9b"><apply id="alg2.l6.m2.9.9.cmml" xref="alg2.l6.m2.9.9"><eq id="alg2.l6.m2.9.9.3.cmml" xref="alg2.l6.m2.9.9.3"></eq><apply id="alg2.l6.m2.9.9.4.cmml" xref="alg2.l6.m2.9.9.4"><csymbol cd="ambiguous" id="alg2.l6.m2.9.9.4.1.cmml" xref="alg2.l6.m2.9.9.4">subscript</csymbol><ci id="alg2.l6.m2.9.9.4.2.cmml" xref="alg2.l6.m2.9.9.4.2">𝐷</ci><ci id="alg2.l6.m2.9.9.4.3.cmml" xref="alg2.l6.m2.9.9.4.3">𝑡</ci></apply><set id="alg2.l6.m2.9.9.2.3.cmml" xref="alg2.l6.m2.9.9.2.2"><vector id="alg2.l6.m2.8.8.1.1.1.4.cmml" xref="alg2.l6.m2.8.8.1.1.1.3"><apply id="alg2.l6.m2.8.8.1.1.1.1.1.cmml" xref="alg2.l6.m2.8.8.1.1.1.1.1"><csymbol cd="ambiguous" id="alg2.l6.m2.8.8.1.1.1.1.1.1.cmml" xref="alg2.l6.m2.8.8.1.1.1.1.1">subscript</csymbol><apply id="alg2.l6.m2.8.8.1.1.1.1.1.2.cmml" xref="alg2.l6.m2.8.8.1.1.1.1.1"><csymbol cd="ambiguous" id="alg2.l6.m2.8.8.1.1.1.1.1.2.1.cmml" xref="alg2.l6.m2.8.8.1.1.1.1.1">superscript</csymbol><ci id="alg2.l6.m2.8.8.1.1.1.1.1.2.2.cmml" xref="alg2.l6.m2.8.8.1.1.1.1.1.2.2">𝑠</ci><ci id="alg2.l6.m2.1.1.1.1.cmml" xref="alg2.l6.m2.1.1.1.1">𝑡</ci></apply><cn type="integer" id="alg2.l6.m2.8.8.1.1.1.1.1.3.cmml" xref="alg2.l6.m2.8.8.1.1.1.1.1.3">1</cn></apply><apply id="alg2.l6.m2.8.8.1.1.1.2.2.cmml" xref="alg2.l6.m2.8.8.1.1.1.2.2"><csymbol cd="ambiguous" id="alg2.l6.m2.8.8.1.1.1.2.2.1.cmml" xref="alg2.l6.m2.8.8.1.1.1.2.2">subscript</csymbol><apply id="alg2.l6.m2.8.8.1.1.1.2.2.2.cmml" xref="alg2.l6.m2.8.8.1.1.1.2.2"><csymbol cd="ambiguous" id="alg2.l6.m2.8.8.1.1.1.2.2.2.1.cmml" xref="alg2.l6.m2.8.8.1.1.1.2.2">superscript</csymbol><ci id="alg2.l6.m2.8.8.1.1.1.2.2.2.2.cmml" xref="alg2.l6.m2.8.8.1.1.1.2.2.2.2">𝑎</ci><ci id="alg2.l6.m2.2.2.1.1.cmml" xref="alg2.l6.m2.2.2.1.1">𝑡</ci></apply><cn type="integer" id="alg2.l6.m2.8.8.1.1.1.2.2.3.cmml" xref="alg2.l6.m2.8.8.1.1.1.2.2.3">1</cn></apply><apply id="alg2.l6.m2.8.8.1.1.1.3.3.cmml" xref="alg2.l6.m2.8.8.1.1.1.3.3"><csymbol cd="ambiguous" id="alg2.l6.m2.8.8.1.1.1.3.3.1.cmml" xref="alg2.l6.m2.8.8.1.1.1.3.3">subscript</csymbol><apply id="alg2.l6.m2.8.8.1.1.1.3.3.2.cmml" xref="alg2.l6.m2.8.8.1.1.1.3.3"><csymbol cd="ambiguous" id="alg2.l6.m2.8.8.1.1.1.3.3.2.1.cmml" xref="alg2.l6.m2.8.8.1.1.1.3.3">superscript</csymbol><ci id="alg2.l6.m2.8.8.1.1.1.3.3.2.2.cmml" xref="alg2.l6.m2.8.8.1.1.1.3.3.2.2">𝑟</ci><ci id="alg2.l6.m2.3.3.1.1.cmml" xref="alg2.l6.m2.3.3.1.1">𝑡</ci></apply><cn type="integer" id="alg2.l6.m2.8.8.1.1.1.3.3.3.cmml" xref="alg2.l6.m2.8.8.1.1.1.3.3.3">1</cn></apply></vector><ci id="alg2.l6.m2.7.7.cmml" xref="alg2.l6.m2.7.7">⋯</ci><vector id="alg2.l6.m2.9.9.2.2.2.4.cmml" xref="alg2.l6.m2.9.9.2.2.2.3"><apply id="alg2.l6.m2.9.9.2.2.2.1.1.cmml" xref="alg2.l6.m2.9.9.2.2.2.1.1"><csymbol cd="ambiguous" id="alg2.l6.m2.9.9.2.2.2.1.1.1.cmml" xref="alg2.l6.m2.9.9.2.2.2.1.1">subscript</csymbol><apply id="alg2.l6.m2.9.9.2.2.2.1.1.2.cmml" xref="alg2.l6.m2.9.9.2.2.2.1.1"><csymbol cd="ambiguous" id="alg2.l6.m2.9.9.2.2.2.1.1.2.1.cmml" xref="alg2.l6.m2.9.9.2.2.2.1.1">superscript</csymbol><ci id="alg2.l6.m2.9.9.2.2.2.1.1.2.2.cmml" xref="alg2.l6.m2.9.9.2.2.2.1.1.2.2">𝑠</ci><ci id="alg2.l6.m2.4.4.1.1.cmml" xref="alg2.l6.m2.4.4.1.1">𝑡</ci></apply><ci id="alg2.l6.m2.9.9.2.2.2.1.1.3.cmml" xref="alg2.l6.m2.9.9.2.2.2.1.1.3">𝑛</ci></apply><apply id="alg2.l6.m2.9.9.2.2.2.2.2.cmml" xref="alg2.l6.m2.9.9.2.2.2.2.2"><csymbol cd="ambiguous" id="alg2.l6.m2.9.9.2.2.2.2.2.1.cmml" xref="alg2.l6.m2.9.9.2.2.2.2.2">subscript</csymbol><apply id="alg2.l6.m2.9.9.2.2.2.2.2.2.cmml" xref="alg2.l6.m2.9.9.2.2.2.2.2"><csymbol cd="ambiguous" id="alg2.l6.m2.9.9.2.2.2.2.2.2.1.cmml" xref="alg2.l6.m2.9.9.2.2.2.2.2">superscript</csymbol><ci id="alg2.l6.m2.9.9.2.2.2.2.2.2.2.cmml" xref="alg2.l6.m2.9.9.2.2.2.2.2.2.2">𝑎</ci><ci id="alg2.l6.m2.5.5.1.1.cmml" xref="alg2.l6.m2.5.5.1.1">𝑡</ci></apply><ci id="alg2.l6.m2.9.9.2.2.2.2.2.3.cmml" xref="alg2.l6.m2.9.9.2.2.2.2.2.3">𝑛</ci></apply><apply id="alg2.l6.m2.9.9.2.2.2.3.3.cmml" xref="alg2.l6.m2.9.9.2.2.2.3.3"><csymbol cd="ambiguous" id="alg2.l6.m2.9.9.2.2.2.3.3.1.cmml" xref="alg2.l6.m2.9.9.2.2.2.3.3">subscript</csymbol><apply id="alg2.l6.m2.9.9.2.2.2.3.3.2.cmml" xref="alg2.l6.m2.9.9.2.2.2.3.3"><csymbol cd="ambiguous" id="alg2.l6.m2.9.9.2.2.2.3.3.2.1.cmml" xref="alg2.l6.m2.9.9.2.2.2.3.3">superscript</csymbol><ci id="alg2.l6.m2.9.9.2.2.2.3.3.2.2.cmml" xref="alg2.l6.m2.9.9.2.2.2.3.3.2.2">𝑟</ci><ci id="alg2.l6.m2.6.6.1.1.cmml" xref="alg2.l6.m2.6.6.1.1">𝑡</ci></apply><ci id="alg2.l6.m2.9.9.2.2.2.3.3.3.cmml" xref="alg2.l6.m2.9.9.2.2.2.3.3.3">𝑛</ci></apply></vector></set></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l6.m2.9c">D_{t}=\{(s^{(t)}_{1},a^{(t)}_{1},r^{(t)}_{1}),\cdots,(s^{(t)}_{n},a^{(t)}_{n},r^{(t)}_{n})\}</annotation></semantics></math>

</div>
<div id="alg2.l7" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">7:</span>     Update policy function according to

</div>
<div id="alg2.l8" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">8:</span>               <math id="alg2.l8.m1.2" class="ltx_Math" alttext="\theta_{t}=\arg\max_{\theta}\mathcal{L}_{APA}(\theta;D_{t})" display="inline"><semantics id="alg2.l8.m1.2a"><mrow id="alg2.l8.m1.2.2" xref="alg2.l8.m1.2.2.cmml"><msub id="alg2.l8.m1.2.2.3" xref="alg2.l8.m1.2.2.3.cmml"><mi id="alg2.l8.m1.2.2.3.2" xref="alg2.l8.m1.2.2.3.2.cmml">θ</mi><mi id="alg2.l8.m1.2.2.3.3" xref="alg2.l8.m1.2.2.3.3.cmml">t</mi></msub><mo id="alg2.l8.m1.2.2.2" xref="alg2.l8.m1.2.2.2.cmml">=</mo><mrow id="alg2.l8.m1.2.2.1" xref="alg2.l8.m1.2.2.1.cmml"><mrow id="alg2.l8.m1.2.2.1.3" xref="alg2.l8.m1.2.2.1.3.cmml"><mi id="alg2.l8.m1.2.2.1.3.1" xref="alg2.l8.m1.2.2.1.3.1.cmml">arg</mi><mo lspace="0.167em" id="alg2.l8.m1.2.2.1.3a" xref="alg2.l8.m1.2.2.1.3.cmml">⁡</mo><mrow id="alg2.l8.m1.2.2.1.3.2" xref="alg2.l8.m1.2.2.1.3.2.cmml"><msub id="alg2.l8.m1.2.2.1.3.2.1" xref="alg2.l8.m1.2.2.1.3.2.1.cmml"><mi id="alg2.l8.m1.2.2.1.3.2.1.2" xref="alg2.l8.m1.2.2.1.3.2.1.2.cmml">max</mi><mi id="alg2.l8.m1.2.2.1.3.2.1.3" xref="alg2.l8.m1.2.2.1.3.2.1.3.cmml">θ</mi></msub><mo lspace="0.167em" id="alg2.l8.m1.2.2.1.3.2a" xref="alg2.l8.m1.2.2.1.3.2.cmml">⁡</mo><msub id="alg2.l8.m1.2.2.1.3.2.2" xref="alg2.l8.m1.2.2.1.3.2.2.cmml"><mi class="ltx_font_mathcaligraphic" id="alg2.l8.m1.2.2.1.3.2.2.2" xref="alg2.l8.m1.2.2.1.3.2.2.2.cmml">ℒ</mi><mrow id="alg2.l8.m1.2.2.1.3.2.2.3" xref="alg2.l8.m1.2.2.1.3.2.2.3.cmml"><mi id="alg2.l8.m1.2.2.1.3.2.2.3.2" xref="alg2.l8.m1.2.2.1.3.2.2.3.2.cmml">A</mi><mo lspace="0em" rspace="0em" id="alg2.l8.m1.2.2.1.3.2.2.3.1" xref="alg2.l8.m1.2.2.1.3.2.2.3.1.cmml">​</mo><mi id="alg2.l8.m1.2.2.1.3.2.2.3.3" xref="alg2.l8.m1.2.2.1.3.2.2.3.3.cmml">P</mi><mo lspace="0em" rspace="0em" id="alg2.l8.m1.2.2.1.3.2.2.3.1a" xref="alg2.l8.m1.2.2.1.3.2.2.3.1.cmml">​</mo><mi id="alg2.l8.m1.2.2.1.3.2.2.3.4" xref="alg2.l8.m1.2.2.1.3.2.2.3.4.cmml">A</mi></mrow></msub></mrow></mrow><mo lspace="0em" rspace="0em" id="alg2.l8.m1.2.2.1.2" xref="alg2.l8.m1.2.2.1.2.cmml">​</mo><mrow id="alg2.l8.m1.2.2.1.1.1" xref="alg2.l8.m1.2.2.1.1.2.cmml"><mo stretchy="false" id="alg2.l8.m1.2.2.1.1.1.2" xref="alg2.l8.m1.2.2.1.1.2.cmml">(</mo><mi id="alg2.l8.m1.1.1" xref="alg2.l8.m1.1.1.cmml">θ</mi><mo id="alg2.l8.m1.2.2.1.1.1.3" xref="alg2.l8.m1.2.2.1.1.2.cmml">;</mo><msub id="alg2.l8.m1.2.2.1.1.1.1" xref="alg2.l8.m1.2.2.1.1.1.1.cmml"><mi id="alg2.l8.m1.2.2.1.1.1.1.2" xref="alg2.l8.m1.2.2.1.1.1.1.2.cmml">D</mi><mi id="alg2.l8.m1.2.2.1.1.1.1.3" xref="alg2.l8.m1.2.2.1.1.1.1.3.cmml">t</mi></msub><mo stretchy="false" id="alg2.l8.m1.2.2.1.1.1.4" xref="alg2.l8.m1.2.2.1.1.2.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg2.l8.m1.2b"><apply id="alg2.l8.m1.2.2.cmml" xref="alg2.l8.m1.2.2"><eq id="alg2.l8.m1.2.2.2.cmml" xref="alg2.l8.m1.2.2.2"></eq><apply id="alg2.l8.m1.2.2.3.cmml" xref="alg2.l8.m1.2.2.3"><csymbol cd="ambiguous" id="alg2.l8.m1.2.2.3.1.cmml" xref="alg2.l8.m1.2.2.3">subscript</csymbol><ci id="alg2.l8.m1.2.2.3.2.cmml" xref="alg2.l8.m1.2.2.3.2">𝜃</ci><ci id="alg2.l8.m1.2.2.3.3.cmml" xref="alg2.l8.m1.2.2.3.3">𝑡</ci></apply><apply id="alg2.l8.m1.2.2.1.cmml" xref="alg2.l8.m1.2.2.1"><times id="alg2.l8.m1.2.2.1.2.cmml" xref="alg2.l8.m1.2.2.1.2"></times><apply id="alg2.l8.m1.2.2.1.3.cmml" xref="alg2.l8.m1.2.2.1.3"><arg id="alg2.l8.m1.2.2.1.3.1.cmml" xref="alg2.l8.m1.2.2.1.3.1"></arg><apply id="alg2.l8.m1.2.2.1.3.2.cmml" xref="alg2.l8.m1.2.2.1.3.2"><apply id="alg2.l8.m1.2.2.1.3.2.1.cmml" xref="alg2.l8.m1.2.2.1.3.2.1"><csymbol cd="ambiguous" id="alg2.l8.m1.2.2.1.3.2.1.1.cmml" xref="alg2.l8.m1.2.2.1.3.2.1">subscript</csymbol><max id="alg2.l8.m1.2.2.1.3.2.1.2.cmml" xref="alg2.l8.m1.2.2.1.3.2.1.2"></max><ci id="alg2.l8.m1.2.2.1.3.2.1.3.cmml" xref="alg2.l8.m1.2.2.1.3.2.1.3">𝜃</ci></apply><apply id="alg2.l8.m1.2.2.1.3.2.2.cmml" xref="alg2.l8.m1.2.2.1.3.2.2"><csymbol cd="ambiguous" id="alg2.l8.m1.2.2.1.3.2.2.1.cmml" xref="alg2.l8.m1.2.2.1.3.2.2">subscript</csymbol><ci id="alg2.l8.m1.2.2.1.3.2.2.2.cmml" xref="alg2.l8.m1.2.2.1.3.2.2.2">ℒ</ci><apply id="alg2.l8.m1.2.2.1.3.2.2.3.cmml" xref="alg2.l8.m1.2.2.1.3.2.2.3"><times id="alg2.l8.m1.2.2.1.3.2.2.3.1.cmml" xref="alg2.l8.m1.2.2.1.3.2.2.3.1"></times><ci id="alg2.l8.m1.2.2.1.3.2.2.3.2.cmml" xref="alg2.l8.m1.2.2.1.3.2.2.3.2">𝐴</ci><ci id="alg2.l8.m1.2.2.1.3.2.2.3.3.cmml" xref="alg2.l8.m1.2.2.1.3.2.2.3.3">𝑃</ci><ci id="alg2.l8.m1.2.2.1.3.2.2.3.4.cmml" xref="alg2.l8.m1.2.2.1.3.2.2.3.4">𝐴</ci></apply></apply></apply></apply><list id="alg2.l8.m1.2.2.1.1.2.cmml" xref="alg2.l8.m1.2.2.1.1.1"><ci id="alg2.l8.m1.1.1.cmml" xref="alg2.l8.m1.1.1">𝜃</ci><apply id="alg2.l8.m1.2.2.1.1.1.1.cmml" xref="alg2.l8.m1.2.2.1.1.1.1"><csymbol cd="ambiguous" id="alg2.l8.m1.2.2.1.1.1.1.1.cmml" xref="alg2.l8.m1.2.2.1.1.1.1">subscript</csymbol><ci id="alg2.l8.m1.2.2.1.1.1.1.2.cmml" xref="alg2.l8.m1.2.2.1.1.1.1.2">𝐷</ci><ci id="alg2.l8.m1.2.2.1.1.1.1.3.cmml" xref="alg2.l8.m1.2.2.1.1.1.1.3">𝑡</ci></apply></list></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l8.m1.2c">\theta_{t}=\arg\max_{\theta}\mathcal{L}_{APA}(\theta;D_{t})</annotation></semantics></math>. <span id="alg2.l8.1" class="ltx_text" style="float:right;"><math id="alg2.l8.1.m1.1" class="ltx_Math" alttext="\triangleright" display="inline"><semantics id="alg2.l8.1.m1.1a"><mo id="alg2.l8.1.m1.1.1" xref="alg2.l8.1.m1.1.1.cmml">▷</mo><annotation-xml encoding="MathML-Content" id="alg2.l8.1.m1.1b"><ci id="alg2.l8.1.m1.1.1.cmml" xref="alg2.l8.1.m1.1.1">▷</ci></annotation-xml><annotation encoding="application/x-tex" id="alg2.l8.1.m1.1c">\triangleright</annotation></semantics></math> We omit the critic loss for simplicity
</span>
</div>
<div id="alg2.l9" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">9:</span>     where

</div>
<div id="alg2.l10" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">10:</span>               <math id="alg2.l10.m1.8" class="ltx_Math" alttext="\mathcal{L}_{APA}(\theta;D)=\frac{1}{|D|}\sum_{(s,a)\in D}\big{(}\log\pi_{\theta}(a|s)-\frac{Adv^{\pi_{\theta_{t-1}}}(s,a)}{\lambda}-\log\pi_{ref}(a|s)\big{)}^{2}" display="inline"><semantics id="alg2.l10.m1.8a"><mrow id="alg2.l10.m1.8.8" xref="alg2.l10.m1.8.8.cmml"><mrow id="alg2.l10.m1.8.8.3" xref="alg2.l10.m1.8.8.3.cmml"><msub id="alg2.l10.m1.8.8.3.2" xref="alg2.l10.m1.8.8.3.2.cmml"><mi class="ltx_font_mathcaligraphic" id="alg2.l10.m1.8.8.3.2.2" xref="alg2.l10.m1.8.8.3.2.2.cmml">ℒ</mi><mrow id="alg2.l10.m1.8.8.3.2.3" xref="alg2.l10.m1.8.8.3.2.3.cmml"><mi id="alg2.l10.m1.8.8.3.2.3.2" xref="alg2.l10.m1.8.8.3.2.3.2.cmml">A</mi><mo lspace="0em" rspace="0em" id="alg2.l10.m1.8.8.3.2.3.1" xref="alg2.l10.m1.8.8.3.2.3.1.cmml">​</mo><mi id="alg2.l10.m1.8.8.3.2.3.3" xref="alg2.l10.m1.8.8.3.2.3.3.cmml">P</mi><mo lspace="0em" rspace="0em" id="alg2.l10.m1.8.8.3.2.3.1a" xref="alg2.l10.m1.8.8.3.2.3.1.cmml">​</mo><mi id="alg2.l10.m1.8.8.3.2.3.4" xref="alg2.l10.m1.8.8.3.2.3.4.cmml">A</mi></mrow></msub><mo lspace="0em" rspace="0em" id="alg2.l10.m1.8.8.3.1" xref="alg2.l10.m1.8.8.3.1.cmml">​</mo><mrow id="alg2.l10.m1.8.8.3.3.2" xref="alg2.l10.m1.8.8.3.3.1.cmml"><mo stretchy="false" id="alg2.l10.m1.8.8.3.3.2.1" xref="alg2.l10.m1.8.8.3.3.1.cmml">(</mo><mi id="alg2.l10.m1.6.6" xref="alg2.l10.m1.6.6.cmml">θ</mi><mo id="alg2.l10.m1.8.8.3.3.2.2" xref="alg2.l10.m1.8.8.3.3.1.cmml">;</mo><mi id="alg2.l10.m1.7.7" xref="alg2.l10.m1.7.7.cmml">D</mi><mo stretchy="false" id="alg2.l10.m1.8.8.3.3.2.3" xref="alg2.l10.m1.8.8.3.3.1.cmml">)</mo></mrow></mrow><mo id="alg2.l10.m1.8.8.2" xref="alg2.l10.m1.8.8.2.cmml">=</mo><mrow id="alg2.l10.m1.8.8.1" xref="alg2.l10.m1.8.8.1.cmml"><mfrac id="alg2.l10.m1.1.1" xref="alg2.l10.m1.1.1.cmml"><mn id="alg2.l10.m1.1.1.3" xref="alg2.l10.m1.1.1.3.cmml">1</mn><mrow id="alg2.l10.m1.1.1.1.3" xref="alg2.l10.m1.1.1.1.2.cmml"><mo stretchy="false" id="alg2.l10.m1.1.1.1.3.1" xref="alg2.l10.m1.1.1.1.2.1.cmml">|</mo><mi id="alg2.l10.m1.1.1.1.1" xref="alg2.l10.m1.1.1.1.1.cmml">D</mi><mo stretchy="false" id="alg2.l10.m1.1.1.1.3.2" xref="alg2.l10.m1.1.1.1.2.1.cmml">|</mo></mrow></mfrac><mo lspace="0em" rspace="0em" id="alg2.l10.m1.8.8.1.2" xref="alg2.l10.m1.8.8.1.2.cmml">​</mo><mrow id="alg2.l10.m1.8.8.1.1" xref="alg2.l10.m1.8.8.1.1.cmml"><msub id="alg2.l10.m1.8.8.1.1.2" xref="alg2.l10.m1.8.8.1.1.2.cmml"><mo rspace="0em" id="alg2.l10.m1.8.8.1.1.2.2" xref="alg2.l10.m1.8.8.1.1.2.2.cmml">∑</mo><mrow id="alg2.l10.m1.3.3.2" xref="alg2.l10.m1.3.3.2.cmml"><mrow id="alg2.l10.m1.3.3.2.4.2" xref="alg2.l10.m1.3.3.2.4.1.cmml"><mo stretchy="false" id="alg2.l10.m1.3.3.2.4.2.1" xref="alg2.l10.m1.3.3.2.4.1.cmml">(</mo><mi id="alg2.l10.m1.2.2.1.1" xref="alg2.l10.m1.2.2.1.1.cmml">s</mi><mo id="alg2.l10.m1.3.3.2.4.2.2" xref="alg2.l10.m1.3.3.2.4.1.cmml">,</mo><mi id="alg2.l10.m1.3.3.2.2" xref="alg2.l10.m1.3.3.2.2.cmml">a</mi><mo stretchy="false" id="alg2.l10.m1.3.3.2.4.2.3" xref="alg2.l10.m1.3.3.2.4.1.cmml">)</mo></mrow><mo id="alg2.l10.m1.3.3.2.3" xref="alg2.l10.m1.3.3.2.3.cmml">∈</mo><mi id="alg2.l10.m1.3.3.2.5" xref="alg2.l10.m1.3.3.2.5.cmml">D</mi></mrow></msub><msup id="alg2.l10.m1.8.8.1.1.1" xref="alg2.l10.m1.8.8.1.1.1.cmml"><mrow id="alg2.l10.m1.8.8.1.1.1.1.1" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.cmml"><mo maxsize="120%" minsize="120%" id="alg2.l10.m1.8.8.1.1.1.1.1.2" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.cmml">(</mo><mrow id="alg2.l10.m1.8.8.1.1.1.1.1.1" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.cmml"><mrow id="alg2.l10.m1.8.8.1.1.1.1.1.1.1" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.1.cmml"><mrow id="alg2.l10.m1.8.8.1.1.1.1.1.1.1.3" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.1.3.cmml"><mi id="alg2.l10.m1.8.8.1.1.1.1.1.1.1.3.1" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.1.3.1.cmml">log</mi><mo lspace="0.167em" id="alg2.l10.m1.8.8.1.1.1.1.1.1.1.3a" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.1.3.cmml">⁡</mo><msub id="alg2.l10.m1.8.8.1.1.1.1.1.1.1.3.2" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.1.3.2.cmml"><mi id="alg2.l10.m1.8.8.1.1.1.1.1.1.1.3.2.2" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.1.3.2.2.cmml">π</mi><mi id="alg2.l10.m1.8.8.1.1.1.1.1.1.1.3.2.3" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.1.3.2.3.cmml">θ</mi></msub></mrow><mo lspace="0em" rspace="0em" id="alg2.l10.m1.8.8.1.1.1.1.1.1.1.2" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.1.2.cmml">​</mo><mrow id="alg2.l10.m1.8.8.1.1.1.1.1.1.1.1.1" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="alg2.l10.m1.8.8.1.1.1.1.1.1.1.1.1.2" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="alg2.l10.m1.8.8.1.1.1.1.1.1.1.1.1.1" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.1.1.1.1.cmml"><mi id="alg2.l10.m1.8.8.1.1.1.1.1.1.1.1.1.1.2" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.1.1.1.1.2.cmml">a</mi><mo fence="false" id="alg2.l10.m1.8.8.1.1.1.1.1.1.1.1.1.1.1" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.1.1.1.1.1.cmml">|</mo><mi id="alg2.l10.m1.8.8.1.1.1.1.1.1.1.1.1.1.3" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.1.1.1.1.3.cmml">s</mi></mrow><mo stretchy="false" id="alg2.l10.m1.8.8.1.1.1.1.1.1.1.1.1.3" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="alg2.l10.m1.8.8.1.1.1.1.1.1.3" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.3.cmml">−</mo><mfrac id="alg2.l10.m1.5.5" xref="alg2.l10.m1.5.5.cmml"><mrow id="alg2.l10.m1.5.5.2" xref="alg2.l10.m1.5.5.2.cmml"><mi id="alg2.l10.m1.5.5.2.4" xref="alg2.l10.m1.5.5.2.4.cmml">A</mi><mo lspace="0em" rspace="0em" id="alg2.l10.m1.5.5.2.3" xref="alg2.l10.m1.5.5.2.3.cmml">​</mo><mi id="alg2.l10.m1.5.5.2.5" xref="alg2.l10.m1.5.5.2.5.cmml">d</mi><mo lspace="0em" rspace="0em" id="alg2.l10.m1.5.5.2.3a" xref="alg2.l10.m1.5.5.2.3.cmml">​</mo><msup id="alg2.l10.m1.5.5.2.6" xref="alg2.l10.m1.5.5.2.6.cmml"><mi id="alg2.l10.m1.5.5.2.6.2" xref="alg2.l10.m1.5.5.2.6.2.cmml">v</mi><msub id="alg2.l10.m1.5.5.2.6.3" xref="alg2.l10.m1.5.5.2.6.3.cmml"><mi id="alg2.l10.m1.5.5.2.6.3.2" xref="alg2.l10.m1.5.5.2.6.3.2.cmml">π</mi><msub id="alg2.l10.m1.5.5.2.6.3.3" xref="alg2.l10.m1.5.5.2.6.3.3.cmml"><mi id="alg2.l10.m1.5.5.2.6.3.3.2" xref="alg2.l10.m1.5.5.2.6.3.3.2.cmml">θ</mi><mrow id="alg2.l10.m1.5.5.2.6.3.3.3" xref="alg2.l10.m1.5.5.2.6.3.3.3.cmml"><mi id="alg2.l10.m1.5.5.2.6.3.3.3.2" xref="alg2.l10.m1.5.5.2.6.3.3.3.2.cmml">t</mi><mo id="alg2.l10.m1.5.5.2.6.3.3.3.1" xref="alg2.l10.m1.5.5.2.6.3.3.3.1.cmml">−</mo><mn id="alg2.l10.m1.5.5.2.6.3.3.3.3" xref="alg2.l10.m1.5.5.2.6.3.3.3.3.cmml">1</mn></mrow></msub></msub></msup><mo lspace="0em" rspace="0em" id="alg2.l10.m1.5.5.2.3b" xref="alg2.l10.m1.5.5.2.3.cmml">​</mo><mrow id="alg2.l10.m1.5.5.2.7.2" xref="alg2.l10.m1.5.5.2.7.1.cmml"><mo stretchy="false" id="alg2.l10.m1.5.5.2.7.2.1" xref="alg2.l10.m1.5.5.2.7.1.cmml">(</mo><mi id="alg2.l10.m1.4.4.1.1" xref="alg2.l10.m1.4.4.1.1.cmml">s</mi><mo id="alg2.l10.m1.5.5.2.7.2.2" xref="alg2.l10.m1.5.5.2.7.1.cmml">,</mo><mi id="alg2.l10.m1.5.5.2.2" xref="alg2.l10.m1.5.5.2.2.cmml">a</mi><mo stretchy="false" id="alg2.l10.m1.5.5.2.7.2.3" xref="alg2.l10.m1.5.5.2.7.1.cmml">)</mo></mrow></mrow><mi id="alg2.l10.m1.5.5.4" xref="alg2.l10.m1.5.5.4.cmml">λ</mi></mfrac><mo id="alg2.l10.m1.8.8.1.1.1.1.1.1.3a" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.3.cmml">−</mo><mrow id="alg2.l10.m1.8.8.1.1.1.1.1.1.2" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.2.cmml"><mrow id="alg2.l10.m1.8.8.1.1.1.1.1.1.2.3" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.2.3.cmml"><mi id="alg2.l10.m1.8.8.1.1.1.1.1.1.2.3.1" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.2.3.1.cmml">log</mi><mo lspace="0.167em" id="alg2.l10.m1.8.8.1.1.1.1.1.1.2.3a" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.2.3.cmml">⁡</mo><msub id="alg2.l10.m1.8.8.1.1.1.1.1.1.2.3.2" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.2.3.2.cmml"><mi id="alg2.l10.m1.8.8.1.1.1.1.1.1.2.3.2.2" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.2.3.2.2.cmml">π</mi><mrow id="alg2.l10.m1.8.8.1.1.1.1.1.1.2.3.2.3" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.2.3.2.3.cmml"><mi id="alg2.l10.m1.8.8.1.1.1.1.1.1.2.3.2.3.2" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.2.3.2.3.2.cmml">r</mi><mo lspace="0em" rspace="0em" id="alg2.l10.m1.8.8.1.1.1.1.1.1.2.3.2.3.1" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.2.3.2.3.1.cmml">​</mo><mi id="alg2.l10.m1.8.8.1.1.1.1.1.1.2.3.2.3.3" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.2.3.2.3.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg2.l10.m1.8.8.1.1.1.1.1.1.2.3.2.3.1a" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.2.3.2.3.1.cmml">​</mo><mi id="alg2.l10.m1.8.8.1.1.1.1.1.1.2.3.2.3.4" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.2.3.2.3.4.cmml">f</mi></mrow></msub></mrow><mo lspace="0em" rspace="0em" id="alg2.l10.m1.8.8.1.1.1.1.1.1.2.2" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.2.2.cmml">​</mo><mrow id="alg2.l10.m1.8.8.1.1.1.1.1.1.2.1.1" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.2.1.1.1.cmml"><mo stretchy="false" id="alg2.l10.m1.8.8.1.1.1.1.1.1.2.1.1.2" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.2.1.1.1.cmml">(</mo><mrow id="alg2.l10.m1.8.8.1.1.1.1.1.1.2.1.1.1" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.2.1.1.1.cmml"><mi id="alg2.l10.m1.8.8.1.1.1.1.1.1.2.1.1.1.2" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.2.1.1.1.2.cmml">a</mi><mo fence="false" id="alg2.l10.m1.8.8.1.1.1.1.1.1.2.1.1.1.1" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.2.1.1.1.1.cmml">|</mo><mi id="alg2.l10.m1.8.8.1.1.1.1.1.1.2.1.1.1.3" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.2.1.1.1.3.cmml">s</mi></mrow><mo stretchy="false" id="alg2.l10.m1.8.8.1.1.1.1.1.1.2.1.1.3" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.2.1.1.1.cmml">)</mo></mrow></mrow></mrow><mo maxsize="120%" minsize="120%" id="alg2.l10.m1.8.8.1.1.1.1.1.3" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.cmml">)</mo></mrow><mn id="alg2.l10.m1.8.8.1.1.1.3" xref="alg2.l10.m1.8.8.1.1.1.3.cmml">2</mn></msup></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg2.l10.m1.8b"><apply id="alg2.l10.m1.8.8.cmml" xref="alg2.l10.m1.8.8"><eq id="alg2.l10.m1.8.8.2.cmml" xref="alg2.l10.m1.8.8.2"></eq><apply id="alg2.l10.m1.8.8.3.cmml" xref="alg2.l10.m1.8.8.3"><times id="alg2.l10.m1.8.8.3.1.cmml" xref="alg2.l10.m1.8.8.3.1"></times><apply id="alg2.l10.m1.8.8.3.2.cmml" xref="alg2.l10.m1.8.8.3.2"><csymbol cd="ambiguous" id="alg2.l10.m1.8.8.3.2.1.cmml" xref="alg2.l10.m1.8.8.3.2">subscript</csymbol><ci id="alg2.l10.m1.8.8.3.2.2.cmml" xref="alg2.l10.m1.8.8.3.2.2">ℒ</ci><apply id="alg2.l10.m1.8.8.3.2.3.cmml" xref="alg2.l10.m1.8.8.3.2.3"><times id="alg2.l10.m1.8.8.3.2.3.1.cmml" xref="alg2.l10.m1.8.8.3.2.3.1"></times><ci id="alg2.l10.m1.8.8.3.2.3.2.cmml" xref="alg2.l10.m1.8.8.3.2.3.2">𝐴</ci><ci id="alg2.l10.m1.8.8.3.2.3.3.cmml" xref="alg2.l10.m1.8.8.3.2.3.3">𝑃</ci><ci id="alg2.l10.m1.8.8.3.2.3.4.cmml" xref="alg2.l10.m1.8.8.3.2.3.4">𝐴</ci></apply></apply><list id="alg2.l10.m1.8.8.3.3.1.cmml" xref="alg2.l10.m1.8.8.3.3.2"><ci id="alg2.l10.m1.6.6.cmml" xref="alg2.l10.m1.6.6">𝜃</ci><ci id="alg2.l10.m1.7.7.cmml" xref="alg2.l10.m1.7.7">𝐷</ci></list></apply><apply id="alg2.l10.m1.8.8.1.cmml" xref="alg2.l10.m1.8.8.1"><times id="alg2.l10.m1.8.8.1.2.cmml" xref="alg2.l10.m1.8.8.1.2"></times><apply id="alg2.l10.m1.1.1.cmml" xref="alg2.l10.m1.1.1"><divide id="alg2.l10.m1.1.1.2.cmml" xref="alg2.l10.m1.1.1"></divide><cn type="integer" id="alg2.l10.m1.1.1.3.cmml" xref="alg2.l10.m1.1.1.3">1</cn><apply id="alg2.l10.m1.1.1.1.2.cmml" xref="alg2.l10.m1.1.1.1.3"><abs id="alg2.l10.m1.1.1.1.2.1.cmml" xref="alg2.l10.m1.1.1.1.3.1"></abs><ci id="alg2.l10.m1.1.1.1.1.cmml" xref="alg2.l10.m1.1.1.1.1">𝐷</ci></apply></apply><apply id="alg2.l10.m1.8.8.1.1.cmml" xref="alg2.l10.m1.8.8.1.1"><apply id="alg2.l10.m1.8.8.1.1.2.cmml" xref="alg2.l10.m1.8.8.1.1.2"><csymbol cd="ambiguous" id="alg2.l10.m1.8.8.1.1.2.1.cmml" xref="alg2.l10.m1.8.8.1.1.2">subscript</csymbol><sum id="alg2.l10.m1.8.8.1.1.2.2.cmml" xref="alg2.l10.m1.8.8.1.1.2.2"></sum><apply id="alg2.l10.m1.3.3.2.cmml" xref="alg2.l10.m1.3.3.2"><in id="alg2.l10.m1.3.3.2.3.cmml" xref="alg2.l10.m1.3.3.2.3"></in><interval closure="open" id="alg2.l10.m1.3.3.2.4.1.cmml" xref="alg2.l10.m1.3.3.2.4.2"><ci id="alg2.l10.m1.2.2.1.1.cmml" xref="alg2.l10.m1.2.2.1.1">𝑠</ci><ci id="alg2.l10.m1.3.3.2.2.cmml" xref="alg2.l10.m1.3.3.2.2">𝑎</ci></interval><ci id="alg2.l10.m1.3.3.2.5.cmml" xref="alg2.l10.m1.3.3.2.5">𝐷</ci></apply></apply><apply id="alg2.l10.m1.8.8.1.1.1.cmml" xref="alg2.l10.m1.8.8.1.1.1"><csymbol cd="ambiguous" id="alg2.l10.m1.8.8.1.1.1.2.cmml" xref="alg2.l10.m1.8.8.1.1.1">superscript</csymbol><apply id="alg2.l10.m1.8.8.1.1.1.1.1.1.cmml" xref="alg2.l10.m1.8.8.1.1.1.1.1"><minus id="alg2.l10.m1.8.8.1.1.1.1.1.1.3.cmml" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.3"></minus><apply id="alg2.l10.m1.8.8.1.1.1.1.1.1.1.cmml" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.1"><times id="alg2.l10.m1.8.8.1.1.1.1.1.1.1.2.cmml" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.1.2"></times><apply id="alg2.l10.m1.8.8.1.1.1.1.1.1.1.3.cmml" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.1.3"><log id="alg2.l10.m1.8.8.1.1.1.1.1.1.1.3.1.cmml" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.1.3.1"></log><apply id="alg2.l10.m1.8.8.1.1.1.1.1.1.1.3.2.cmml" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.1.3.2"><csymbol cd="ambiguous" id="alg2.l10.m1.8.8.1.1.1.1.1.1.1.3.2.1.cmml" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.1.3.2">subscript</csymbol><ci id="alg2.l10.m1.8.8.1.1.1.1.1.1.1.3.2.2.cmml" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.1.3.2.2">𝜋</ci><ci id="alg2.l10.m1.8.8.1.1.1.1.1.1.1.3.2.3.cmml" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.1.3.2.3">𝜃</ci></apply></apply><apply id="alg2.l10.m1.8.8.1.1.1.1.1.1.1.1.1.1.cmml" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.1.1.1"><csymbol cd="latexml" id="alg2.l10.m1.8.8.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.1.1.1.1.1">conditional</csymbol><ci id="alg2.l10.m1.8.8.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.1.1.1.1.2">𝑎</ci><ci id="alg2.l10.m1.8.8.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.1.1.1.1.3">𝑠</ci></apply></apply><apply id="alg2.l10.m1.5.5.cmml" xref="alg2.l10.m1.5.5"><divide id="alg2.l10.m1.5.5.3.cmml" xref="alg2.l10.m1.5.5"></divide><apply id="alg2.l10.m1.5.5.2.cmml" xref="alg2.l10.m1.5.5.2"><times id="alg2.l10.m1.5.5.2.3.cmml" xref="alg2.l10.m1.5.5.2.3"></times><ci id="alg2.l10.m1.5.5.2.4.cmml" xref="alg2.l10.m1.5.5.2.4">𝐴</ci><ci id="alg2.l10.m1.5.5.2.5.cmml" xref="alg2.l10.m1.5.5.2.5">𝑑</ci><apply id="alg2.l10.m1.5.5.2.6.cmml" xref="alg2.l10.m1.5.5.2.6"><csymbol cd="ambiguous" id="alg2.l10.m1.5.5.2.6.1.cmml" xref="alg2.l10.m1.5.5.2.6">superscript</csymbol><ci id="alg2.l10.m1.5.5.2.6.2.cmml" xref="alg2.l10.m1.5.5.2.6.2">𝑣</ci><apply id="alg2.l10.m1.5.5.2.6.3.cmml" xref="alg2.l10.m1.5.5.2.6.3"><csymbol cd="ambiguous" id="alg2.l10.m1.5.5.2.6.3.1.cmml" xref="alg2.l10.m1.5.5.2.6.3">subscript</csymbol><ci id="alg2.l10.m1.5.5.2.6.3.2.cmml" xref="alg2.l10.m1.5.5.2.6.3.2">𝜋</ci><apply id="alg2.l10.m1.5.5.2.6.3.3.cmml" xref="alg2.l10.m1.5.5.2.6.3.3"><csymbol cd="ambiguous" id="alg2.l10.m1.5.5.2.6.3.3.1.cmml" xref="alg2.l10.m1.5.5.2.6.3.3">subscript</csymbol><ci id="alg2.l10.m1.5.5.2.6.3.3.2.cmml" xref="alg2.l10.m1.5.5.2.6.3.3.2">𝜃</ci><apply id="alg2.l10.m1.5.5.2.6.3.3.3.cmml" xref="alg2.l10.m1.5.5.2.6.3.3.3"><minus id="alg2.l10.m1.5.5.2.6.3.3.3.1.cmml" xref="alg2.l10.m1.5.5.2.6.3.3.3.1"></minus><ci id="alg2.l10.m1.5.5.2.6.3.3.3.2.cmml" xref="alg2.l10.m1.5.5.2.6.3.3.3.2">𝑡</ci><cn type="integer" id="alg2.l10.m1.5.5.2.6.3.3.3.3.cmml" xref="alg2.l10.m1.5.5.2.6.3.3.3.3">1</cn></apply></apply></apply></apply><interval closure="open" id="alg2.l10.m1.5.5.2.7.1.cmml" xref="alg2.l10.m1.5.5.2.7.2"><ci id="alg2.l10.m1.4.4.1.1.cmml" xref="alg2.l10.m1.4.4.1.1">𝑠</ci><ci id="alg2.l10.m1.5.5.2.2.cmml" xref="alg2.l10.m1.5.5.2.2">𝑎</ci></interval></apply><ci id="alg2.l10.m1.5.5.4.cmml" xref="alg2.l10.m1.5.5.4">𝜆</ci></apply><apply id="alg2.l10.m1.8.8.1.1.1.1.1.1.2.cmml" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.2"><times id="alg2.l10.m1.8.8.1.1.1.1.1.1.2.2.cmml" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.2.2"></times><apply id="alg2.l10.m1.8.8.1.1.1.1.1.1.2.3.cmml" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.2.3"><log id="alg2.l10.m1.8.8.1.1.1.1.1.1.2.3.1.cmml" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.2.3.1"></log><apply id="alg2.l10.m1.8.8.1.1.1.1.1.1.2.3.2.cmml" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.2.3.2"><csymbol cd="ambiguous" id="alg2.l10.m1.8.8.1.1.1.1.1.1.2.3.2.1.cmml" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.2.3.2">subscript</csymbol><ci id="alg2.l10.m1.8.8.1.1.1.1.1.1.2.3.2.2.cmml" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.2.3.2.2">𝜋</ci><apply id="alg2.l10.m1.8.8.1.1.1.1.1.1.2.3.2.3.cmml" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.2.3.2.3"><times id="alg2.l10.m1.8.8.1.1.1.1.1.1.2.3.2.3.1.cmml" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.2.3.2.3.1"></times><ci id="alg2.l10.m1.8.8.1.1.1.1.1.1.2.3.2.3.2.cmml" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.2.3.2.3.2">𝑟</ci><ci id="alg2.l10.m1.8.8.1.1.1.1.1.1.2.3.2.3.3.cmml" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.2.3.2.3.3">𝑒</ci><ci id="alg2.l10.m1.8.8.1.1.1.1.1.1.2.3.2.3.4.cmml" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.2.3.2.3.4">𝑓</ci></apply></apply></apply><apply id="alg2.l10.m1.8.8.1.1.1.1.1.1.2.1.1.1.cmml" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.2.1.1"><csymbol cd="latexml" id="alg2.l10.m1.8.8.1.1.1.1.1.1.2.1.1.1.1.cmml" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.2.1.1.1.1">conditional</csymbol><ci id="alg2.l10.m1.8.8.1.1.1.1.1.1.2.1.1.1.2.cmml" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.2.1.1.1.2">𝑎</ci><ci id="alg2.l10.m1.8.8.1.1.1.1.1.1.2.1.1.1.3.cmml" xref="alg2.l10.m1.8.8.1.1.1.1.1.1.2.1.1.1.3">𝑠</ci></apply></apply></apply><cn type="integer" id="alg2.l10.m1.8.8.1.1.1.3.cmml" xref="alg2.l10.m1.8.8.1.1.1.3">2</cn></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l10.m1.8c">\mathcal{L}_{APA}(\theta;D)=\frac{1}{|D|}\sum_{(s,a)\in D}\big{(}\log\pi_{\theta}(a|s)-\frac{Adv^{\pi_{\theta_{t-1}}}(s,a)}{\lambda}-\log\pi_{ref}(a|s)\big{)}^{2}</annotation></semantics></math>.

</div>
<div id="alg2.l11" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">11:</span>     If validation reward converges, update <math id="alg2.l11.m1.1" class="ltx_Math" alttext="\pi_{ref}" display="inline"><semantics id="alg2.l11.m1.1a"><msub id="alg2.l11.m1.1.1" xref="alg2.l11.m1.1.1.cmml"><mi id="alg2.l11.m1.1.1.2" xref="alg2.l11.m1.1.1.2.cmml">π</mi><mrow id="alg2.l11.m1.1.1.3" xref="alg2.l11.m1.1.1.3.cmml"><mi id="alg2.l11.m1.1.1.3.2" xref="alg2.l11.m1.1.1.3.2.cmml">r</mi><mo lspace="0em" rspace="0em" id="alg2.l11.m1.1.1.3.1" xref="alg2.l11.m1.1.1.3.1.cmml">​</mo><mi id="alg2.l11.m1.1.1.3.3" xref="alg2.l11.m1.1.1.3.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg2.l11.m1.1.1.3.1a" xref="alg2.l11.m1.1.1.3.1.cmml">​</mo><mi id="alg2.l11.m1.1.1.3.4" xref="alg2.l11.m1.1.1.3.4.cmml">f</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="alg2.l11.m1.1b"><apply id="alg2.l11.m1.1.1.cmml" xref="alg2.l11.m1.1.1"><csymbol cd="ambiguous" id="alg2.l11.m1.1.1.1.cmml" xref="alg2.l11.m1.1.1">subscript</csymbol><ci id="alg2.l11.m1.1.1.2.cmml" xref="alg2.l11.m1.1.1.2">𝜋</ci><apply id="alg2.l11.m1.1.1.3.cmml" xref="alg2.l11.m1.1.1.3"><times id="alg2.l11.m1.1.1.3.1.cmml" xref="alg2.l11.m1.1.1.3.1"></times><ci id="alg2.l11.m1.1.1.3.2.cmml" xref="alg2.l11.m1.1.1.3.2">𝑟</ci><ci id="alg2.l11.m1.1.1.3.3.cmml" xref="alg2.l11.m1.1.1.3.3">𝑒</ci><ci id="alg2.l11.m1.1.1.3.4.cmml" xref="alg2.l11.m1.1.1.3.4">𝑓</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l11.m1.1c">\pi_{ref}</annotation></semantics></math>

</div>
<div id="alg2.l12" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">12:</span>               <math id="alg2.l12.m1.1" class="ltx_Math" alttext="\pi_{ref}\leftarrow\pi_{\theta_{t}}" display="inline"><semantics id="alg2.l12.m1.1a"><mrow id="alg2.l12.m1.1.1" xref="alg2.l12.m1.1.1.cmml"><msub id="alg2.l12.m1.1.1.2" xref="alg2.l12.m1.1.1.2.cmml"><mi id="alg2.l12.m1.1.1.2.2" xref="alg2.l12.m1.1.1.2.2.cmml">π</mi><mrow id="alg2.l12.m1.1.1.2.3" xref="alg2.l12.m1.1.1.2.3.cmml"><mi id="alg2.l12.m1.1.1.2.3.2" xref="alg2.l12.m1.1.1.2.3.2.cmml">r</mi><mo lspace="0em" rspace="0em" id="alg2.l12.m1.1.1.2.3.1" xref="alg2.l12.m1.1.1.2.3.1.cmml">​</mo><mi id="alg2.l12.m1.1.1.2.3.3" xref="alg2.l12.m1.1.1.2.3.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg2.l12.m1.1.1.2.3.1a" xref="alg2.l12.m1.1.1.2.3.1.cmml">​</mo><mi id="alg2.l12.m1.1.1.2.3.4" xref="alg2.l12.m1.1.1.2.3.4.cmml">f</mi></mrow></msub><mo stretchy="false" id="alg2.l12.m1.1.1.1" xref="alg2.l12.m1.1.1.1.cmml">←</mo><msub id="alg2.l12.m1.1.1.3" xref="alg2.l12.m1.1.1.3.cmml"><mi id="alg2.l12.m1.1.1.3.2" xref="alg2.l12.m1.1.1.3.2.cmml">π</mi><msub id="alg2.l12.m1.1.1.3.3" xref="alg2.l12.m1.1.1.3.3.cmml"><mi id="alg2.l12.m1.1.1.3.3.2" xref="alg2.l12.m1.1.1.3.3.2.cmml">θ</mi><mi id="alg2.l12.m1.1.1.3.3.3" xref="alg2.l12.m1.1.1.3.3.3.cmml">t</mi></msub></msub></mrow><annotation-xml encoding="MathML-Content" id="alg2.l12.m1.1b"><apply id="alg2.l12.m1.1.1.cmml" xref="alg2.l12.m1.1.1"><ci id="alg2.l12.m1.1.1.1.cmml" xref="alg2.l12.m1.1.1.1">←</ci><apply id="alg2.l12.m1.1.1.2.cmml" xref="alg2.l12.m1.1.1.2"><csymbol cd="ambiguous" id="alg2.l12.m1.1.1.2.1.cmml" xref="alg2.l12.m1.1.1.2">subscript</csymbol><ci id="alg2.l12.m1.1.1.2.2.cmml" xref="alg2.l12.m1.1.1.2.2">𝜋</ci><apply id="alg2.l12.m1.1.1.2.3.cmml" xref="alg2.l12.m1.1.1.2.3"><times id="alg2.l12.m1.1.1.2.3.1.cmml" xref="alg2.l12.m1.1.1.2.3.1"></times><ci id="alg2.l12.m1.1.1.2.3.2.cmml" xref="alg2.l12.m1.1.1.2.3.2">𝑟</ci><ci id="alg2.l12.m1.1.1.2.3.3.cmml" xref="alg2.l12.m1.1.1.2.3.3">𝑒</ci><ci id="alg2.l12.m1.1.1.2.3.4.cmml" xref="alg2.l12.m1.1.1.2.3.4">𝑓</ci></apply></apply><apply id="alg2.l12.m1.1.1.3.cmml" xref="alg2.l12.m1.1.1.3"><csymbol cd="ambiguous" id="alg2.l12.m1.1.1.3.1.cmml" xref="alg2.l12.m1.1.1.3">subscript</csymbol><ci id="alg2.l12.m1.1.1.3.2.cmml" xref="alg2.l12.m1.1.1.3.2">𝜋</ci><apply id="alg2.l12.m1.1.1.3.3.cmml" xref="alg2.l12.m1.1.1.3.3"><csymbol cd="ambiguous" id="alg2.l12.m1.1.1.3.3.1.cmml" xref="alg2.l12.m1.1.1.3.3">subscript</csymbol><ci id="alg2.l12.m1.1.1.3.3.2.cmml" xref="alg2.l12.m1.1.1.3.3.2">𝜃</ci><ci id="alg2.l12.m1.1.1.3.3.3.cmml" xref="alg2.l12.m1.1.1.3.3.3">𝑡</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l12.m1.1c">\pi_{ref}\leftarrow\pi_{\theta_{t}}</annotation></semantics></math>

</div>
<div id="alg2.l13" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">13:</span><span id="alg2.l13.1" class="ltx_text ltx_font_bold">end</span> <span id="alg2.l13.2" class="ltx_text ltx_font_bold">for</span>
</div>
</div>
</figure>
<figure id="S6.F4" class="ltx_figure"><img src="/html/2404.03683/assets/x4.png" id="S6.F4.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="422" height="135" alt="Refer to caption">
<br class="ltx_break ltx_centering">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 4: </span>
(a) Improvement in accuracy with different policy improvement methods. (b) Accuracy on the validation set of problems during APA training. We reset the reference policy after the validation accuracy converges. Different colors represent training after resetting the reference policy to the current model. We stop seeing improvements after 3 such resets. (c) Improvements in test accuracies with each STaR iteration. Error bars represent 95% Binomial Confidence Intervals.
</figcaption>
</figure>
<div id="S6.p5" class="ltx_para ltx_noindent">
<p id="S6.p5.1" class="ltx_p"><span id="S6.p5.1.1" class="ltx_text ltx_font_bold">Results.</span> The SoS models converge after 3 iterations of STaR finetuning (<a href="#S6.F4" title="Figure 4 ‣ 6 Policy Improvement with Stream of Search ‣ Stream of Search (SoS): Learning to Search in Language" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Fig. 4</span></a>). We see that after three iterations, the finetuned SoS+STaR model solves an additional 5% of the held-out inputs test set beyond the base SoS model. A similar pattern is seen for the held-out targets test set. When SoS models are finetuned with APA, we see that the validation accuracy stops improving after about 4000 training steps. We reset the reference policy 3 times, when the validation accuracy stops improving — see <a href="#S6.F4" title="Figure 4 ‣ 6 Policy Improvement with Stream of Search ‣ Stream of Search (SoS): Learning to Search in Language" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Fig. 4</span></a>b; different colors represent training after resetting the reference policy. Overall, we see an improvement of about 6% over the base SoS model.</p>
</div>
<div id="S6.p6" class="ltx_para ltx_noindent">
<p id="S6.p6.1" class="ltx_p">When we analyze the difference between the base and finetuned models in terms of the alignment of the states visited, we observe (<a href="#S6.F5" title="Figure 5 ‣ 6 Policy Improvement with Stream of Search ‣ Stream of Search (SoS): Learning to Search in Language" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Fig. 5</span></a>a) that both the STaR and APA models visit more states associated with the ‘multiply’ heuristic, which measures distance to the factors of the target. Further, we note that the APA model is less aligned with the symbolic strategies compared to the base SoS model, indicating that it diverges more from the symbolic strategies and employs different strategies for searching. These state visitation metrics provide insights into how the SoS+STaR and SoS+APA models can flexibly utilize various search strategies, potentially discovering novel heuristics and search methods.</p>
</div>
<div id="S6.p7" class="ltx_para ltx_noindent">
<p id="S6.p7.1" class="ltx_p">To further evaluate the performance of the improved models, we select 10,000 problems from the SoS training set that were unsolved by symbolic strategies when the dataset was generated, and 10,000 difficult problems that none of the symbolic strategies used to train the SoS models can solve. Remarkably, the models are able to solve approximately 36% of the previously unsolved problems (<a href="#S6.F5" title="Figure 5 ‣ 6 Policy Improvement with Stream of Search ‣ Stream of Search (SoS): Learning to Search in Language" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Fig. 5</span></a>b) and about 4% of the difficult problems (<a href="#S6.F5" title="Figure 5 ‣ 6 Policy Improvement with Stream of Search ‣ Stream of Search (SoS): Learning to Search in Language" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Fig. 5</span></a>c). Finally, SoS + APA and SoS + STaR models also have better models of the environment, making fewer errors while searching (<a href="#S6.F6.fig1" title="Figure 6 ‣ 6 Policy Improvement with Stream of Search ‣ Stream of Search (SoS): Learning to Search in Language" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Fig. 6</span></a>; App. <a href="#A4.T2" title="Table 2 ‣ Appendix D Analysis of generated trajectories ‣ Stream of Search (SoS): Learning to Search in Language" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Tab. 2</span></a>), and finding the solution more quickly (<a href="#S6.F6.fig1" title="Figure 6 ‣ 6 Policy Improvement with Stream of Search ‣ Stream of Search (SoS): Learning to Search in Language" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Fig. 6</span></a>).</p>
</div>
<figure id="S6.F5" class="ltx_figure"><img src="/html/2404.03683/assets/x5.png" id="S6.F5.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="422" height="150" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 5: </span>(a) Difference in the states visited compared to the base Stream of Search Model after finetuning with APA and STaR. We see that the model prefers the states visited by certain symbolic strategies after improvement. (b) The percentage of previously unsolved problems from the training set that the trained models were able to solve. (c) The percentage of difficult problems, defined as those not solvable by any symbolic algorithms, that the trained SoS models successfully solved. Error bars are 95% Binomial Confidence Intervals.</figcaption>
</figure>
<figure id="S6.F6.fig1" class="ltx_figure ltx_figure_panel ltx_minipage ltx_align_middle" style="width:99.4pt;">
<div id="S6.F6.1" class="ltx_block ltx_figure_panel ltx_minipage ltx_align_middle" style="width:278.2pt;">
<img src="/html/2404.03683/assets/x6.png" id="S6.F6.1.g1" class="ltx_graphics ltx_img_landscape" width="422" height="276" alt="Refer to caption">
</div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 6: </span>(left) Average number of arithmetic errors made per search trajectory by different models. Policy improvement leads to fewer errors. (right) Average number of states visited per trajectory for correct solutions. Policy improvement leads to more efficient solutions.</figcaption>
</figure>
</section>
<section id="S7" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">7 </span>Discussion</h2>

<div id="S7.p1" class="ltx_para ltx_noindent">
<p id="S7.p1.1" class="ltx_p">We have introduced the Stream of Search (SoS) framework enabling language models to learn to solve problems by searching in language, without any external structure or components. By systematizing the elements of search into a unified language, we are able to represent various search strategies in a common format to construct a dataset with diverse streams of search. Our experiments demonstrate that training language models to search leads to superior performance compared to models trained solely on optimal trajectories. This highlights the importance of exposing models to the messy process of problem solving, with exploration and backtracking, instead of only the ideal solution steps. SoS models can then self-improve by optimizing for correctness, using STaR and APA.</p>
</div>
<div id="S7.p2" class="ltx_para ltx_noindent">
<p id="S7.p2.1" class="ltx_p">The SoS framework may address criticisms <cite class="ltx_cite ltx_citemacro_citep">(LeCun, <a href="#bib.bib12" title="" class="ltx_ref">2023</a>; Bachmann &amp; Nagarajan, <a href="#bib.bib3" title="" class="ltx_ref">2024</a>)</cite> of language models for planning and problem solving.
The problem of snowballing errors is addressed by teaching a model to backtrack. Search allows models to explore alternative paths, overcoming failures in lookahead tasks by considering multiple possible outcomes before committing to a course of action. Crucially, SoS leads language models to learn an internal ’world model’ for search. Unlike symbolic search that relies on an explicit environment model, SoS models simulate state transitions themselves.
Using a learned world models allows more adaptable and generalizable search <cite class="ltx_cite ltx_citemacro_citep">(Cf. Schrittwieser et al., <a href="#bib.bib18" title="" class="ltx_ref">2020</a>)</cite> and addresses a key criticism of pretrained LMs.</p>
</div>
<div id="S7.p3" class="ltx_para ltx_noindent">
<p id="S7.p3.1" class="ltx_p">Our empirical results were restricted to the game of Countdown. Yet Countdown, with a high branching factor and variable goal states, captures the characteristics of complex planning problems. We are optimistic that SoS extends to more challenging, real-world tasks. In the short-term, externally-structured search methods such as Tree of Thought <cite class="ltx_cite ltx_citemacro_citep">(Yao et al., <a href="#bib.bib26" title="" class="ltx_ref">2024</a>)</cite> are likely to be more efficient for these tasks; in the longer run the increased flexibility and learnability of internally-structured search (i.e. SoS) may prevail. It is also plausible these approaches can be combined, for sinstance, with with ToT providing initial training data for SoS.</p>
</div>
<div id="S7.p4" class="ltx_para ltx_noindent">
<p id="S7.p4.1" class="ltx_p">While we leave the evaluation of states to be done implicitly by the network in our current work, explicitly representing state evaluations <cite class="ltx_cite ltx_citemacro_citep">(Gandhi et al., <a href="#bib.bib8" title="" class="ltx_ref">2023</a>)</cite> and introducing other formalizable operations such as limits, summarization, cycle checks, and subgoal setting could enhance the SoS framework.
Future research could explore integrating subgoals and hierarchical planning, as well as incorporating reflection and self-evaluation to enable models to discover and improve novel search strategies <cite class="ltx_cite ltx_citemacro_citep">(Huang et al., <a href="#bib.bib11" title="" class="ltx_ref">2023</a>; Shinn et al., <a href="#bib.bib20" title="" class="ltx_ref">2024</a>; Stechly et al., <a href="#bib.bib22" title="" class="ltx_ref">2024</a>)</cite>.
Generating the initial SoS dataset can be challenging, as it is not always feasible to create symbolic search algorithms to solve problems.
An important question is how well search abilities transfer between domains and between formal and informal domains.</p>
</div>
<div id="S7.p5" class="ltx_para ltx_noindent">
<p id="S7.p5.1" class="ltx_p">In conclusion, we have shown that hallmarks of symbolic reasoning—structured search with backtracking, heuristic state evaluation and world modeling—can be achieved within a sequence modeling paradigm.
To do so requires showing language models examples of productive mistakes, not only the optimal final solutions.
By embracing the diversity of search strategies and iteratively refining models, we can unlock the potential of language models to tackle complex problems and discover new ways to solve them.</p>
</div>
</section>
<section id="Sx1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">Acknowledgements</h2>

<div id="Sx1.p1" class="ltx_para ltx_noindent">
<p id="Sx1.p1.1" class="ltx_p">We would like to thank Gabriel Poesia, Jacob Andreas, Joy He-Yueya, Dongwei Jiang, Eric Zelikman, Jan-Philipp Fränken and Ced Zhang for their discussions and support. This worked was supported by the Stanford Human-Centered Artifical Intelligence (HAI) - Google grant, and the NSF Expeditions Grant, Award Number (FAIN) 1918771.</p>
</div>
</section>
<section id="bib" class="ltx_bibliography">
<h2 class="ltx_title ltx_title_bibliography">References</h2>

<ul class="ltx_biblist">
<li id="bib.bib1" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ahn et al. (2022)</span>
<span class="ltx_bibblock">
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al.

</span>
<span class="ltx_bibblock">Do as i can, not as i say: Grounding language in robotic affordances.

</span>
<span class="ltx_bibblock"><em id="bib.bib1.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2204.01691</em>, 2022.

</span>
</li>
<li id="bib.bib2" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Arora et al. (2022)</span>
<span class="ltx_bibblock">
Kushal Arora, Layla El Asri, Hareesh Bahuleyan, and Jackie Chi Kit Cheung.

</span>
<span class="ltx_bibblock">Why exposure bias matters: An imitation learning perspective of error accumulation in language generation.

</span>
<span class="ltx_bibblock"><em id="bib.bib2.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2204.01171</em>, 2022.

</span>
</li>
<li id="bib.bib3" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bachmann &amp; Nagarajan (2024)</span>
<span class="ltx_bibblock">
Gregor Bachmann and Vaishnavh Nagarajan.

</span>
<span class="ltx_bibblock">The pitfalls of next-token prediction.

</span>
<span class="ltx_bibblock"><em id="bib.bib3.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2403.06963</em>, 2024.

</span>
</li>
<li id="bib.bib4" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Besta et al. (2023)</span>
<span class="ltx_bibblock">
Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Michal Podstawski, Hubert Niewiadomski, Piotr Nyczyk, et al.

</span>
<span class="ltx_bibblock">Graph of thoughts: Solving elaborate problems with large language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib4.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2308.09687</em>, 2023.

</span>
</li>
<li id="bib.bib5" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Countdown (2024)</span>
<span class="ltx_bibblock">
Countdown.

</span>
<span class="ltx_bibblock">Countdown (game show).

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://en.wikipedia.org/wiki/Countdown_(game_show)" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://en.wikipedia.org/wiki/Countdown_(game_show)</a>, 2024.

</span>
<span class="ltx_bibblock">[Online; accessed 29-March-2024].

</span>
</li>
<li id="bib.bib6" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dao et al. (2022)</span>
<span class="ltx_bibblock">
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré.

</span>
<span class="ltx_bibblock">Flashattention: Fast and memory-efficient exact attention with io-awareness.

</span>
<span class="ltx_bibblock"><em id="bib.bib6.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, 35:16344–16359, 2022.

</span>
</li>
<li id="bib.bib7" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Fikes &amp; Nilsson (1971)</span>
<span class="ltx_bibblock">
Richard E Fikes and Nils J Nilsson.

</span>
<span class="ltx_bibblock">Strips: A new approach to the application of theorem proving to problem solving.

</span>
<span class="ltx_bibblock"><em id="bib.bib7.1.1" class="ltx_emph ltx_font_italic">Artificial intelligence</em>, 2(3-4):189–208, 1971.

</span>
</li>
<li id="bib.bib8" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gandhi et al. (2023)</span>
<span class="ltx_bibblock">
Kanishk Gandhi, Dorsa Sadigh, and Noah D Goodman.

</span>
<span class="ltx_bibblock">Strategic reasoning with language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib8.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2305.19165</em>, 2023.

</span>
</li>
<li id="bib.bib9" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gao et al. (2020)</span>
<span class="ltx_bibblock">
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al.

</span>
<span class="ltx_bibblock">The pile: An 800gb dataset of diverse text for language modeling.

</span>
<span class="ltx_bibblock"><em id="bib.bib9.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2101.00027</em>, 2020.

</span>
</li>
<li id="bib.bib10" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gulcehre et al. (2023)</span>
<span class="ltx_bibblock">
Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, et al.

</span>
<span class="ltx_bibblock">Reinforced self-training (rest) for language modeling.

</span>
<span class="ltx_bibblock"><em id="bib.bib10.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2308.08998</em>, 2023.

</span>
</li>
<li id="bib.bib11" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Huang et al. (2023)</span>
<span class="ltx_bibblock">
Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou.

</span>
<span class="ltx_bibblock">Large language models cannot self-correct reasoning yet.

</span>
<span class="ltx_bibblock"><em id="bib.bib11.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2310.01798</em>, 2023.

</span>
</li>
<li id="bib.bib12" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">LeCun (2023)</span>
<span class="ltx_bibblock">
Y LeCun.

</span>
<span class="ltx_bibblock">Do large language models need sensory grounding for meaning and understanding.

</span>
<span class="ltx_bibblock">In <em id="bib.bib12.1.1" class="ltx_emph ltx_font_italic">Workshop on Philosophy of Deep Learning, NYU Center for Mind, Brain, and Consciousness and the Columbia Center for Science and Society</em>, 2023.

</span>
</li>
<li id="bib.bib13" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lehnert et al. (2024)</span>
<span class="ltx_bibblock">
Lucas Lehnert, Sainbayar Sukhbaatar, Paul Mcvay, Michael Rabbat, and Yuandong Tian.

</span>
<span class="ltx_bibblock">Beyond a*: Better planning with transformers via search dynamics bootstrapping.

</span>
<span class="ltx_bibblock"><em id="bib.bib13.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2402.14083</em>, 2024.

</span>
</li>
<li id="bib.bib14" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lightman et al. (2023)</span>
<span class="ltx_bibblock">
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe.

</span>
<span class="ltx_bibblock">Let’s verify step by step.

</span>
<span class="ltx_bibblock"><em id="bib.bib14.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2305.20050</em>, 2023.

</span>
</li>
<li id="bib.bib15" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Momennejad et al. (2024)</span>
<span class="ltx_bibblock">
Ida Momennejad, Hosein Hasanbeig, Felipe Vieira Frujeri, Hiteshi Sharma, Nebojsa Jojic, Hamid Palangi, Robert Ness, and Jonathan Larson.

</span>
<span class="ltx_bibblock">Evaluating cognitive maps and planning in large language models with cogeval.

</span>
<span class="ltx_bibblock"><em id="bib.bib15.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, 36, 2024.

</span>
</li>
<li id="bib.bib16" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Pallagani et al. (2023)</span>
<span class="ltx_bibblock">
Vishal Pallagani, Bharath Muppasani, Keerthiram Murugesan, Francesca Rossi, Biplav Srivastava, Lior Horesh, Francesco Fabiano, and Andrea Loreggia.

</span>
<span class="ltx_bibblock">Understanding the capabilities of large language models for automated planning.

</span>
<span class="ltx_bibblock"><em id="bib.bib16.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2305.16151</em>, 2023.

</span>
</li>
<li id="bib.bib17" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ross et al. (2011)</span>
<span class="ltx_bibblock">
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell.

</span>
<span class="ltx_bibblock">A reduction of imitation learning and structured prediction to no-regret online learning.

</span>
<span class="ltx_bibblock">In <em id="bib.bib17.1.1" class="ltx_emph ltx_font_italic">Proceedings of the fourteenth international conference on artificial intelligence and statistics</em>, pp.  627–635. JMLR Workshop and Conference Proceedings, 2011.

</span>
</li>
<li id="bib.bib18" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Schrittwieser et al. (2020)</span>
<span class="ltx_bibblock">
Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al.

</span>
<span class="ltx_bibblock">Mastering atari, go, chess and shogi by planning with a learned model.

</span>
<span class="ltx_bibblock"><em id="bib.bib18.1.1" class="ltx_emph ltx_font_italic">Nature</em>, 588(7839):604–609, 2020.

</span>
</li>
<li id="bib.bib19" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Sel et al. (2023)</span>
<span class="ltx_bibblock">
Bilgehan Sel, Ahmad Al-Tawaha, Vanshaj Khattar, Lu Wang, Ruoxi Jia, and Ming Jin.

</span>
<span class="ltx_bibblock">Algorithm of thoughts: Enhancing exploration of ideas in large language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib19.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2308.10379</em>, 2023.

</span>
</li>
<li id="bib.bib20" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Shinn et al. (2024)</span>
<span class="ltx_bibblock">
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao.

</span>
<span class="ltx_bibblock">Reflexion: Language agents with verbal reinforcement learning.

</span>
<span class="ltx_bibblock"><em id="bib.bib20.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, 36, 2024.

</span>
</li>
<li id="bib.bib21" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Silver et al. (2018)</span>
<span class="ltx_bibblock">
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al.

</span>
<span class="ltx_bibblock">A general reinforcement learning algorithm that masters chess, shogi, and go through self-play.

</span>
<span class="ltx_bibblock"><em id="bib.bib21.1.1" class="ltx_emph ltx_font_italic">Science</em>, 362(6419):1140–1144, 2018.

</span>
</li>
<li id="bib.bib22" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Stechly et al. (2024)</span>
<span class="ltx_bibblock">
Kaya Stechly, Karthik Valmeekam, and Subbarao Kambhampati.

</span>
<span class="ltx_bibblock">On the self-verification limitations of large language models on reasoning and planning tasks.

</span>
<span class="ltx_bibblock"><em id="bib.bib22.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2402.08115</em>, 2024.

</span>
</li>
<li id="bib.bib23" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Sutton &amp; Barto (2018)</span>
<span class="ltx_bibblock">
Richard S Sutton and Andrew G Barto.

</span>
<span class="ltx_bibblock"><em id="bib.bib23.1.1" class="ltx_emph ltx_font_italic">Reinforcement learning: An introduction</em>.

</span>
<span class="ltx_bibblock">MIT press, 2018.

</span>
</li>
<li id="bib.bib24" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Valmeekam et al. (2024)</span>
<span class="ltx_bibblock">
Karthik Valmeekam, Matthew Marquez, Sarath Sreedharan, and Subbarao Kambhampati.

</span>
<span class="ltx_bibblock">On the planning abilities of large language models-a critical investigation.

</span>
<span class="ltx_bibblock"><em id="bib.bib24.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, 36, 2024.

</span>
</li>
<li id="bib.bib25" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yang et al. (2022)</span>
<span class="ltx_bibblock">
Mengjiao Sherry Yang, Dale Schuurmans, Pieter Abbeel, and Ofir Nachum.

</span>
<span class="ltx_bibblock">Chain of thought imitation with procedure cloning.

</span>
<span class="ltx_bibblock"><em id="bib.bib25.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, 35:36366–36381, 2022.

</span>
</li>
<li id="bib.bib26" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yao et al. (2024)</span>
<span class="ltx_bibblock">
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan.

</span>
<span class="ltx_bibblock">Tree of thoughts: Deliberate problem solving with large language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib26.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, 36, 2024.

</span>
</li>
<li id="bib.bib27" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zelikman et al. (2022)</span>
<span class="ltx_bibblock">
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman.

</span>
<span class="ltx_bibblock">Star: Bootstrapping reasoning with reasoning.

</span>
<span class="ltx_bibblock"><em id="bib.bib27.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, 35:15476–15488, 2022.

</span>
</li>
<li id="bib.bib28" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhu et al. (2023)</span>
<span class="ltx_bibblock">
Banghua Zhu, Hiteshi Sharma, Felipe Vieira Frujeri, Shi Dong, Chenguang Zhu, Michael I Jordan, and Jiantao Jiao.

</span>
<span class="ltx_bibblock">Fine-tuning language models with advantage-induced policy alignment.

</span>
<span class="ltx_bibblock"><em id="bib.bib28.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2306.02231</em>, 2023.

</span>
</li>
</ul>
</section>
<div class="ltx_pagination ltx_role_newpage"></div>
<section id="A1" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix A </span>Reproducibility Statement</h2>

<div id="A1.p1" class="ltx_para ltx_noindent">
<p id="A1.p1.1" class="ltx_p">We release the code to replicate our experiments here: <a target="_blank" href="https://github.com/kanishkg/stream-of-search" title="" class="ltx_ref ltx_href">https://github.com/kanishkg/stream-of-search</a>. We use the APA implementation available here: <a target="_blank" href="https://github.com/microsoft/RLHF-APA/tree/main/examples" title="" class="ltx_ref ltx_href">https://github.com/microsoft/RLHF-APA/tree/main/examples</a>. We optimize their code for parallelism, memory, mixed precision training and to use flash-attention <cite class="ltx_cite ltx_citemacro_citep">(Dao et al., <a href="#bib.bib6" title="" class="ltx_ref">2022</a>)</cite>. See our training fork here: <a target="_blank" href="https://github.com/kanishkg/RLHF-APA" title="" class="ltx_ref ltx_href">https://github.com/kanishkg/RLHF-APA</a>.</p>
</div>
</section>
<section id="A2" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix B </span>Creation of the SoS Dataset</h2>

<div id="A2.p1" class="ltx_para ltx_noindent">
<p id="A2.p1.9" class="ltx_p"><span id="A2.p1.9.1" class="ltx_text ltx_font_bold">SoS Dataset.</span>To generate the Stream of Search dataset, we use two symbolic strategies, BFS (<a href="#alg4" title="Algorithm 4 ‣ Appendix B Creation of the SoS Dataset ‣ Stream of Search (SoS): Learning to Search in Language" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Alg. 4</span></a>) and DFS (<a href="#alg3" title="Algorithm 3 ‣ Appendix B Creation of the SoS Dataset ‣ Stream of Search (SoS): Learning to Search in Language" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Alg. 3</span></a>). These strategies are guided by two heuristic functinos to generate trajectories to create the SoS training set. Let the input numbers for the Countdown problem be <math id="A2.p1.1.m1.3" class="ltx_Math" alttext="I=[n_{1},n_{2},\dots n_{k}]" display="inline"><semantics id="A2.p1.1.m1.3a"><mrow id="A2.p1.1.m1.3.3" xref="A2.p1.1.m1.3.3.cmml"><mi id="A2.p1.1.m1.3.3.5" xref="A2.p1.1.m1.3.3.5.cmml">I</mi><mo id="A2.p1.1.m1.3.3.4" xref="A2.p1.1.m1.3.3.4.cmml">=</mo><mrow id="A2.p1.1.m1.3.3.3.3" xref="A2.p1.1.m1.3.3.3.4.cmml"><mo stretchy="false" id="A2.p1.1.m1.3.3.3.3.4" xref="A2.p1.1.m1.3.3.3.4.cmml">[</mo><msub id="A2.p1.1.m1.1.1.1.1.1" xref="A2.p1.1.m1.1.1.1.1.1.cmml"><mi id="A2.p1.1.m1.1.1.1.1.1.2" xref="A2.p1.1.m1.1.1.1.1.1.2.cmml">n</mi><mn id="A2.p1.1.m1.1.1.1.1.1.3" xref="A2.p1.1.m1.1.1.1.1.1.3.cmml">1</mn></msub><mo id="A2.p1.1.m1.3.3.3.3.5" xref="A2.p1.1.m1.3.3.3.4.cmml">,</mo><msub id="A2.p1.1.m1.2.2.2.2.2" xref="A2.p1.1.m1.2.2.2.2.2.cmml"><mi id="A2.p1.1.m1.2.2.2.2.2.2" xref="A2.p1.1.m1.2.2.2.2.2.2.cmml">n</mi><mn id="A2.p1.1.m1.2.2.2.2.2.3" xref="A2.p1.1.m1.2.2.2.2.2.3.cmml">2</mn></msub><mo id="A2.p1.1.m1.3.3.3.3.6" xref="A2.p1.1.m1.3.3.3.4.cmml">,</mo><mrow id="A2.p1.1.m1.3.3.3.3.3" xref="A2.p1.1.m1.3.3.3.3.3.cmml"><mi mathvariant="normal" id="A2.p1.1.m1.3.3.3.3.3.2" xref="A2.p1.1.m1.3.3.3.3.3.2.cmml">…</mi><mo lspace="0em" rspace="0em" id="A2.p1.1.m1.3.3.3.3.3.1" xref="A2.p1.1.m1.3.3.3.3.3.1.cmml">​</mo><msub id="A2.p1.1.m1.3.3.3.3.3.3" xref="A2.p1.1.m1.3.3.3.3.3.3.cmml"><mi id="A2.p1.1.m1.3.3.3.3.3.3.2" xref="A2.p1.1.m1.3.3.3.3.3.3.2.cmml">n</mi><mi id="A2.p1.1.m1.3.3.3.3.3.3.3" xref="A2.p1.1.m1.3.3.3.3.3.3.3.cmml">k</mi></msub></mrow><mo stretchy="false" id="A2.p1.1.m1.3.3.3.3.7" xref="A2.p1.1.m1.3.3.3.4.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A2.p1.1.m1.3b"><apply id="A2.p1.1.m1.3.3.cmml" xref="A2.p1.1.m1.3.3"><eq id="A2.p1.1.m1.3.3.4.cmml" xref="A2.p1.1.m1.3.3.4"></eq><ci id="A2.p1.1.m1.3.3.5.cmml" xref="A2.p1.1.m1.3.3.5">𝐼</ci><list id="A2.p1.1.m1.3.3.3.4.cmml" xref="A2.p1.1.m1.3.3.3.3"><apply id="A2.p1.1.m1.1.1.1.1.1.cmml" xref="A2.p1.1.m1.1.1.1.1.1"><csymbol cd="ambiguous" id="A2.p1.1.m1.1.1.1.1.1.1.cmml" xref="A2.p1.1.m1.1.1.1.1.1">subscript</csymbol><ci id="A2.p1.1.m1.1.1.1.1.1.2.cmml" xref="A2.p1.1.m1.1.1.1.1.1.2">𝑛</ci><cn type="integer" id="A2.p1.1.m1.1.1.1.1.1.3.cmml" xref="A2.p1.1.m1.1.1.1.1.1.3">1</cn></apply><apply id="A2.p1.1.m1.2.2.2.2.2.cmml" xref="A2.p1.1.m1.2.2.2.2.2"><csymbol cd="ambiguous" id="A2.p1.1.m1.2.2.2.2.2.1.cmml" xref="A2.p1.1.m1.2.2.2.2.2">subscript</csymbol><ci id="A2.p1.1.m1.2.2.2.2.2.2.cmml" xref="A2.p1.1.m1.2.2.2.2.2.2">𝑛</ci><cn type="integer" id="A2.p1.1.m1.2.2.2.2.2.3.cmml" xref="A2.p1.1.m1.2.2.2.2.2.3">2</cn></apply><apply id="A2.p1.1.m1.3.3.3.3.3.cmml" xref="A2.p1.1.m1.3.3.3.3.3"><times id="A2.p1.1.m1.3.3.3.3.3.1.cmml" xref="A2.p1.1.m1.3.3.3.3.3.1"></times><ci id="A2.p1.1.m1.3.3.3.3.3.2.cmml" xref="A2.p1.1.m1.3.3.3.3.3.2">…</ci><apply id="A2.p1.1.m1.3.3.3.3.3.3.cmml" xref="A2.p1.1.m1.3.3.3.3.3.3"><csymbol cd="ambiguous" id="A2.p1.1.m1.3.3.3.3.3.3.1.cmml" xref="A2.p1.1.m1.3.3.3.3.3.3">subscript</csymbol><ci id="A2.p1.1.m1.3.3.3.3.3.3.2.cmml" xref="A2.p1.1.m1.3.3.3.3.3.3.2">𝑛</ci><ci id="A2.p1.1.m1.3.3.3.3.3.3.3.cmml" xref="A2.p1.1.m1.3.3.3.3.3.3.3">𝑘</ci></apply></apply></list></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.p1.1.m1.3c">I=[n_{1},n_{2},\dots n_{k}]</annotation></semantics></math>, and the target number be <math id="A2.p1.2.m2.1" class="ltx_Math" alttext="T" display="inline"><semantics id="A2.p1.2.m2.1a"><mi id="A2.p1.2.m2.1.1" xref="A2.p1.2.m2.1.1.cmml">T</mi><annotation-xml encoding="MathML-Content" id="A2.p1.2.m2.1b"><ci id="A2.p1.2.m2.1.1.cmml" xref="A2.p1.2.m2.1.1">𝑇</ci></annotation-xml><annotation encoding="application/x-tex" id="A2.p1.2.m2.1c">T</annotation></semantics></math>. The sum heuristic measures the distance between the sum of all input numbers to the target number: <math id="A2.p1.3.m3.3" class="ltx_Math" alttext="h_{sum}(I,T)=|T-\sum_{i=1}^{i=k}n_{i}|" display="inline"><semantics id="A2.p1.3.m3.3a"><mrow id="A2.p1.3.m3.3.3" xref="A2.p1.3.m3.3.3.cmml"><mrow id="A2.p1.3.m3.3.3.3" xref="A2.p1.3.m3.3.3.3.cmml"><msub id="A2.p1.3.m3.3.3.3.2" xref="A2.p1.3.m3.3.3.3.2.cmml"><mi id="A2.p1.3.m3.3.3.3.2.2" xref="A2.p1.3.m3.3.3.3.2.2.cmml">h</mi><mrow id="A2.p1.3.m3.3.3.3.2.3" xref="A2.p1.3.m3.3.3.3.2.3.cmml"><mi id="A2.p1.3.m3.3.3.3.2.3.2" xref="A2.p1.3.m3.3.3.3.2.3.2.cmml">s</mi><mo lspace="0em" rspace="0em" id="A2.p1.3.m3.3.3.3.2.3.1" xref="A2.p1.3.m3.3.3.3.2.3.1.cmml">​</mo><mi id="A2.p1.3.m3.3.3.3.2.3.3" xref="A2.p1.3.m3.3.3.3.2.3.3.cmml">u</mi><mo lspace="0em" rspace="0em" id="A2.p1.3.m3.3.3.3.2.3.1a" xref="A2.p1.3.m3.3.3.3.2.3.1.cmml">​</mo><mi id="A2.p1.3.m3.3.3.3.2.3.4" xref="A2.p1.3.m3.3.3.3.2.3.4.cmml">m</mi></mrow></msub><mo lspace="0em" rspace="0em" id="A2.p1.3.m3.3.3.3.1" xref="A2.p1.3.m3.3.3.3.1.cmml">​</mo><mrow id="A2.p1.3.m3.3.3.3.3.2" xref="A2.p1.3.m3.3.3.3.3.1.cmml"><mo stretchy="false" id="A2.p1.3.m3.3.3.3.3.2.1" xref="A2.p1.3.m3.3.3.3.3.1.cmml">(</mo><mi id="A2.p1.3.m3.1.1" xref="A2.p1.3.m3.1.1.cmml">I</mi><mo id="A2.p1.3.m3.3.3.3.3.2.2" xref="A2.p1.3.m3.3.3.3.3.1.cmml">,</mo><mi id="A2.p1.3.m3.2.2" xref="A2.p1.3.m3.2.2.cmml">T</mi><mo stretchy="false" id="A2.p1.3.m3.3.3.3.3.2.3" xref="A2.p1.3.m3.3.3.3.3.1.cmml">)</mo></mrow></mrow><mo id="A2.p1.3.m3.3.3.2" xref="A2.p1.3.m3.3.3.2.cmml">=</mo><mrow id="A2.p1.3.m3.3.3.1.1" xref="A2.p1.3.m3.3.3.1.2.cmml"><mo stretchy="false" id="A2.p1.3.m3.3.3.1.1.2" xref="A2.p1.3.m3.3.3.1.2.1.cmml">|</mo><mrow id="A2.p1.3.m3.3.3.1.1.1" xref="A2.p1.3.m3.3.3.1.1.1.cmml"><mi id="A2.p1.3.m3.3.3.1.1.1.2" xref="A2.p1.3.m3.3.3.1.1.1.2.cmml">T</mi><mo rspace="0.055em" id="A2.p1.3.m3.3.3.1.1.1.1" xref="A2.p1.3.m3.3.3.1.1.1.1.cmml">−</mo><mrow id="A2.p1.3.m3.3.3.1.1.1.3" xref="A2.p1.3.m3.3.3.1.1.1.3.cmml"><msubsup id="A2.p1.3.m3.3.3.1.1.1.3.1" xref="A2.p1.3.m3.3.3.1.1.1.3.1.cmml"><mo id="A2.p1.3.m3.3.3.1.1.1.3.1.2.2" xref="A2.p1.3.m3.3.3.1.1.1.3.1.2.2.cmml">∑</mo><mrow id="A2.p1.3.m3.3.3.1.1.1.3.1.2.3" xref="A2.p1.3.m3.3.3.1.1.1.3.1.2.3.cmml"><mi id="A2.p1.3.m3.3.3.1.1.1.3.1.2.3.2" xref="A2.p1.3.m3.3.3.1.1.1.3.1.2.3.2.cmml">i</mi><mo id="A2.p1.3.m3.3.3.1.1.1.3.1.2.3.1" xref="A2.p1.3.m3.3.3.1.1.1.3.1.2.3.1.cmml">=</mo><mn id="A2.p1.3.m3.3.3.1.1.1.3.1.2.3.3" xref="A2.p1.3.m3.3.3.1.1.1.3.1.2.3.3.cmml">1</mn></mrow><mrow id="A2.p1.3.m3.3.3.1.1.1.3.1.3" xref="A2.p1.3.m3.3.3.1.1.1.3.1.3.cmml"><mi id="A2.p1.3.m3.3.3.1.1.1.3.1.3.2" xref="A2.p1.3.m3.3.3.1.1.1.3.1.3.2.cmml">i</mi><mo id="A2.p1.3.m3.3.3.1.1.1.3.1.3.1" xref="A2.p1.3.m3.3.3.1.1.1.3.1.3.1.cmml">=</mo><mi id="A2.p1.3.m3.3.3.1.1.1.3.1.3.3" xref="A2.p1.3.m3.3.3.1.1.1.3.1.3.3.cmml">k</mi></mrow></msubsup><msub id="A2.p1.3.m3.3.3.1.1.1.3.2" xref="A2.p1.3.m3.3.3.1.1.1.3.2.cmml"><mi id="A2.p1.3.m3.3.3.1.1.1.3.2.2" xref="A2.p1.3.m3.3.3.1.1.1.3.2.2.cmml">n</mi><mi id="A2.p1.3.m3.3.3.1.1.1.3.2.3" xref="A2.p1.3.m3.3.3.1.1.1.3.2.3.cmml">i</mi></msub></mrow></mrow><mo stretchy="false" id="A2.p1.3.m3.3.3.1.1.3" xref="A2.p1.3.m3.3.3.1.2.1.cmml">|</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A2.p1.3.m3.3b"><apply id="A2.p1.3.m3.3.3.cmml" xref="A2.p1.3.m3.3.3"><eq id="A2.p1.3.m3.3.3.2.cmml" xref="A2.p1.3.m3.3.3.2"></eq><apply id="A2.p1.3.m3.3.3.3.cmml" xref="A2.p1.3.m3.3.3.3"><times id="A2.p1.3.m3.3.3.3.1.cmml" xref="A2.p1.3.m3.3.3.3.1"></times><apply id="A2.p1.3.m3.3.3.3.2.cmml" xref="A2.p1.3.m3.3.3.3.2"><csymbol cd="ambiguous" id="A2.p1.3.m3.3.3.3.2.1.cmml" xref="A2.p1.3.m3.3.3.3.2">subscript</csymbol><ci id="A2.p1.3.m3.3.3.3.2.2.cmml" xref="A2.p1.3.m3.3.3.3.2.2">ℎ</ci><apply id="A2.p1.3.m3.3.3.3.2.3.cmml" xref="A2.p1.3.m3.3.3.3.2.3"><times id="A2.p1.3.m3.3.3.3.2.3.1.cmml" xref="A2.p1.3.m3.3.3.3.2.3.1"></times><ci id="A2.p1.3.m3.3.3.3.2.3.2.cmml" xref="A2.p1.3.m3.3.3.3.2.3.2">𝑠</ci><ci id="A2.p1.3.m3.3.3.3.2.3.3.cmml" xref="A2.p1.3.m3.3.3.3.2.3.3">𝑢</ci><ci id="A2.p1.3.m3.3.3.3.2.3.4.cmml" xref="A2.p1.3.m3.3.3.3.2.3.4">𝑚</ci></apply></apply><interval closure="open" id="A2.p1.3.m3.3.3.3.3.1.cmml" xref="A2.p1.3.m3.3.3.3.3.2"><ci id="A2.p1.3.m3.1.1.cmml" xref="A2.p1.3.m3.1.1">𝐼</ci><ci id="A2.p1.3.m3.2.2.cmml" xref="A2.p1.3.m3.2.2">𝑇</ci></interval></apply><apply id="A2.p1.3.m3.3.3.1.2.cmml" xref="A2.p1.3.m3.3.3.1.1"><abs id="A2.p1.3.m3.3.3.1.2.1.cmml" xref="A2.p1.3.m3.3.3.1.1.2"></abs><apply id="A2.p1.3.m3.3.3.1.1.1.cmml" xref="A2.p1.3.m3.3.3.1.1.1"><minus id="A2.p1.3.m3.3.3.1.1.1.1.cmml" xref="A2.p1.3.m3.3.3.1.1.1.1"></minus><ci id="A2.p1.3.m3.3.3.1.1.1.2.cmml" xref="A2.p1.3.m3.3.3.1.1.1.2">𝑇</ci><apply id="A2.p1.3.m3.3.3.1.1.1.3.cmml" xref="A2.p1.3.m3.3.3.1.1.1.3"><apply id="A2.p1.3.m3.3.3.1.1.1.3.1.cmml" xref="A2.p1.3.m3.3.3.1.1.1.3.1"><csymbol cd="ambiguous" id="A2.p1.3.m3.3.3.1.1.1.3.1.1.cmml" xref="A2.p1.3.m3.3.3.1.1.1.3.1">superscript</csymbol><apply id="A2.p1.3.m3.3.3.1.1.1.3.1.2.cmml" xref="A2.p1.3.m3.3.3.1.1.1.3.1"><csymbol cd="ambiguous" id="A2.p1.3.m3.3.3.1.1.1.3.1.2.1.cmml" xref="A2.p1.3.m3.3.3.1.1.1.3.1">subscript</csymbol><sum id="A2.p1.3.m3.3.3.1.1.1.3.1.2.2.cmml" xref="A2.p1.3.m3.3.3.1.1.1.3.1.2.2"></sum><apply id="A2.p1.3.m3.3.3.1.1.1.3.1.2.3.cmml" xref="A2.p1.3.m3.3.3.1.1.1.3.1.2.3"><eq id="A2.p1.3.m3.3.3.1.1.1.3.1.2.3.1.cmml" xref="A2.p1.3.m3.3.3.1.1.1.3.1.2.3.1"></eq><ci id="A2.p1.3.m3.3.3.1.1.1.3.1.2.3.2.cmml" xref="A2.p1.3.m3.3.3.1.1.1.3.1.2.3.2">𝑖</ci><cn type="integer" id="A2.p1.3.m3.3.3.1.1.1.3.1.2.3.3.cmml" xref="A2.p1.3.m3.3.3.1.1.1.3.1.2.3.3">1</cn></apply></apply><apply id="A2.p1.3.m3.3.3.1.1.1.3.1.3.cmml" xref="A2.p1.3.m3.3.3.1.1.1.3.1.3"><eq id="A2.p1.3.m3.3.3.1.1.1.3.1.3.1.cmml" xref="A2.p1.3.m3.3.3.1.1.1.3.1.3.1"></eq><ci id="A2.p1.3.m3.3.3.1.1.1.3.1.3.2.cmml" xref="A2.p1.3.m3.3.3.1.1.1.3.1.3.2">𝑖</ci><ci id="A2.p1.3.m3.3.3.1.1.1.3.1.3.3.cmml" xref="A2.p1.3.m3.3.3.1.1.1.3.1.3.3">𝑘</ci></apply></apply><apply id="A2.p1.3.m3.3.3.1.1.1.3.2.cmml" xref="A2.p1.3.m3.3.3.1.1.1.3.2"><csymbol cd="ambiguous" id="A2.p1.3.m3.3.3.1.1.1.3.2.1.cmml" xref="A2.p1.3.m3.3.3.1.1.1.3.2">subscript</csymbol><ci id="A2.p1.3.m3.3.3.1.1.1.3.2.2.cmml" xref="A2.p1.3.m3.3.3.1.1.1.3.2.2">𝑛</ci><ci id="A2.p1.3.m3.3.3.1.1.1.3.2.3.cmml" xref="A2.p1.3.m3.3.3.1.1.1.3.2.3">𝑖</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.p1.3.m3.3c">h_{sum}(I,T)=|T-\sum_{i=1}^{i=k}n_{i}|</annotation></semantics></math>. The multiply heuristic measures the minimum distance between the sum of inputs and the factors of the target. If <math id="A2.p1.4.m4.1" class="ltx_Math" alttext="T" display="inline"><semantics id="A2.p1.4.m4.1a"><mi id="A2.p1.4.m4.1.1" xref="A2.p1.4.m4.1.1.cmml">T</mi><annotation-xml encoding="MathML-Content" id="A2.p1.4.m4.1b"><ci id="A2.p1.4.m4.1.1.cmml" xref="A2.p1.4.m4.1.1">𝑇</ci></annotation-xml><annotation encoding="application/x-tex" id="A2.p1.4.m4.1c">T</annotation></semantics></math> has factors <math id="A2.p1.5.m5.2" class="ltx_Math" alttext="[f_{1},f_{2}\dots f_{m}]" display="inline"><semantics id="A2.p1.5.m5.2a"><mrow id="A2.p1.5.m5.2.2.2" xref="A2.p1.5.m5.2.2.3.cmml"><mo stretchy="false" id="A2.p1.5.m5.2.2.2.3" xref="A2.p1.5.m5.2.2.3.cmml">[</mo><msub id="A2.p1.5.m5.1.1.1.1" xref="A2.p1.5.m5.1.1.1.1.cmml"><mi id="A2.p1.5.m5.1.1.1.1.2" xref="A2.p1.5.m5.1.1.1.1.2.cmml">f</mi><mn id="A2.p1.5.m5.1.1.1.1.3" xref="A2.p1.5.m5.1.1.1.1.3.cmml">1</mn></msub><mo id="A2.p1.5.m5.2.2.2.4" xref="A2.p1.5.m5.2.2.3.cmml">,</mo><mrow id="A2.p1.5.m5.2.2.2.2" xref="A2.p1.5.m5.2.2.2.2.cmml"><msub id="A2.p1.5.m5.2.2.2.2.2" xref="A2.p1.5.m5.2.2.2.2.2.cmml"><mi id="A2.p1.5.m5.2.2.2.2.2.2" xref="A2.p1.5.m5.2.2.2.2.2.2.cmml">f</mi><mn id="A2.p1.5.m5.2.2.2.2.2.3" xref="A2.p1.5.m5.2.2.2.2.2.3.cmml">2</mn></msub><mo lspace="0em" rspace="0em" id="A2.p1.5.m5.2.2.2.2.1" xref="A2.p1.5.m5.2.2.2.2.1.cmml">​</mo><mi mathvariant="normal" id="A2.p1.5.m5.2.2.2.2.3" xref="A2.p1.5.m5.2.2.2.2.3.cmml">…</mi><mo lspace="0em" rspace="0em" id="A2.p1.5.m5.2.2.2.2.1a" xref="A2.p1.5.m5.2.2.2.2.1.cmml">​</mo><msub id="A2.p1.5.m5.2.2.2.2.4" xref="A2.p1.5.m5.2.2.2.2.4.cmml"><mi id="A2.p1.5.m5.2.2.2.2.4.2" xref="A2.p1.5.m5.2.2.2.2.4.2.cmml">f</mi><mi id="A2.p1.5.m5.2.2.2.2.4.3" xref="A2.p1.5.m5.2.2.2.2.4.3.cmml">m</mi></msub></mrow><mo stretchy="false" id="A2.p1.5.m5.2.2.2.5" xref="A2.p1.5.m5.2.2.3.cmml">]</mo></mrow><annotation-xml encoding="MathML-Content" id="A2.p1.5.m5.2b"><interval closure="closed" id="A2.p1.5.m5.2.2.3.cmml" xref="A2.p1.5.m5.2.2.2"><apply id="A2.p1.5.m5.1.1.1.1.cmml" xref="A2.p1.5.m5.1.1.1.1"><csymbol cd="ambiguous" id="A2.p1.5.m5.1.1.1.1.1.cmml" xref="A2.p1.5.m5.1.1.1.1">subscript</csymbol><ci id="A2.p1.5.m5.1.1.1.1.2.cmml" xref="A2.p1.5.m5.1.1.1.1.2">𝑓</ci><cn type="integer" id="A2.p1.5.m5.1.1.1.1.3.cmml" xref="A2.p1.5.m5.1.1.1.1.3">1</cn></apply><apply id="A2.p1.5.m5.2.2.2.2.cmml" xref="A2.p1.5.m5.2.2.2.2"><times id="A2.p1.5.m5.2.2.2.2.1.cmml" xref="A2.p1.5.m5.2.2.2.2.1"></times><apply id="A2.p1.5.m5.2.2.2.2.2.cmml" xref="A2.p1.5.m5.2.2.2.2.2"><csymbol cd="ambiguous" id="A2.p1.5.m5.2.2.2.2.2.1.cmml" xref="A2.p1.5.m5.2.2.2.2.2">subscript</csymbol><ci id="A2.p1.5.m5.2.2.2.2.2.2.cmml" xref="A2.p1.5.m5.2.2.2.2.2.2">𝑓</ci><cn type="integer" id="A2.p1.5.m5.2.2.2.2.2.3.cmml" xref="A2.p1.5.m5.2.2.2.2.2.3">2</cn></apply><ci id="A2.p1.5.m5.2.2.2.2.3.cmml" xref="A2.p1.5.m5.2.2.2.2.3">…</ci><apply id="A2.p1.5.m5.2.2.2.2.4.cmml" xref="A2.p1.5.m5.2.2.2.2.4"><csymbol cd="ambiguous" id="A2.p1.5.m5.2.2.2.2.4.1.cmml" xref="A2.p1.5.m5.2.2.2.2.4">subscript</csymbol><ci id="A2.p1.5.m5.2.2.2.2.4.2.cmml" xref="A2.p1.5.m5.2.2.2.2.4.2">𝑓</ci><ci id="A2.p1.5.m5.2.2.2.2.4.3.cmml" xref="A2.p1.5.m5.2.2.2.2.4.3">𝑚</ci></apply></apply></interval></annotation-xml><annotation encoding="application/x-tex" id="A2.p1.5.m5.2c">[f_{1},f_{2}\dots f_{m}]</annotation></semantics></math>. So, the multiply heuristic can be written as <math id="A2.p1.6.m6.3" class="ltx_Math" alttext="h_{multiply}=\text{min}([|f_{j}-\sum_{i=1}^{i=k}n_{i}|]\forall j\in[1,m])" display="inline"><semantics id="A2.p1.6.m6.3a"><mrow id="A2.p1.6.m6.3.3" xref="A2.p1.6.m6.3.3.cmml"><msub id="A2.p1.6.m6.3.3.3" xref="A2.p1.6.m6.3.3.3.cmml"><mi id="A2.p1.6.m6.3.3.3.2" xref="A2.p1.6.m6.3.3.3.2.cmml">h</mi><mrow id="A2.p1.6.m6.3.3.3.3" xref="A2.p1.6.m6.3.3.3.3.cmml"><mi id="A2.p1.6.m6.3.3.3.3.2" xref="A2.p1.6.m6.3.3.3.3.2.cmml">m</mi><mo lspace="0em" rspace="0em" id="A2.p1.6.m6.3.3.3.3.1" xref="A2.p1.6.m6.3.3.3.3.1.cmml">​</mo><mi id="A2.p1.6.m6.3.3.3.3.3" xref="A2.p1.6.m6.3.3.3.3.3.cmml">u</mi><mo lspace="0em" rspace="0em" id="A2.p1.6.m6.3.3.3.3.1a" xref="A2.p1.6.m6.3.3.3.3.1.cmml">​</mo><mi id="A2.p1.6.m6.3.3.3.3.4" xref="A2.p1.6.m6.3.3.3.3.4.cmml">l</mi><mo lspace="0em" rspace="0em" id="A2.p1.6.m6.3.3.3.3.1b" xref="A2.p1.6.m6.3.3.3.3.1.cmml">​</mo><mi id="A2.p1.6.m6.3.3.3.3.5" xref="A2.p1.6.m6.3.3.3.3.5.cmml">t</mi><mo lspace="0em" rspace="0em" id="A2.p1.6.m6.3.3.3.3.1c" xref="A2.p1.6.m6.3.3.3.3.1.cmml">​</mo><mi id="A2.p1.6.m6.3.3.3.3.6" xref="A2.p1.6.m6.3.3.3.3.6.cmml">i</mi><mo lspace="0em" rspace="0em" id="A2.p1.6.m6.3.3.3.3.1d" xref="A2.p1.6.m6.3.3.3.3.1.cmml">​</mo><mi id="A2.p1.6.m6.3.3.3.3.7" xref="A2.p1.6.m6.3.3.3.3.7.cmml">p</mi><mo lspace="0em" rspace="0em" id="A2.p1.6.m6.3.3.3.3.1e" xref="A2.p1.6.m6.3.3.3.3.1.cmml">​</mo><mi id="A2.p1.6.m6.3.3.3.3.8" xref="A2.p1.6.m6.3.3.3.3.8.cmml">l</mi><mo lspace="0em" rspace="0em" id="A2.p1.6.m6.3.3.3.3.1f" xref="A2.p1.6.m6.3.3.3.3.1.cmml">​</mo><mi id="A2.p1.6.m6.3.3.3.3.9" xref="A2.p1.6.m6.3.3.3.3.9.cmml">y</mi></mrow></msub><mo id="A2.p1.6.m6.3.3.2" xref="A2.p1.6.m6.3.3.2.cmml">=</mo><mrow id="A2.p1.6.m6.3.3.1" xref="A2.p1.6.m6.3.3.1.cmml"><mtext id="A2.p1.6.m6.3.3.1.3" xref="A2.p1.6.m6.3.3.1.3a.cmml">min</mtext><mo lspace="0em" rspace="0em" id="A2.p1.6.m6.3.3.1.2" xref="A2.p1.6.m6.3.3.1.2.cmml">​</mo><mrow id="A2.p1.6.m6.3.3.1.1.1" xref="A2.p1.6.m6.3.3.1.1.1.1.cmml"><mo stretchy="false" id="A2.p1.6.m6.3.3.1.1.1.2" xref="A2.p1.6.m6.3.3.1.1.1.1.cmml">(</mo><mrow id="A2.p1.6.m6.3.3.1.1.1.1" xref="A2.p1.6.m6.3.3.1.1.1.1.cmml"><mrow id="A2.p1.6.m6.3.3.1.1.1.1.1" xref="A2.p1.6.m6.3.3.1.1.1.1.1.cmml"><mrow id="A2.p1.6.m6.3.3.1.1.1.1.1.1.1" xref="A2.p1.6.m6.3.3.1.1.1.1.1.1.2.cmml"><mo stretchy="false" id="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.2" xref="A2.p1.6.m6.3.3.1.1.1.1.1.1.2.1.cmml">[</mo><mrow id="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1" xref="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.2.cmml"><mo stretchy="false" id="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.2" xref="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.2.1.cmml">|</mo><mrow id="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1" xref="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.cmml"><msub id="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.2" xref="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.2.cmml"><mi id="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.2.2" xref="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.2.2.cmml">f</mi><mi id="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.2.3" xref="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.2.3.cmml">j</mi></msub><mo rspace="0.055em" id="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.1" xref="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.1.cmml">−</mo><mrow id="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.3" xref="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.3.cmml"><msubsup id="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.3.1" xref="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.3.1.cmml"><mo id="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.3.1.2.2" xref="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.3.1.2.2.cmml">∑</mo><mrow id="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.3.1.2.3" xref="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.3.1.2.3.cmml"><mi id="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.3.1.2.3.2" xref="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.3.1.2.3.2.cmml">i</mi><mo id="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.3.1.2.3.1" xref="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.3.1.2.3.1.cmml">=</mo><mn id="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.3.1.2.3.3" xref="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.3.1.2.3.3.cmml">1</mn></mrow><mrow id="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.3.1.3" xref="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.3.1.3.cmml"><mi id="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.3.1.3.2" xref="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.3.1.3.2.cmml">i</mi><mo id="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.3.1.3.1" xref="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.3.1.3.1.cmml">=</mo><mi id="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.3.1.3.3" xref="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.3.1.3.3.cmml">k</mi></mrow></msubsup><msub id="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.3.2" xref="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.3.2.cmml"><mi id="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.3.2.2" xref="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.3.2.2.cmml">n</mi><mi id="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.3.2.3" xref="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.3.2.3.cmml">i</mi></msub></mrow></mrow><mo stretchy="false" id="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.3" xref="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.2.1.cmml">|</mo></mrow><mo stretchy="false" id="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.3" xref="A2.p1.6.m6.3.3.1.1.1.1.1.1.2.1.cmml">]</mo></mrow><mo lspace="0.167em" rspace="0em" id="A2.p1.6.m6.3.3.1.1.1.1.1.2" xref="A2.p1.6.m6.3.3.1.1.1.1.1.2.cmml">​</mo><mrow id="A2.p1.6.m6.3.3.1.1.1.1.1.3" xref="A2.p1.6.m6.3.3.1.1.1.1.1.3.cmml"><mo rspace="0.167em" id="A2.p1.6.m6.3.3.1.1.1.1.1.3.1" xref="A2.p1.6.m6.3.3.1.1.1.1.1.3.1.cmml">∀</mo><mi id="A2.p1.6.m6.3.3.1.1.1.1.1.3.2" xref="A2.p1.6.m6.3.3.1.1.1.1.1.3.2.cmml">j</mi></mrow></mrow><mo id="A2.p1.6.m6.3.3.1.1.1.1.2" xref="A2.p1.6.m6.3.3.1.1.1.1.2.cmml">∈</mo><mrow id="A2.p1.6.m6.3.3.1.1.1.1.3.2" xref="A2.p1.6.m6.3.3.1.1.1.1.3.1.cmml"><mo stretchy="false" id="A2.p1.6.m6.3.3.1.1.1.1.3.2.1" xref="A2.p1.6.m6.3.3.1.1.1.1.3.1.cmml">[</mo><mn id="A2.p1.6.m6.1.1" xref="A2.p1.6.m6.1.1.cmml">1</mn><mo id="A2.p1.6.m6.3.3.1.1.1.1.3.2.2" xref="A2.p1.6.m6.3.3.1.1.1.1.3.1.cmml">,</mo><mi id="A2.p1.6.m6.2.2" xref="A2.p1.6.m6.2.2.cmml">m</mi><mo stretchy="false" id="A2.p1.6.m6.3.3.1.1.1.1.3.2.3" xref="A2.p1.6.m6.3.3.1.1.1.1.3.1.cmml">]</mo></mrow></mrow><mo stretchy="false" id="A2.p1.6.m6.3.3.1.1.1.3" xref="A2.p1.6.m6.3.3.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A2.p1.6.m6.3b"><apply id="A2.p1.6.m6.3.3.cmml" xref="A2.p1.6.m6.3.3"><eq id="A2.p1.6.m6.3.3.2.cmml" xref="A2.p1.6.m6.3.3.2"></eq><apply id="A2.p1.6.m6.3.3.3.cmml" xref="A2.p1.6.m6.3.3.3"><csymbol cd="ambiguous" id="A2.p1.6.m6.3.3.3.1.cmml" xref="A2.p1.6.m6.3.3.3">subscript</csymbol><ci id="A2.p1.6.m6.3.3.3.2.cmml" xref="A2.p1.6.m6.3.3.3.2">ℎ</ci><apply id="A2.p1.6.m6.3.3.3.3.cmml" xref="A2.p1.6.m6.3.3.3.3"><times id="A2.p1.6.m6.3.3.3.3.1.cmml" xref="A2.p1.6.m6.3.3.3.3.1"></times><ci id="A2.p1.6.m6.3.3.3.3.2.cmml" xref="A2.p1.6.m6.3.3.3.3.2">𝑚</ci><ci id="A2.p1.6.m6.3.3.3.3.3.cmml" xref="A2.p1.6.m6.3.3.3.3.3">𝑢</ci><ci id="A2.p1.6.m6.3.3.3.3.4.cmml" xref="A2.p1.6.m6.3.3.3.3.4">𝑙</ci><ci id="A2.p1.6.m6.3.3.3.3.5.cmml" xref="A2.p1.6.m6.3.3.3.3.5">𝑡</ci><ci id="A2.p1.6.m6.3.3.3.3.6.cmml" xref="A2.p1.6.m6.3.3.3.3.6">𝑖</ci><ci id="A2.p1.6.m6.3.3.3.3.7.cmml" xref="A2.p1.6.m6.3.3.3.3.7">𝑝</ci><ci id="A2.p1.6.m6.3.3.3.3.8.cmml" xref="A2.p1.6.m6.3.3.3.3.8">𝑙</ci><ci id="A2.p1.6.m6.3.3.3.3.9.cmml" xref="A2.p1.6.m6.3.3.3.3.9">𝑦</ci></apply></apply><apply id="A2.p1.6.m6.3.3.1.cmml" xref="A2.p1.6.m6.3.3.1"><times id="A2.p1.6.m6.3.3.1.2.cmml" xref="A2.p1.6.m6.3.3.1.2"></times><ci id="A2.p1.6.m6.3.3.1.3a.cmml" xref="A2.p1.6.m6.3.3.1.3"><mtext id="A2.p1.6.m6.3.3.1.3.cmml" xref="A2.p1.6.m6.3.3.1.3">min</mtext></ci><apply id="A2.p1.6.m6.3.3.1.1.1.1.cmml" xref="A2.p1.6.m6.3.3.1.1.1"><in id="A2.p1.6.m6.3.3.1.1.1.1.2.cmml" xref="A2.p1.6.m6.3.3.1.1.1.1.2"></in><apply id="A2.p1.6.m6.3.3.1.1.1.1.1.cmml" xref="A2.p1.6.m6.3.3.1.1.1.1.1"><times id="A2.p1.6.m6.3.3.1.1.1.1.1.2.cmml" xref="A2.p1.6.m6.3.3.1.1.1.1.1.2"></times><apply id="A2.p1.6.m6.3.3.1.1.1.1.1.1.2.cmml" xref="A2.p1.6.m6.3.3.1.1.1.1.1.1.1"><csymbol cd="latexml" id="A2.p1.6.m6.3.3.1.1.1.1.1.1.2.1.cmml" xref="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.2">delimited-[]</csymbol><apply id="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.2.cmml" xref="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1"><abs id="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.2.1.cmml" xref="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.2"></abs><apply id="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.cmml" xref="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1"><minus id="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.1"></minus><apply id="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.2.1.cmml" xref="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.2">subscript</csymbol><ci id="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.2.2.cmml" xref="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.2.2">𝑓</ci><ci id="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.2.3.cmml" xref="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.2.3">𝑗</ci></apply><apply id="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.3"><apply id="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.3.1.cmml" xref="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.3.1"><csymbol cd="ambiguous" id="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.3.1.1.cmml" xref="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.3.1">superscript</csymbol><apply id="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.3.1.2.cmml" xref="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.3.1"><csymbol cd="ambiguous" id="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.3.1.2.1.cmml" xref="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.3.1">subscript</csymbol><sum id="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.3.1.2.2.cmml" xref="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.3.1.2.2"></sum><apply id="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.3.1.2.3.cmml" xref="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.3.1.2.3"><eq id="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.3.1.2.3.1.cmml" xref="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.3.1.2.3.1"></eq><ci id="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.3.1.2.3.2.cmml" xref="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.3.1.2.3.2">𝑖</ci><cn type="integer" id="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.3.1.2.3.3.cmml" xref="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.3.1.2.3.3">1</cn></apply></apply><apply id="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.3.1.3.cmml" xref="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.3.1.3"><eq id="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.3.1.3.1.cmml" xref="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.3.1.3.1"></eq><ci id="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.3.1.3.2.cmml" xref="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.3.1.3.2">𝑖</ci><ci id="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.3.1.3.3.cmml" xref="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.3.1.3.3">𝑘</ci></apply></apply><apply id="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.3.2.cmml" xref="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.3.2"><csymbol cd="ambiguous" id="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.3.2.1.cmml" xref="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.3.2">subscript</csymbol><ci id="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.3.2.2.cmml" xref="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.3.2.2">𝑛</ci><ci id="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.3.2.3.cmml" xref="A2.p1.6.m6.3.3.1.1.1.1.1.1.1.1.1.1.3.2.3">𝑖</ci></apply></apply></apply></apply></apply><apply id="A2.p1.6.m6.3.3.1.1.1.1.1.3.cmml" xref="A2.p1.6.m6.3.3.1.1.1.1.1.3"><csymbol cd="latexml" id="A2.p1.6.m6.3.3.1.1.1.1.1.3.1.cmml" xref="A2.p1.6.m6.3.3.1.1.1.1.1.3.1">for-all</csymbol><ci id="A2.p1.6.m6.3.3.1.1.1.1.1.3.2.cmml" xref="A2.p1.6.m6.3.3.1.1.1.1.1.3.2">𝑗</ci></apply></apply><interval closure="closed" id="A2.p1.6.m6.3.3.1.1.1.1.3.1.cmml" xref="A2.p1.6.m6.3.3.1.1.1.1.3.2"><cn type="integer" id="A2.p1.6.m6.1.1.cmml" xref="A2.p1.6.m6.1.1">1</cn><ci id="A2.p1.6.m6.2.2.cmml" xref="A2.p1.6.m6.2.2">𝑚</ci></interval></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.p1.6.m6.3c">h_{multiply}=\text{min}([|f_{j}-\sum_{i=1}^{i=k}n_{i}|]\forall j\in[1,m])</annotation></semantics></math>. For DFS, the threshold for the heuristic is set to the value of the target <math id="A2.p1.7.m7.1" class="ltx_Math" alttext="h_{th}=T" display="inline"><semantics id="A2.p1.7.m7.1a"><mrow id="A2.p1.7.m7.1.1" xref="A2.p1.7.m7.1.1.cmml"><msub id="A2.p1.7.m7.1.1.2" xref="A2.p1.7.m7.1.1.2.cmml"><mi id="A2.p1.7.m7.1.1.2.2" xref="A2.p1.7.m7.1.1.2.2.cmml">h</mi><mrow id="A2.p1.7.m7.1.1.2.3" xref="A2.p1.7.m7.1.1.2.3.cmml"><mi id="A2.p1.7.m7.1.1.2.3.2" xref="A2.p1.7.m7.1.1.2.3.2.cmml">t</mi><mo lspace="0em" rspace="0em" id="A2.p1.7.m7.1.1.2.3.1" xref="A2.p1.7.m7.1.1.2.3.1.cmml">​</mo><mi id="A2.p1.7.m7.1.1.2.3.3" xref="A2.p1.7.m7.1.1.2.3.3.cmml">h</mi></mrow></msub><mo id="A2.p1.7.m7.1.1.1" xref="A2.p1.7.m7.1.1.1.cmml">=</mo><mi id="A2.p1.7.m7.1.1.3" xref="A2.p1.7.m7.1.1.3.cmml">T</mi></mrow><annotation-xml encoding="MathML-Content" id="A2.p1.7.m7.1b"><apply id="A2.p1.7.m7.1.1.cmml" xref="A2.p1.7.m7.1.1"><eq id="A2.p1.7.m7.1.1.1.cmml" xref="A2.p1.7.m7.1.1.1"></eq><apply id="A2.p1.7.m7.1.1.2.cmml" xref="A2.p1.7.m7.1.1.2"><csymbol cd="ambiguous" id="A2.p1.7.m7.1.1.2.1.cmml" xref="A2.p1.7.m7.1.1.2">subscript</csymbol><ci id="A2.p1.7.m7.1.1.2.2.cmml" xref="A2.p1.7.m7.1.1.2.2">ℎ</ci><apply id="A2.p1.7.m7.1.1.2.3.cmml" xref="A2.p1.7.m7.1.1.2.3"><times id="A2.p1.7.m7.1.1.2.3.1.cmml" xref="A2.p1.7.m7.1.1.2.3.1"></times><ci id="A2.p1.7.m7.1.1.2.3.2.cmml" xref="A2.p1.7.m7.1.1.2.3.2">𝑡</ci><ci id="A2.p1.7.m7.1.1.2.3.3.cmml" xref="A2.p1.7.m7.1.1.2.3.3">ℎ</ci></apply></apply><ci id="A2.p1.7.m7.1.1.3.cmml" xref="A2.p1.7.m7.1.1.3">𝑇</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.p1.7.m7.1c">h_{th}=T</annotation></semantics></math>. For BFS, a breadth limit <math id="A2.p1.8.m8.5" class="ltx_Math" alttext="b\in[1,2,3,4,5]" display="inline"><semantics id="A2.p1.8.m8.5a"><mrow id="A2.p1.8.m8.5.6" xref="A2.p1.8.m8.5.6.cmml"><mi id="A2.p1.8.m8.5.6.2" xref="A2.p1.8.m8.5.6.2.cmml">b</mi><mo id="A2.p1.8.m8.5.6.1" xref="A2.p1.8.m8.5.6.1.cmml">∈</mo><mrow id="A2.p1.8.m8.5.6.3.2" xref="A2.p1.8.m8.5.6.3.1.cmml"><mo stretchy="false" id="A2.p1.8.m8.5.6.3.2.1" xref="A2.p1.8.m8.5.6.3.1.cmml">[</mo><mn id="A2.p1.8.m8.1.1" xref="A2.p1.8.m8.1.1.cmml">1</mn><mo id="A2.p1.8.m8.5.6.3.2.2" xref="A2.p1.8.m8.5.6.3.1.cmml">,</mo><mn id="A2.p1.8.m8.2.2" xref="A2.p1.8.m8.2.2.cmml">2</mn><mo id="A2.p1.8.m8.5.6.3.2.3" xref="A2.p1.8.m8.5.6.3.1.cmml">,</mo><mn id="A2.p1.8.m8.3.3" xref="A2.p1.8.m8.3.3.cmml">3</mn><mo id="A2.p1.8.m8.5.6.3.2.4" xref="A2.p1.8.m8.5.6.3.1.cmml">,</mo><mn id="A2.p1.8.m8.4.4" xref="A2.p1.8.m8.4.4.cmml">4</mn><mo id="A2.p1.8.m8.5.6.3.2.5" xref="A2.p1.8.m8.5.6.3.1.cmml">,</mo><mn id="A2.p1.8.m8.5.5" xref="A2.p1.8.m8.5.5.cmml">5</mn><mo stretchy="false" id="A2.p1.8.m8.5.6.3.2.6" xref="A2.p1.8.m8.5.6.3.1.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A2.p1.8.m8.5b"><apply id="A2.p1.8.m8.5.6.cmml" xref="A2.p1.8.m8.5.6"><in id="A2.p1.8.m8.5.6.1.cmml" xref="A2.p1.8.m8.5.6.1"></in><ci id="A2.p1.8.m8.5.6.2.cmml" xref="A2.p1.8.m8.5.6.2">𝑏</ci><list id="A2.p1.8.m8.5.6.3.1.cmml" xref="A2.p1.8.m8.5.6.3.2"><cn type="integer" id="A2.p1.8.m8.1.1.cmml" xref="A2.p1.8.m8.1.1">1</cn><cn type="integer" id="A2.p1.8.m8.2.2.cmml" xref="A2.p1.8.m8.2.2">2</cn><cn type="integer" id="A2.p1.8.m8.3.3.cmml" xref="A2.p1.8.m8.3.3">3</cn><cn type="integer" id="A2.p1.8.m8.4.4.cmml" xref="A2.p1.8.m8.4.4">4</cn><cn type="integer" id="A2.p1.8.m8.5.5.cmml" xref="A2.p1.8.m8.5.5">5</cn></list></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.p1.8.m8.5c">b\in[1,2,3,4,5]</annotation></semantics></math> is used to keep only the top <math id="A2.p1.9.m9.1" class="ltx_Math" alttext="b" display="inline"><semantics id="A2.p1.9.m9.1a"><mi id="A2.p1.9.m9.1.1" xref="A2.p1.9.m9.1.1.cmml">b</mi><annotation-xml encoding="MathML-Content" id="A2.p1.9.m9.1b"><ci id="A2.p1.9.m9.1.1.cmml" xref="A2.p1.9.m9.1.1">𝑏</ci></annotation-xml><annotation encoding="application/x-tex" id="A2.p1.9.m9.1c">b</annotation></semantics></math> branches each state expansion. These hyperparameters provide a simple and intuitive way to design symbolic algorithms to generate the SoS dataset.</p>
</div>
<div id="A2.p2" class="ltx_para ltx_noindent">
<p id="A2.p2.1" class="ltx_p">The SoS dataset has 285,501 (57.1%) correct trajectories.</p>
</div>
<div id="A2.p3" class="ltx_para ltx_noindent">
<p id="A2.p3.1" class="ltx_p"><span id="A2.p3.1.1" class="ltx_text ltx_font_bold">Optimal Paths dataset.</span> The optimal trajectories only include the correct steps to reach the goal state. There are 500,000 correct solutions shown to the optimal paths model.</p>
</div>
<figure id="alg3" class="ltx_float ltx_float_algorithm ltx_framed ltx_framed_top">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_float"><span id="alg3.2.1.1" class="ltx_text ltx_font_bold">Algorithm 3</span> </span> SoS-DFS</figcaption>
<div id="alg3.3" class="ltx_listing ltx_listing">
<div id="alg3.l1" class="ltx_listingline">Problem <math id="alg3.l1.m1.1" class="ltx_Math" alttext="x" display="inline"><semantics id="alg3.l1.m1.1a"><mi id="alg3.l1.m1.1.1" xref="alg3.l1.m1.1.1.cmml">x</mi><annotation-xml encoding="MathML-Content" id="alg3.l1.m1.1b"><ci id="alg3.l1.m1.1.1.cmml" xref="alg3.l1.m1.1.1">𝑥</ci></annotation-xml><annotation encoding="application/x-tex" id="alg3.l1.m1.1c">x</annotation></semantics></math>, Heuristic <math id="alg3.l1.m2.1" class="ltx_Math" alttext="h" display="inline"><semantics id="alg3.l1.m2.1a"><mi id="alg3.l1.m2.1.1" xref="alg3.l1.m2.1.1.cmml">h</mi><annotation-xml encoding="MathML-Content" id="alg3.l1.m2.1b"><ci id="alg3.l1.m2.1.1.cmml" xref="alg3.l1.m2.1.1">ℎ</ci></annotation-xml><annotation encoding="application/x-tex" id="alg3.l1.m2.1c">h</annotation></semantics></math> State Expansion Function <math id="alg3.l1.m3.1" class="ltx_Math" alttext="SE" display="inline"><semantics id="alg3.l1.m3.1a"><mrow id="alg3.l1.m3.1.1" xref="alg3.l1.m3.1.1.cmml"><mi id="alg3.l1.m3.1.1.2" xref="alg3.l1.m3.1.1.2.cmml">S</mi><mo lspace="0em" rspace="0em" id="alg3.l1.m3.1.1.1" xref="alg3.l1.m3.1.1.1.cmml">​</mo><mi id="alg3.l1.m3.1.1.3" xref="alg3.l1.m3.1.1.3.cmml">E</mi></mrow><annotation-xml encoding="MathML-Content" id="alg3.l1.m3.1b"><apply id="alg3.l1.m3.1.1.cmml" xref="alg3.l1.m3.1.1"><times id="alg3.l1.m3.1.1.1.cmml" xref="alg3.l1.m3.1.1.1"></times><ci id="alg3.l1.m3.1.1.2.cmml" xref="alg3.l1.m3.1.1.2">𝑆</ci><ci id="alg3.l1.m3.1.1.3.cmml" xref="alg3.l1.m3.1.1.3">𝐸</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg3.l1.m3.1c">SE</annotation></semantics></math>, threshold <math id="alg3.l1.m4.1" class="ltx_Math" alttext="h_{th}" display="inline"><semantics id="alg3.l1.m4.1a"><msub id="alg3.l1.m4.1.1" xref="alg3.l1.m4.1.1.cmml"><mi id="alg3.l1.m4.1.1.2" xref="alg3.l1.m4.1.1.2.cmml">h</mi><mrow id="alg3.l1.m4.1.1.3" xref="alg3.l1.m4.1.1.3.cmml"><mi id="alg3.l1.m4.1.1.3.2" xref="alg3.l1.m4.1.1.3.2.cmml">t</mi><mo lspace="0em" rspace="0em" id="alg3.l1.m4.1.1.3.1" xref="alg3.l1.m4.1.1.3.1.cmml">​</mo><mi id="alg3.l1.m4.1.1.3.3" xref="alg3.l1.m4.1.1.3.3.cmml">h</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="alg3.l1.m4.1b"><apply id="alg3.l1.m4.1.1.cmml" xref="alg3.l1.m4.1.1"><csymbol cd="ambiguous" id="alg3.l1.m4.1.1.1.cmml" xref="alg3.l1.m4.1.1">subscript</csymbol><ci id="alg3.l1.m4.1.1.2.cmml" xref="alg3.l1.m4.1.1.2">ℎ</ci><apply id="alg3.l1.m4.1.1.3.cmml" xref="alg3.l1.m4.1.1.3"><times id="alg3.l1.m4.1.1.3.1.cmml" xref="alg3.l1.m4.1.1.3.1"></times><ci id="alg3.l1.m4.1.1.3.2.cmml" xref="alg3.l1.m4.1.1.3.2">𝑡</ci><ci id="alg3.l1.m4.1.1.3.3.cmml" xref="alg3.l1.m4.1.1.3.3">ℎ</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg3.l1.m4.1c">h_{th}</annotation></semantics></math>, Goal State <math id="alg3.l1.m5.1" class="ltx_Math" alttext="s_{g}" display="inline"><semantics id="alg3.l1.m5.1a"><msub id="alg3.l1.m5.1.1" xref="alg3.l1.m5.1.1.cmml"><mi id="alg3.l1.m5.1.1.2" xref="alg3.l1.m5.1.1.2.cmml">s</mi><mi id="alg3.l1.m5.1.1.3" xref="alg3.l1.m5.1.1.3.cmml">g</mi></msub><annotation-xml encoding="MathML-Content" id="alg3.l1.m5.1b"><apply id="alg3.l1.m5.1.1.cmml" xref="alg3.l1.m5.1.1"><csymbol cd="ambiguous" id="alg3.l1.m5.1.1.1.cmml" xref="alg3.l1.m5.1.1">subscript</csymbol><ci id="alg3.l1.m5.1.1.2.cmml" xref="alg3.l1.m5.1.1.2">𝑠</ci><ci id="alg3.l1.m5.1.1.3.cmml" xref="alg3.l1.m5.1.1.3">𝑔</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg3.l1.m5.1c">s_{g}</annotation></semantics></math>.

</div>
<div id="alg3.l2" class="ltx_listingline">SoS = “”

</div>
<div id="alg3.l3" class="ltx_listingline">
<math id="alg3.l3.m1.1" class="ltx_Math" alttext="s_{c}\leftarrow x" display="inline"><semantics id="alg3.l3.m1.1a"><mrow id="alg3.l3.m1.1.1" xref="alg3.l3.m1.1.1.cmml"><msub id="alg3.l3.m1.1.1.2" xref="alg3.l3.m1.1.1.2.cmml"><mi id="alg3.l3.m1.1.1.2.2" xref="alg3.l3.m1.1.1.2.2.cmml">s</mi><mi id="alg3.l3.m1.1.1.2.3" xref="alg3.l3.m1.1.1.2.3.cmml">c</mi></msub><mo stretchy="false" id="alg3.l3.m1.1.1.1" xref="alg3.l3.m1.1.1.1.cmml">←</mo><mi id="alg3.l3.m1.1.1.3" xref="alg3.l3.m1.1.1.3.cmml">x</mi></mrow><annotation-xml encoding="MathML-Content" id="alg3.l3.m1.1b"><apply id="alg3.l3.m1.1.1.cmml" xref="alg3.l3.m1.1.1"><ci id="alg3.l3.m1.1.1.1.cmml" xref="alg3.l3.m1.1.1.1">←</ci><apply id="alg3.l3.m1.1.1.2.cmml" xref="alg3.l3.m1.1.1.2"><csymbol cd="ambiguous" id="alg3.l3.m1.1.1.2.1.cmml" xref="alg3.l3.m1.1.1.2">subscript</csymbol><ci id="alg3.l3.m1.1.1.2.2.cmml" xref="alg3.l3.m1.1.1.2.2">𝑠</ci><ci id="alg3.l3.m1.1.1.2.3.cmml" xref="alg3.l3.m1.1.1.2.3">𝑐</ci></apply><ci id="alg3.l3.m1.1.1.3.cmml" xref="alg3.l3.m1.1.1.3">𝑥</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg3.l3.m1.1c">s_{c}\leftarrow x</annotation></semantics></math>

</div>
<div id="alg3.l4" class="ltx_listingline">
<span id="alg3.l4.1" class="ltx_text ltx_font_bold">if</span> <math id="alg3.l4.m1.1" class="ltx_Math" alttext="s_{c}" display="inline"><semantics id="alg3.l4.m1.1a"><msub id="alg3.l4.m1.1.1" xref="alg3.l4.m1.1.1.cmml"><mi id="alg3.l4.m1.1.1.2" xref="alg3.l4.m1.1.1.2.cmml">s</mi><mi id="alg3.l4.m1.1.1.3" xref="alg3.l4.m1.1.1.3.cmml">c</mi></msub><annotation-xml encoding="MathML-Content" id="alg3.l4.m1.1b"><apply id="alg3.l4.m1.1.1.cmml" xref="alg3.l4.m1.1.1"><csymbol cd="ambiguous" id="alg3.l4.m1.1.1.1.cmml" xref="alg3.l4.m1.1.1">subscript</csymbol><ci id="alg3.l4.m1.1.1.2.cmml" xref="alg3.l4.m1.1.1.2">𝑠</ci><ci id="alg3.l4.m1.1.1.3.cmml" xref="alg3.l4.m1.1.1.3">𝑐</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg3.l4.m1.1c">s_{c}</annotation></semantics></math> == <math id="alg3.l4.m2.1" class="ltx_Math" alttext="s_{g}" display="inline"><semantics id="alg3.l4.m2.1a"><msub id="alg3.l4.m2.1.1" xref="alg3.l4.m2.1.1.cmml"><mi id="alg3.l4.m2.1.1.2" xref="alg3.l4.m2.1.1.2.cmml">s</mi><mi id="alg3.l4.m2.1.1.3" xref="alg3.l4.m2.1.1.3.cmml">g</mi></msub><annotation-xml encoding="MathML-Content" id="alg3.l4.m2.1b"><apply id="alg3.l4.m2.1.1.cmml" xref="alg3.l4.m2.1.1"><csymbol cd="ambiguous" id="alg3.l4.m2.1.1.1.cmml" xref="alg3.l4.m2.1.1">subscript</csymbol><ci id="alg3.l4.m2.1.1.2.cmml" xref="alg3.l4.m2.1.1.2">𝑠</ci><ci id="alg3.l4.m2.1.1.3.cmml" xref="alg3.l4.m2.1.1.3">𝑔</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg3.l4.m2.1c">s_{g}</annotation></semantics></math> <span id="alg3.l4.2" class="ltx_text ltx_font_bold">then</span>

</div>
<div id="alg3.l5" class="ltx_listingline">     <span id="alg3.l5.1" class="ltx_text ltx_font_bold">return</span> SoS + “Goal Reached”

</div>
<div id="alg3.l6" class="ltx_listingline">
<span id="alg3.l6.1" class="ltx_text ltx_font_bold">end</span> <span id="alg3.l6.2" class="ltx_text ltx_font_bold">if</span>
</div>
<div id="alg3.l7" class="ltx_listingline">
<span id="alg3.l7.1" class="ltx_text ltx_font_bold">if</span> <math id="alg3.l7.m1.1" class="ltx_Math" alttext="G(s_{c})" display="inline"><semantics id="alg3.l7.m1.1a"><mrow id="alg3.l7.m1.1.1" xref="alg3.l7.m1.1.1.cmml"><mi id="alg3.l7.m1.1.1.3" xref="alg3.l7.m1.1.1.3.cmml">G</mi><mo lspace="0em" rspace="0em" id="alg3.l7.m1.1.1.2" xref="alg3.l7.m1.1.1.2.cmml">​</mo><mrow id="alg3.l7.m1.1.1.1.1" xref="alg3.l7.m1.1.1.1.1.1.cmml"><mo stretchy="false" id="alg3.l7.m1.1.1.1.1.2" xref="alg3.l7.m1.1.1.1.1.1.cmml">(</mo><msub id="alg3.l7.m1.1.1.1.1.1" xref="alg3.l7.m1.1.1.1.1.1.cmml"><mi id="alg3.l7.m1.1.1.1.1.1.2" xref="alg3.l7.m1.1.1.1.1.1.2.cmml">s</mi><mi id="alg3.l7.m1.1.1.1.1.1.3" xref="alg3.l7.m1.1.1.1.1.1.3.cmml">c</mi></msub><mo stretchy="false" id="alg3.l7.m1.1.1.1.1.3" xref="alg3.l7.m1.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg3.l7.m1.1b"><apply id="alg3.l7.m1.1.1.cmml" xref="alg3.l7.m1.1.1"><times id="alg3.l7.m1.1.1.2.cmml" xref="alg3.l7.m1.1.1.2"></times><ci id="alg3.l7.m1.1.1.3.cmml" xref="alg3.l7.m1.1.1.3">𝐺</ci><apply id="alg3.l7.m1.1.1.1.1.1.cmml" xref="alg3.l7.m1.1.1.1.1"><csymbol cd="ambiguous" id="alg3.l7.m1.1.1.1.1.1.1.cmml" xref="alg3.l7.m1.1.1.1.1">subscript</csymbol><ci id="alg3.l7.m1.1.1.1.1.1.2.cmml" xref="alg3.l7.m1.1.1.1.1.1.2">𝑠</ci><ci id="alg3.l7.m1.1.1.1.1.1.3.cmml" xref="alg3.l7.m1.1.1.1.1.1.3">𝑐</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg3.l7.m1.1c">G(s_{c})</annotation></semantics></math> is Null and <math id="alg3.l7.m2.1" class="ltx_Math" alttext="s_{c}" display="inline"><semantics id="alg3.l7.m2.1a"><msub id="alg3.l7.m2.1.1" xref="alg3.l7.m2.1.1.cmml"><mi id="alg3.l7.m2.1.1.2" xref="alg3.l7.m2.1.1.2.cmml">s</mi><mi id="alg3.l7.m2.1.1.3" xref="alg3.l7.m2.1.1.3.cmml">c</mi></msub><annotation-xml encoding="MathML-Content" id="alg3.l7.m2.1b"><apply id="alg3.l7.m2.1.1.cmml" xref="alg3.l7.m2.1.1"><csymbol cd="ambiguous" id="alg3.l7.m2.1.1.1.cmml" xref="alg3.l7.m2.1.1">subscript</csymbol><ci id="alg3.l7.m2.1.1.2.cmml" xref="alg3.l7.m2.1.1.2">𝑠</ci><ci id="alg3.l7.m2.1.1.3.cmml" xref="alg3.l7.m2.1.1.3">𝑐</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg3.l7.m2.1c">s_{c}</annotation></semantics></math> != <math id="alg3.l7.m3.1" class="ltx_Math" alttext="s_{g}" display="inline"><semantics id="alg3.l7.m3.1a"><msub id="alg3.l7.m3.1.1" xref="alg3.l7.m3.1.1.cmml"><mi id="alg3.l7.m3.1.1.2" xref="alg3.l7.m3.1.1.2.cmml">s</mi><mi id="alg3.l7.m3.1.1.3" xref="alg3.l7.m3.1.1.3.cmml">g</mi></msub><annotation-xml encoding="MathML-Content" id="alg3.l7.m3.1b"><apply id="alg3.l7.m3.1.1.cmml" xref="alg3.l7.m3.1.1"><csymbol cd="ambiguous" id="alg3.l7.m3.1.1.1.cmml" xref="alg3.l7.m3.1.1">subscript</csymbol><ci id="alg3.l7.m3.1.1.2.cmml" xref="alg3.l7.m3.1.1.2">𝑠</ci><ci id="alg3.l7.m3.1.1.3.cmml" xref="alg3.l7.m3.1.1.3">𝑔</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg3.l7.m3.1c">s_{g}</annotation></semantics></math> <span id="alg3.l7.2" class="ltx_text ltx_font_bold">then</span>

</div>
<div id="alg3.l8" class="ltx_listingline">     <span id="alg3.l8.1" class="ltx_text ltx_font_bold">return</span> SoS

</div>
<div id="alg3.l9" class="ltx_listingline">
<span id="alg3.l9.1" class="ltx_text ltx_font_bold">end</span> <span id="alg3.l9.2" class="ltx_text ltx_font_bold">if</span>
</div>
<div id="alg3.l10" class="ltx_listingline">
<math id="alg3.l10.m1.1" class="ltx_Math" alttext="S_{q}\leftarrow s\ \forall\ s\in SE(s_{c})" display="inline"><semantics id="alg3.l10.m1.1a"><mrow id="alg3.l10.m1.1.1" xref="alg3.l10.m1.1.1.cmml"><msub id="alg3.l10.m1.1.1.3" xref="alg3.l10.m1.1.1.3.cmml"><mi id="alg3.l10.m1.1.1.3.2" xref="alg3.l10.m1.1.1.3.2.cmml">S</mi><mi id="alg3.l10.m1.1.1.3.3" xref="alg3.l10.m1.1.1.3.3.cmml">q</mi></msub><mo stretchy="false" id="alg3.l10.m1.1.1.4" xref="alg3.l10.m1.1.1.4.cmml">←</mo><mrow id="alg3.l10.m1.1.1.5" xref="alg3.l10.m1.1.1.5.cmml"><mi id="alg3.l10.m1.1.1.5.2" xref="alg3.l10.m1.1.1.5.2.cmml">s</mi><mo lspace="0.667em" rspace="0em" id="alg3.l10.m1.1.1.5.1" xref="alg3.l10.m1.1.1.5.1.cmml">​</mo><mrow id="alg3.l10.m1.1.1.5.3" xref="alg3.l10.m1.1.1.5.3.cmml"><mo rspace="0.667em" id="alg3.l10.m1.1.1.5.3.1" xref="alg3.l10.m1.1.1.5.3.1.cmml">∀</mo><mi id="alg3.l10.m1.1.1.5.3.2" xref="alg3.l10.m1.1.1.5.3.2.cmml">s</mi></mrow></mrow><mo id="alg3.l10.m1.1.1.6" xref="alg3.l10.m1.1.1.6.cmml">∈</mo><mrow id="alg3.l10.m1.1.1.1" xref="alg3.l10.m1.1.1.1.cmml"><mi id="alg3.l10.m1.1.1.1.3" xref="alg3.l10.m1.1.1.1.3.cmml">S</mi><mo lspace="0em" rspace="0em" id="alg3.l10.m1.1.1.1.2" xref="alg3.l10.m1.1.1.1.2.cmml">​</mo><mi id="alg3.l10.m1.1.1.1.4" xref="alg3.l10.m1.1.1.1.4.cmml">E</mi><mo lspace="0em" rspace="0em" id="alg3.l10.m1.1.1.1.2a" xref="alg3.l10.m1.1.1.1.2.cmml">​</mo><mrow id="alg3.l10.m1.1.1.1.1.1" xref="alg3.l10.m1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="alg3.l10.m1.1.1.1.1.1.2" xref="alg3.l10.m1.1.1.1.1.1.1.cmml">(</mo><msub id="alg3.l10.m1.1.1.1.1.1.1" xref="alg3.l10.m1.1.1.1.1.1.1.cmml"><mi id="alg3.l10.m1.1.1.1.1.1.1.2" xref="alg3.l10.m1.1.1.1.1.1.1.2.cmml">s</mi><mi id="alg3.l10.m1.1.1.1.1.1.1.3" xref="alg3.l10.m1.1.1.1.1.1.1.3.cmml">c</mi></msub><mo stretchy="false" id="alg3.l10.m1.1.1.1.1.1.3" xref="alg3.l10.m1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg3.l10.m1.1b"><apply id="alg3.l10.m1.1.1.cmml" xref="alg3.l10.m1.1.1"><and id="alg3.l10.m1.1.1a.cmml" xref="alg3.l10.m1.1.1"></and><apply id="alg3.l10.m1.1.1b.cmml" xref="alg3.l10.m1.1.1"><ci id="alg3.l10.m1.1.1.4.cmml" xref="alg3.l10.m1.1.1.4">←</ci><apply id="alg3.l10.m1.1.1.3.cmml" xref="alg3.l10.m1.1.1.3"><csymbol cd="ambiguous" id="alg3.l10.m1.1.1.3.1.cmml" xref="alg3.l10.m1.1.1.3">subscript</csymbol><ci id="alg3.l10.m1.1.1.3.2.cmml" xref="alg3.l10.m1.1.1.3.2">𝑆</ci><ci id="alg3.l10.m1.1.1.3.3.cmml" xref="alg3.l10.m1.1.1.3.3">𝑞</ci></apply><apply id="alg3.l10.m1.1.1.5.cmml" xref="alg3.l10.m1.1.1.5"><times id="alg3.l10.m1.1.1.5.1.cmml" xref="alg3.l10.m1.1.1.5.1"></times><ci id="alg3.l10.m1.1.1.5.2.cmml" xref="alg3.l10.m1.1.1.5.2">𝑠</ci><apply id="alg3.l10.m1.1.1.5.3.cmml" xref="alg3.l10.m1.1.1.5.3"><csymbol cd="latexml" id="alg3.l10.m1.1.1.5.3.1.cmml" xref="alg3.l10.m1.1.1.5.3.1">for-all</csymbol><ci id="alg3.l10.m1.1.1.5.3.2.cmml" xref="alg3.l10.m1.1.1.5.3.2">𝑠</ci></apply></apply></apply><apply id="alg3.l10.m1.1.1c.cmml" xref="alg3.l10.m1.1.1"><in id="alg3.l10.m1.1.1.6.cmml" xref="alg3.l10.m1.1.1.6"></in><share href="#alg3.l10.m1.1.1.5.cmml" id="alg3.l10.m1.1.1d.cmml" xref="alg3.l10.m1.1.1"></share><apply id="alg3.l10.m1.1.1.1.cmml" xref="alg3.l10.m1.1.1.1"><times id="alg3.l10.m1.1.1.1.2.cmml" xref="alg3.l10.m1.1.1.1.2"></times><ci id="alg3.l10.m1.1.1.1.3.cmml" xref="alg3.l10.m1.1.1.1.3">𝑆</ci><ci id="alg3.l10.m1.1.1.1.4.cmml" xref="alg3.l10.m1.1.1.1.4">𝐸</ci><apply id="alg3.l10.m1.1.1.1.1.1.1.cmml" xref="alg3.l10.m1.1.1.1.1.1"><csymbol cd="ambiguous" id="alg3.l10.m1.1.1.1.1.1.1.1.cmml" xref="alg3.l10.m1.1.1.1.1.1">subscript</csymbol><ci id="alg3.l10.m1.1.1.1.1.1.1.2.cmml" xref="alg3.l10.m1.1.1.1.1.1.1.2">𝑠</ci><ci id="alg3.l10.m1.1.1.1.1.1.1.3.cmml" xref="alg3.l10.m1.1.1.1.1.1.1.3">𝑐</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg3.l10.m1.1c">S_{q}\leftarrow s\ \forall\ s\in SE(s_{c})</annotation></semantics></math> s.t. <math id="alg3.l10.m2.1" class="ltx_Math" alttext="h(s)&gt;h_{th}" display="inline"><semantics id="alg3.l10.m2.1a"><mrow id="alg3.l10.m2.1.2" xref="alg3.l10.m2.1.2.cmml"><mrow id="alg3.l10.m2.1.2.2" xref="alg3.l10.m2.1.2.2.cmml"><mi id="alg3.l10.m2.1.2.2.2" xref="alg3.l10.m2.1.2.2.2.cmml">h</mi><mo lspace="0em" rspace="0em" id="alg3.l10.m2.1.2.2.1" xref="alg3.l10.m2.1.2.2.1.cmml">​</mo><mrow id="alg3.l10.m2.1.2.2.3.2" xref="alg3.l10.m2.1.2.2.cmml"><mo stretchy="false" id="alg3.l10.m2.1.2.2.3.2.1" xref="alg3.l10.m2.1.2.2.cmml">(</mo><mi id="alg3.l10.m2.1.1" xref="alg3.l10.m2.1.1.cmml">s</mi><mo stretchy="false" id="alg3.l10.m2.1.2.2.3.2.2" xref="alg3.l10.m2.1.2.2.cmml">)</mo></mrow></mrow><mo id="alg3.l10.m2.1.2.1" xref="alg3.l10.m2.1.2.1.cmml">&gt;</mo><msub id="alg3.l10.m2.1.2.3" xref="alg3.l10.m2.1.2.3.cmml"><mi id="alg3.l10.m2.1.2.3.2" xref="alg3.l10.m2.1.2.3.2.cmml">h</mi><mrow id="alg3.l10.m2.1.2.3.3" xref="alg3.l10.m2.1.2.3.3.cmml"><mi id="alg3.l10.m2.1.2.3.3.2" xref="alg3.l10.m2.1.2.3.3.2.cmml">t</mi><mo lspace="0em" rspace="0em" id="alg3.l10.m2.1.2.3.3.1" xref="alg3.l10.m2.1.2.3.3.1.cmml">​</mo><mi id="alg3.l10.m2.1.2.3.3.3" xref="alg3.l10.m2.1.2.3.3.3.cmml">h</mi></mrow></msub></mrow><annotation-xml encoding="MathML-Content" id="alg3.l10.m2.1b"><apply id="alg3.l10.m2.1.2.cmml" xref="alg3.l10.m2.1.2"><gt id="alg3.l10.m2.1.2.1.cmml" xref="alg3.l10.m2.1.2.1"></gt><apply id="alg3.l10.m2.1.2.2.cmml" xref="alg3.l10.m2.1.2.2"><times id="alg3.l10.m2.1.2.2.1.cmml" xref="alg3.l10.m2.1.2.2.1"></times><ci id="alg3.l10.m2.1.2.2.2.cmml" xref="alg3.l10.m2.1.2.2.2">ℎ</ci><ci id="alg3.l10.m2.1.1.cmml" xref="alg3.l10.m2.1.1">𝑠</ci></apply><apply id="alg3.l10.m2.1.2.3.cmml" xref="alg3.l10.m2.1.2.3"><csymbol cd="ambiguous" id="alg3.l10.m2.1.2.3.1.cmml" xref="alg3.l10.m2.1.2.3">subscript</csymbol><ci id="alg3.l10.m2.1.2.3.2.cmml" xref="alg3.l10.m2.1.2.3.2">ℎ</ci><apply id="alg3.l10.m2.1.2.3.3.cmml" xref="alg3.l10.m2.1.2.3.3"><times id="alg3.l10.m2.1.2.3.3.1.cmml" xref="alg3.l10.m2.1.2.3.3.1"></times><ci id="alg3.l10.m2.1.2.3.3.2.cmml" xref="alg3.l10.m2.1.2.3.3.2">𝑡</ci><ci id="alg3.l10.m2.1.2.3.3.3.cmml" xref="alg3.l10.m2.1.2.3.3.3">ℎ</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg3.l10.m2.1c">h(s)&gt;h_{th}</annotation></semantics></math>

</div>
<div id="alg3.l11" class="ltx_listingline">
<span id="alg3.l11.1" class="ltx_text ltx_font_bold">for</span> <math id="alg3.l11.m1.1" class="ltx_Math" alttext="s\in S" display="inline"><semantics id="alg3.l11.m1.1a"><mrow id="alg3.l11.m1.1.1" xref="alg3.l11.m1.1.1.cmml"><mi id="alg3.l11.m1.1.1.2" xref="alg3.l11.m1.1.1.2.cmml">s</mi><mo id="alg3.l11.m1.1.1.1" xref="alg3.l11.m1.1.1.1.cmml">∈</mo><mi id="alg3.l11.m1.1.1.3" xref="alg3.l11.m1.1.1.3.cmml">S</mi></mrow><annotation-xml encoding="MathML-Content" id="alg3.l11.m1.1b"><apply id="alg3.l11.m1.1.1.cmml" xref="alg3.l11.m1.1.1"><in id="alg3.l11.m1.1.1.1.cmml" xref="alg3.l11.m1.1.1.1"></in><ci id="alg3.l11.m1.1.1.2.cmml" xref="alg3.l11.m1.1.1.2">𝑠</ci><ci id="alg3.l11.m1.1.1.3.cmml" xref="alg3.l11.m1.1.1.3">𝑆</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg3.l11.m1.1c">s\in S</annotation></semantics></math> <span id="alg3.l11.2" class="ltx_text ltx_font_bold">do</span>

</div>
<div id="alg3.l12" class="ltx_listingline">     <math id="alg3.l12.m1.1" class="ltx_Math" alttext="\text{SoS += }s" display="inline"><semantics id="alg3.l12.m1.1a"><mrow id="alg3.l12.m1.1.1" xref="alg3.l12.m1.1.1.cmml"><mtext id="alg3.l12.m1.1.1.2" xref="alg3.l12.m1.1.1.2a.cmml">SoS += </mtext><mo lspace="0em" rspace="0em" id="alg3.l12.m1.1.1.1" xref="alg3.l12.m1.1.1.1.cmml">​</mo><mi id="alg3.l12.m1.1.1.3" xref="alg3.l12.m1.1.1.3.cmml">s</mi></mrow><annotation-xml encoding="MathML-Content" id="alg3.l12.m1.1b"><apply id="alg3.l12.m1.1.1.cmml" xref="alg3.l12.m1.1.1"><times id="alg3.l12.m1.1.1.1.cmml" xref="alg3.l12.m1.1.1.1"></times><ci id="alg3.l12.m1.1.1.2a.cmml" xref="alg3.l12.m1.1.1.2"><mtext id="alg3.l12.m1.1.1.2.cmml" xref="alg3.l12.m1.1.1.2">SoS += </mtext></ci><ci id="alg3.l12.m1.1.1.3.cmml" xref="alg3.l12.m1.1.1.3">𝑠</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg3.l12.m1.1c">\text{SoS += }s</annotation></semantics></math>

</div>
<div id="alg3.l13" class="ltx_listingline">     SoS += DFS(<math id="alg3.l13.m1.1" class="ltx_Math" alttext="s" display="inline"><semantics id="alg3.l13.m1.1a"><mi id="alg3.l13.m1.1.1" xref="alg3.l13.m1.1.1.cmml">s</mi><annotation-xml encoding="MathML-Content" id="alg3.l13.m1.1b"><ci id="alg3.l13.m1.1.1.cmml" xref="alg3.l13.m1.1.1">𝑠</ci></annotation-xml><annotation encoding="application/x-tex" id="alg3.l13.m1.1c">s</annotation></semantics></math>)

</div>
<div id="alg3.l14" class="ltx_listingline">
<span id="alg3.l14.1" class="ltx_text ltx_font_bold">end</span> <span id="alg3.l14.2" class="ltx_text ltx_font_bold">for</span>
</div>
<div id="alg3.l15" class="ltx_listingline">
<span id="alg3.l15.1" class="ltx_text ltx_font_bold">return SoS</span>

</div>
</div>
</figure>
<figure id="alg4" class="ltx_float ltx_float_algorithm ltx_framed ltx_framed_top">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_float"><span id="alg4.2.1.1" class="ltx_text ltx_font_bold">Algorithm 4</span> </span> SoS-BFS</figcaption>
<div id="alg4.3" class="ltx_listing ltx_listing">
<div id="alg4.l1" class="ltx_listingline">Problem <math id="alg4.l1.m1.1" class="ltx_Math" alttext="x" display="inline"><semantics id="alg4.l1.m1.1a"><mi id="alg4.l1.m1.1.1" xref="alg4.l1.m1.1.1.cmml">x</mi><annotation-xml encoding="MathML-Content" id="alg4.l1.m1.1b"><ci id="alg4.l1.m1.1.1.cmml" xref="alg4.l1.m1.1.1">𝑥</ci></annotation-xml><annotation encoding="application/x-tex" id="alg4.l1.m1.1c">x</annotation></semantics></math>, Heuristic <math id="alg4.l1.m2.1" class="ltx_Math" alttext="h" display="inline"><semantics id="alg4.l1.m2.1a"><mi id="alg4.l1.m2.1.1" xref="alg4.l1.m2.1.1.cmml">h</mi><annotation-xml encoding="MathML-Content" id="alg4.l1.m2.1b"><ci id="alg4.l1.m2.1.1.cmml" xref="alg4.l1.m2.1.1">ℎ</ci></annotation-xml><annotation encoding="application/x-tex" id="alg4.l1.m2.1c">h</annotation></semantics></math>, breadth limit <math id="alg4.l1.m3.1" class="ltx_Math" alttext="b" display="inline"><semantics id="alg4.l1.m3.1a"><mi id="alg4.l1.m3.1.1" xref="alg4.l1.m3.1.1.cmml">b</mi><annotation-xml encoding="MathML-Content" id="alg4.l1.m3.1b"><ci id="alg4.l1.m3.1.1.cmml" xref="alg4.l1.m3.1.1">𝑏</ci></annotation-xml><annotation encoding="application/x-tex" id="alg4.l1.m3.1c">b</annotation></semantics></math>, Next State Expander <math id="alg4.l1.m4.1" class="ltx_Math" alttext="SE" display="inline"><semantics id="alg4.l1.m4.1a"><mrow id="alg4.l1.m4.1.1" xref="alg4.l1.m4.1.1.cmml"><mi id="alg4.l1.m4.1.1.2" xref="alg4.l1.m4.1.1.2.cmml">S</mi><mo lspace="0em" rspace="0em" id="alg4.l1.m4.1.1.1" xref="alg4.l1.m4.1.1.1.cmml">​</mo><mi id="alg4.l1.m4.1.1.3" xref="alg4.l1.m4.1.1.3.cmml">E</mi></mrow><annotation-xml encoding="MathML-Content" id="alg4.l1.m4.1b"><apply id="alg4.l1.m4.1.1.cmml" xref="alg4.l1.m4.1.1"><times id="alg4.l1.m4.1.1.1.cmml" xref="alg4.l1.m4.1.1.1"></times><ci id="alg4.l1.m4.1.1.2.cmml" xref="alg4.l1.m4.1.1.2">𝑆</ci><ci id="alg4.l1.m4.1.1.3.cmml" xref="alg4.l1.m4.1.1.3">𝐸</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg4.l1.m4.1c">SE</annotation></semantics></math>, Goal State <math id="alg4.l1.m5.1" class="ltx_Math" alttext="s_{g}" display="inline"><semantics id="alg4.l1.m5.1a"><msub id="alg4.l1.m5.1.1" xref="alg4.l1.m5.1.1.cmml"><mi id="alg4.l1.m5.1.1.2" xref="alg4.l1.m5.1.1.2.cmml">s</mi><mi id="alg4.l1.m5.1.1.3" xref="alg4.l1.m5.1.1.3.cmml">g</mi></msub><annotation-xml encoding="MathML-Content" id="alg4.l1.m5.1b"><apply id="alg4.l1.m5.1.1.cmml" xref="alg4.l1.m5.1.1"><csymbol cd="ambiguous" id="alg4.l1.m5.1.1.1.cmml" xref="alg4.l1.m5.1.1">subscript</csymbol><ci id="alg4.l1.m5.1.1.2.cmml" xref="alg4.l1.m5.1.1.2">𝑠</ci><ci id="alg4.l1.m5.1.1.3.cmml" xref="alg4.l1.m5.1.1.3">𝑔</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg4.l1.m5.1c">s_{g}</annotation></semantics></math>

</div>
<div id="alg4.l2" class="ltx_listingline">
<math id="alg4.l2.m1.2" class="ltx_Math" alttext="S_{q,0}\leftarrow{x}" display="inline"><semantics id="alg4.l2.m1.2a"><mrow id="alg4.l2.m1.2.3" xref="alg4.l2.m1.2.3.cmml"><msub id="alg4.l2.m1.2.3.2" xref="alg4.l2.m1.2.3.2.cmml"><mi id="alg4.l2.m1.2.3.2.2" xref="alg4.l2.m1.2.3.2.2.cmml">S</mi><mrow id="alg4.l2.m1.2.2.2.4" xref="alg4.l2.m1.2.2.2.3.cmml"><mi id="alg4.l2.m1.1.1.1.1" xref="alg4.l2.m1.1.1.1.1.cmml">q</mi><mo id="alg4.l2.m1.2.2.2.4.1" xref="alg4.l2.m1.2.2.2.3.cmml">,</mo><mn id="alg4.l2.m1.2.2.2.2" xref="alg4.l2.m1.2.2.2.2.cmml">0</mn></mrow></msub><mo stretchy="false" id="alg4.l2.m1.2.3.1" xref="alg4.l2.m1.2.3.1.cmml">←</mo><mi id="alg4.l2.m1.2.3.3" xref="alg4.l2.m1.2.3.3.cmml">x</mi></mrow><annotation-xml encoding="MathML-Content" id="alg4.l2.m1.2b"><apply id="alg4.l2.m1.2.3.cmml" xref="alg4.l2.m1.2.3"><ci id="alg4.l2.m1.2.3.1.cmml" xref="alg4.l2.m1.2.3.1">←</ci><apply id="alg4.l2.m1.2.3.2.cmml" xref="alg4.l2.m1.2.3.2"><csymbol cd="ambiguous" id="alg4.l2.m1.2.3.2.1.cmml" xref="alg4.l2.m1.2.3.2">subscript</csymbol><ci id="alg4.l2.m1.2.3.2.2.cmml" xref="alg4.l2.m1.2.3.2.2">𝑆</ci><list id="alg4.l2.m1.2.2.2.3.cmml" xref="alg4.l2.m1.2.2.2.4"><ci id="alg4.l2.m1.1.1.1.1.cmml" xref="alg4.l2.m1.1.1.1.1">𝑞</ci><cn type="integer" id="alg4.l2.m1.2.2.2.2.cmml" xref="alg4.l2.m1.2.2.2.2">0</cn></list></apply><ci id="alg4.l2.m1.2.3.3.cmml" xref="alg4.l2.m1.2.3.3">𝑥</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg4.l2.m1.2c">S_{q,0}\leftarrow{x}</annotation></semantics></math>

</div>
<div id="alg4.l3" class="ltx_listingline">SoS = “”

</div>
<div id="alg4.l4" class="ltx_listingline">
<span id="alg4.l4.1" class="ltx_text ltx_font_bold">for</span> <math id="alg4.l4.m1.3" class="ltx_Math" alttext="i=1,\dots,T" display="inline"><semantics id="alg4.l4.m1.3a"><mrow id="alg4.l4.m1.3.4" xref="alg4.l4.m1.3.4.cmml"><mi id="alg4.l4.m1.3.4.2" xref="alg4.l4.m1.3.4.2.cmml">i</mi><mo id="alg4.l4.m1.3.4.1" xref="alg4.l4.m1.3.4.1.cmml">=</mo><mrow id="alg4.l4.m1.3.4.3.2" xref="alg4.l4.m1.3.4.3.1.cmml"><mn id="alg4.l4.m1.1.1" xref="alg4.l4.m1.1.1.cmml">1</mn><mo id="alg4.l4.m1.3.4.3.2.1" xref="alg4.l4.m1.3.4.3.1.cmml">,</mo><mi mathvariant="normal" id="alg4.l4.m1.2.2" xref="alg4.l4.m1.2.2.cmml">…</mi><mo id="alg4.l4.m1.3.4.3.2.2" xref="alg4.l4.m1.3.4.3.1.cmml">,</mo><mi id="alg4.l4.m1.3.3" xref="alg4.l4.m1.3.3.cmml">T</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg4.l4.m1.3b"><apply id="alg4.l4.m1.3.4.cmml" xref="alg4.l4.m1.3.4"><eq id="alg4.l4.m1.3.4.1.cmml" xref="alg4.l4.m1.3.4.1"></eq><ci id="alg4.l4.m1.3.4.2.cmml" xref="alg4.l4.m1.3.4.2">𝑖</ci><list id="alg4.l4.m1.3.4.3.1.cmml" xref="alg4.l4.m1.3.4.3.2"><cn type="integer" id="alg4.l4.m1.1.1.cmml" xref="alg4.l4.m1.1.1">1</cn><ci id="alg4.l4.m1.2.2.cmml" xref="alg4.l4.m1.2.2">…</ci><ci id="alg4.l4.m1.3.3.cmml" xref="alg4.l4.m1.3.3">𝑇</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="alg4.l4.m1.3c">i=1,\dots,T</annotation></semantics></math> <span id="alg4.l4.2" class="ltx_text ltx_font_bold">do</span> <span id="alg4.l4.3" class="ltx_text ltx_font_bold">do</span>

</div>
<div id="alg4.l5" class="ltx_listingline">     <math id="alg4.l5.m1.2" class="ltx_Math" alttext="s_{c}\leftarrow s_{i-1}\text{ where }s_{i-1}\in S_{q,i-1}" display="inline"><semantics id="alg4.l5.m1.2a"><mrow id="alg4.l5.m1.2.3" xref="alg4.l5.m1.2.3.cmml"><msub id="alg4.l5.m1.2.3.2" xref="alg4.l5.m1.2.3.2.cmml"><mi id="alg4.l5.m1.2.3.2.2" xref="alg4.l5.m1.2.3.2.2.cmml">s</mi><mi id="alg4.l5.m1.2.3.2.3" xref="alg4.l5.m1.2.3.2.3.cmml">c</mi></msub><mo stretchy="false" id="alg4.l5.m1.2.3.3" xref="alg4.l5.m1.2.3.3.cmml">←</mo><mrow id="alg4.l5.m1.2.3.4" xref="alg4.l5.m1.2.3.4.cmml"><msub id="alg4.l5.m1.2.3.4.2" xref="alg4.l5.m1.2.3.4.2.cmml"><mi id="alg4.l5.m1.2.3.4.2.2" xref="alg4.l5.m1.2.3.4.2.2.cmml">s</mi><mrow id="alg4.l5.m1.2.3.4.2.3" xref="alg4.l5.m1.2.3.4.2.3.cmml"><mi id="alg4.l5.m1.2.3.4.2.3.2" xref="alg4.l5.m1.2.3.4.2.3.2.cmml">i</mi><mo id="alg4.l5.m1.2.3.4.2.3.1" xref="alg4.l5.m1.2.3.4.2.3.1.cmml">−</mo><mn id="alg4.l5.m1.2.3.4.2.3.3" xref="alg4.l5.m1.2.3.4.2.3.3.cmml">1</mn></mrow></msub><mo lspace="0em" rspace="0em" id="alg4.l5.m1.2.3.4.1" xref="alg4.l5.m1.2.3.4.1.cmml">​</mo><mtext id="alg4.l5.m1.2.3.4.3" xref="alg4.l5.m1.2.3.4.3a.cmml"> where </mtext><mo lspace="0em" rspace="0em" id="alg4.l5.m1.2.3.4.1a" xref="alg4.l5.m1.2.3.4.1.cmml">​</mo><msub id="alg4.l5.m1.2.3.4.4" xref="alg4.l5.m1.2.3.4.4.cmml"><mi id="alg4.l5.m1.2.3.4.4.2" xref="alg4.l5.m1.2.3.4.4.2.cmml">s</mi><mrow id="alg4.l5.m1.2.3.4.4.3" xref="alg4.l5.m1.2.3.4.4.3.cmml"><mi id="alg4.l5.m1.2.3.4.4.3.2" xref="alg4.l5.m1.2.3.4.4.3.2.cmml">i</mi><mo id="alg4.l5.m1.2.3.4.4.3.1" xref="alg4.l5.m1.2.3.4.4.3.1.cmml">−</mo><mn id="alg4.l5.m1.2.3.4.4.3.3" xref="alg4.l5.m1.2.3.4.4.3.3.cmml">1</mn></mrow></msub></mrow><mo id="alg4.l5.m1.2.3.5" xref="alg4.l5.m1.2.3.5.cmml">∈</mo><msub id="alg4.l5.m1.2.3.6" xref="alg4.l5.m1.2.3.6.cmml"><mi id="alg4.l5.m1.2.3.6.2" xref="alg4.l5.m1.2.3.6.2.cmml">S</mi><mrow id="alg4.l5.m1.2.2.2.2" xref="alg4.l5.m1.2.2.2.3.cmml"><mi id="alg4.l5.m1.1.1.1.1" xref="alg4.l5.m1.1.1.1.1.cmml">q</mi><mo id="alg4.l5.m1.2.2.2.2.2" xref="alg4.l5.m1.2.2.2.3.cmml">,</mo><mrow id="alg4.l5.m1.2.2.2.2.1" xref="alg4.l5.m1.2.2.2.2.1.cmml"><mi id="alg4.l5.m1.2.2.2.2.1.2" xref="alg4.l5.m1.2.2.2.2.1.2.cmml">i</mi><mo id="alg4.l5.m1.2.2.2.2.1.1" xref="alg4.l5.m1.2.2.2.2.1.1.cmml">−</mo><mn id="alg4.l5.m1.2.2.2.2.1.3" xref="alg4.l5.m1.2.2.2.2.1.3.cmml">1</mn></mrow></mrow></msub></mrow><annotation-xml encoding="MathML-Content" id="alg4.l5.m1.2b"><apply id="alg4.l5.m1.2.3.cmml" xref="alg4.l5.m1.2.3"><and id="alg4.l5.m1.2.3a.cmml" xref="alg4.l5.m1.2.3"></and><apply id="alg4.l5.m1.2.3b.cmml" xref="alg4.l5.m1.2.3"><ci id="alg4.l5.m1.2.3.3.cmml" xref="alg4.l5.m1.2.3.3">←</ci><apply id="alg4.l5.m1.2.3.2.cmml" xref="alg4.l5.m1.2.3.2"><csymbol cd="ambiguous" id="alg4.l5.m1.2.3.2.1.cmml" xref="alg4.l5.m1.2.3.2">subscript</csymbol><ci id="alg4.l5.m1.2.3.2.2.cmml" xref="alg4.l5.m1.2.3.2.2">𝑠</ci><ci id="alg4.l5.m1.2.3.2.3.cmml" xref="alg4.l5.m1.2.3.2.3">𝑐</ci></apply><apply id="alg4.l5.m1.2.3.4.cmml" xref="alg4.l5.m1.2.3.4"><times id="alg4.l5.m1.2.3.4.1.cmml" xref="alg4.l5.m1.2.3.4.1"></times><apply id="alg4.l5.m1.2.3.4.2.cmml" xref="alg4.l5.m1.2.3.4.2"><csymbol cd="ambiguous" id="alg4.l5.m1.2.3.4.2.1.cmml" xref="alg4.l5.m1.2.3.4.2">subscript</csymbol><ci id="alg4.l5.m1.2.3.4.2.2.cmml" xref="alg4.l5.m1.2.3.4.2.2">𝑠</ci><apply id="alg4.l5.m1.2.3.4.2.3.cmml" xref="alg4.l5.m1.2.3.4.2.3"><minus id="alg4.l5.m1.2.3.4.2.3.1.cmml" xref="alg4.l5.m1.2.3.4.2.3.1"></minus><ci id="alg4.l5.m1.2.3.4.2.3.2.cmml" xref="alg4.l5.m1.2.3.4.2.3.2">𝑖</ci><cn type="integer" id="alg4.l5.m1.2.3.4.2.3.3.cmml" xref="alg4.l5.m1.2.3.4.2.3.3">1</cn></apply></apply><ci id="alg4.l5.m1.2.3.4.3a.cmml" xref="alg4.l5.m1.2.3.4.3"><mtext id="alg4.l5.m1.2.3.4.3.cmml" xref="alg4.l5.m1.2.3.4.3"> where </mtext></ci><apply id="alg4.l5.m1.2.3.4.4.cmml" xref="alg4.l5.m1.2.3.4.4"><csymbol cd="ambiguous" id="alg4.l5.m1.2.3.4.4.1.cmml" xref="alg4.l5.m1.2.3.4.4">subscript</csymbol><ci id="alg4.l5.m1.2.3.4.4.2.cmml" xref="alg4.l5.m1.2.3.4.4.2">𝑠</ci><apply id="alg4.l5.m1.2.3.4.4.3.cmml" xref="alg4.l5.m1.2.3.4.4.3"><minus id="alg4.l5.m1.2.3.4.4.3.1.cmml" xref="alg4.l5.m1.2.3.4.4.3.1"></minus><ci id="alg4.l5.m1.2.3.4.4.3.2.cmml" xref="alg4.l5.m1.2.3.4.4.3.2">𝑖</ci><cn type="integer" id="alg4.l5.m1.2.3.4.4.3.3.cmml" xref="alg4.l5.m1.2.3.4.4.3.3">1</cn></apply></apply></apply></apply><apply id="alg4.l5.m1.2.3c.cmml" xref="alg4.l5.m1.2.3"><in id="alg4.l5.m1.2.3.5.cmml" xref="alg4.l5.m1.2.3.5"></in><share href="#alg4.l5.m1.2.3.4.cmml" id="alg4.l5.m1.2.3d.cmml" xref="alg4.l5.m1.2.3"></share><apply id="alg4.l5.m1.2.3.6.cmml" xref="alg4.l5.m1.2.3.6"><csymbol cd="ambiguous" id="alg4.l5.m1.2.3.6.1.cmml" xref="alg4.l5.m1.2.3.6">subscript</csymbol><ci id="alg4.l5.m1.2.3.6.2.cmml" xref="alg4.l5.m1.2.3.6.2">𝑆</ci><list id="alg4.l5.m1.2.2.2.3.cmml" xref="alg4.l5.m1.2.2.2.2"><ci id="alg4.l5.m1.1.1.1.1.cmml" xref="alg4.l5.m1.1.1.1.1">𝑞</ci><apply id="alg4.l5.m1.2.2.2.2.1.cmml" xref="alg4.l5.m1.2.2.2.2.1"><minus id="alg4.l5.m1.2.2.2.2.1.1.cmml" xref="alg4.l5.m1.2.2.2.2.1.1"></minus><ci id="alg4.l5.m1.2.2.2.2.1.2.cmml" xref="alg4.l5.m1.2.2.2.2.1.2">𝑖</ci><cn type="integer" id="alg4.l5.m1.2.2.2.2.1.3.cmml" xref="alg4.l5.m1.2.2.2.2.1.3">1</cn></apply></list></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg4.l5.m1.2c">s_{c}\leftarrow s_{i-1}\text{ where }s_{i-1}\in S_{q,i-1}</annotation></semantics></math>

</div>
<div id="alg4.l6" class="ltx_listingline">     <math id="alg4.l6.m1.1" class="ltx_Math" alttext="\text{SoS += }(s_{c})" display="inline"><semantics id="alg4.l6.m1.1a"><mrow id="alg4.l6.m1.1.1" xref="alg4.l6.m1.1.1.cmml"><mtext id="alg4.l6.m1.1.1.3" xref="alg4.l6.m1.1.1.3a.cmml">SoS += </mtext><mo lspace="0em" rspace="0em" id="alg4.l6.m1.1.1.2" xref="alg4.l6.m1.1.1.2.cmml">​</mo><mrow id="alg4.l6.m1.1.1.1.1" xref="alg4.l6.m1.1.1.1.1.1.cmml"><mo stretchy="false" id="alg4.l6.m1.1.1.1.1.2" xref="alg4.l6.m1.1.1.1.1.1.cmml">(</mo><msub id="alg4.l6.m1.1.1.1.1.1" xref="alg4.l6.m1.1.1.1.1.1.cmml"><mi id="alg4.l6.m1.1.1.1.1.1.2" xref="alg4.l6.m1.1.1.1.1.1.2.cmml">s</mi><mi id="alg4.l6.m1.1.1.1.1.1.3" xref="alg4.l6.m1.1.1.1.1.1.3.cmml">c</mi></msub><mo stretchy="false" id="alg4.l6.m1.1.1.1.1.3" xref="alg4.l6.m1.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg4.l6.m1.1b"><apply id="alg4.l6.m1.1.1.cmml" xref="alg4.l6.m1.1.1"><times id="alg4.l6.m1.1.1.2.cmml" xref="alg4.l6.m1.1.1.2"></times><ci id="alg4.l6.m1.1.1.3a.cmml" xref="alg4.l6.m1.1.1.3"><mtext id="alg4.l6.m1.1.1.3.cmml" xref="alg4.l6.m1.1.1.3">SoS += </mtext></ci><apply id="alg4.l6.m1.1.1.1.1.1.cmml" xref="alg4.l6.m1.1.1.1.1"><csymbol cd="ambiguous" id="alg4.l6.m1.1.1.1.1.1.1.cmml" xref="alg4.l6.m1.1.1.1.1">subscript</csymbol><ci id="alg4.l6.m1.1.1.1.1.1.2.cmml" xref="alg4.l6.m1.1.1.1.1.1.2">𝑠</ci><ci id="alg4.l6.m1.1.1.1.1.1.3.cmml" xref="alg4.l6.m1.1.1.1.1.1.3">𝑐</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg4.l6.m1.1c">\text{SoS += }(s_{c})</annotation></semantics></math>

</div>
<div id="alg4.l7" class="ltx_listingline">     <span id="alg4.l7.1" class="ltx_text ltx_font_bold">if</span> <math id="alg4.l7.m1.1" class="ltx_math_unparsed" alttext="s_{c}==s_{g}" display="inline"><semantics id="alg4.l7.m1.1a"><mrow id="alg4.l7.m1.1b"><msub id="alg4.l7.m1.1.1"><mi id="alg4.l7.m1.1.1.2">s</mi><mi id="alg4.l7.m1.1.1.3">c</mi></msub><mo rspace="0em" id="alg4.l7.m1.1.2">=</mo><mo lspace="0em" id="alg4.l7.m1.1.3">=</mo><msub id="alg4.l7.m1.1.4"><mi id="alg4.l7.m1.1.4.2">s</mi><mi id="alg4.l7.m1.1.4.3">g</mi></msub></mrow><annotation encoding="application/x-tex" id="alg4.l7.m1.1c">s_{c}==s_{g}</annotation></semantics></math> <span id="alg4.l7.2" class="ltx_text ltx_font_bold">then</span>

</div>
<div id="alg4.l8" class="ltx_listingline">         <span id="alg4.l8.1" class="ltx_text">return</span> SoS += “Goal Reached”

</div>
<div id="alg4.l9" class="ltx_listingline">     <span id="alg4.l9.1" class="ltx_text ltx_font_bold">end</span> <span id="alg4.l9.2" class="ltx_text ltx_font_bold">if</span>
</div>
<div id="alg4.l10" class="ltx_listingline">     <math id="alg4.l10.m1.1" class="ltx_Math" alttext="S_{i}\leftarrow SE(s_{c})" display="inline"><semantics id="alg4.l10.m1.1a"><mrow id="alg4.l10.m1.1.1" xref="alg4.l10.m1.1.1.cmml"><msub id="alg4.l10.m1.1.1.3" xref="alg4.l10.m1.1.1.3.cmml"><mi id="alg4.l10.m1.1.1.3.2" xref="alg4.l10.m1.1.1.3.2.cmml">S</mi><mi id="alg4.l10.m1.1.1.3.3" xref="alg4.l10.m1.1.1.3.3.cmml">i</mi></msub><mo stretchy="false" id="alg4.l10.m1.1.1.2" xref="alg4.l10.m1.1.1.2.cmml">←</mo><mrow id="alg4.l10.m1.1.1.1" xref="alg4.l10.m1.1.1.1.cmml"><mi id="alg4.l10.m1.1.1.1.3" xref="alg4.l10.m1.1.1.1.3.cmml">S</mi><mo lspace="0em" rspace="0em" id="alg4.l10.m1.1.1.1.2" xref="alg4.l10.m1.1.1.1.2.cmml">​</mo><mi id="alg4.l10.m1.1.1.1.4" xref="alg4.l10.m1.1.1.1.4.cmml">E</mi><mo lspace="0em" rspace="0em" id="alg4.l10.m1.1.1.1.2a" xref="alg4.l10.m1.1.1.1.2.cmml">​</mo><mrow id="alg4.l10.m1.1.1.1.1.1" xref="alg4.l10.m1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="alg4.l10.m1.1.1.1.1.1.2" xref="alg4.l10.m1.1.1.1.1.1.1.cmml">(</mo><msub id="alg4.l10.m1.1.1.1.1.1.1" xref="alg4.l10.m1.1.1.1.1.1.1.cmml"><mi id="alg4.l10.m1.1.1.1.1.1.1.2" xref="alg4.l10.m1.1.1.1.1.1.1.2.cmml">s</mi><mi id="alg4.l10.m1.1.1.1.1.1.1.3" xref="alg4.l10.m1.1.1.1.1.1.1.3.cmml">c</mi></msub><mo stretchy="false" id="alg4.l10.m1.1.1.1.1.1.3" xref="alg4.l10.m1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg4.l10.m1.1b"><apply id="alg4.l10.m1.1.1.cmml" xref="alg4.l10.m1.1.1"><ci id="alg4.l10.m1.1.1.2.cmml" xref="alg4.l10.m1.1.1.2">←</ci><apply id="alg4.l10.m1.1.1.3.cmml" xref="alg4.l10.m1.1.1.3"><csymbol cd="ambiguous" id="alg4.l10.m1.1.1.3.1.cmml" xref="alg4.l10.m1.1.1.3">subscript</csymbol><ci id="alg4.l10.m1.1.1.3.2.cmml" xref="alg4.l10.m1.1.1.3.2">𝑆</ci><ci id="alg4.l10.m1.1.1.3.3.cmml" xref="alg4.l10.m1.1.1.3.3">𝑖</ci></apply><apply id="alg4.l10.m1.1.1.1.cmml" xref="alg4.l10.m1.1.1.1"><times id="alg4.l10.m1.1.1.1.2.cmml" xref="alg4.l10.m1.1.1.1.2"></times><ci id="alg4.l10.m1.1.1.1.3.cmml" xref="alg4.l10.m1.1.1.1.3">𝑆</ci><ci id="alg4.l10.m1.1.1.1.4.cmml" xref="alg4.l10.m1.1.1.1.4">𝐸</ci><apply id="alg4.l10.m1.1.1.1.1.1.1.cmml" xref="alg4.l10.m1.1.1.1.1.1"><csymbol cd="ambiguous" id="alg4.l10.m1.1.1.1.1.1.1.1.cmml" xref="alg4.l10.m1.1.1.1.1.1">subscript</csymbol><ci id="alg4.l10.m1.1.1.1.1.1.1.2.cmml" xref="alg4.l10.m1.1.1.1.1.1.1.2">𝑠</ci><ci id="alg4.l10.m1.1.1.1.1.1.1.3.cmml" xref="alg4.l10.m1.1.1.1.1.1.1.3">𝑐</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg4.l10.m1.1c">S_{i}\leftarrow SE(s_{c})</annotation></semantics></math>

</div>
<div id="alg4.l11" class="ltx_listingline">     <math id="alg4.l11.m1.1" class="ltx_Math" alttext="H_{i}\leftarrow h(S_{i})" display="inline"><semantics id="alg4.l11.m1.1a"><mrow id="alg4.l11.m1.1.1" xref="alg4.l11.m1.1.1.cmml"><msub id="alg4.l11.m1.1.1.3" xref="alg4.l11.m1.1.1.3.cmml"><mi id="alg4.l11.m1.1.1.3.2" xref="alg4.l11.m1.1.1.3.2.cmml">H</mi><mi id="alg4.l11.m1.1.1.3.3" xref="alg4.l11.m1.1.1.3.3.cmml">i</mi></msub><mo stretchy="false" id="alg4.l11.m1.1.1.2" xref="alg4.l11.m1.1.1.2.cmml">←</mo><mrow id="alg4.l11.m1.1.1.1" xref="alg4.l11.m1.1.1.1.cmml"><mi id="alg4.l11.m1.1.1.1.3" xref="alg4.l11.m1.1.1.1.3.cmml">h</mi><mo lspace="0em" rspace="0em" id="alg4.l11.m1.1.1.1.2" xref="alg4.l11.m1.1.1.1.2.cmml">​</mo><mrow id="alg4.l11.m1.1.1.1.1.1" xref="alg4.l11.m1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="alg4.l11.m1.1.1.1.1.1.2" xref="alg4.l11.m1.1.1.1.1.1.1.cmml">(</mo><msub id="alg4.l11.m1.1.1.1.1.1.1" xref="alg4.l11.m1.1.1.1.1.1.1.cmml"><mi id="alg4.l11.m1.1.1.1.1.1.1.2" xref="alg4.l11.m1.1.1.1.1.1.1.2.cmml">S</mi><mi id="alg4.l11.m1.1.1.1.1.1.1.3" xref="alg4.l11.m1.1.1.1.1.1.1.3.cmml">i</mi></msub><mo stretchy="false" id="alg4.l11.m1.1.1.1.1.1.3" xref="alg4.l11.m1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg4.l11.m1.1b"><apply id="alg4.l11.m1.1.1.cmml" xref="alg4.l11.m1.1.1"><ci id="alg4.l11.m1.1.1.2.cmml" xref="alg4.l11.m1.1.1.2">←</ci><apply id="alg4.l11.m1.1.1.3.cmml" xref="alg4.l11.m1.1.1.3"><csymbol cd="ambiguous" id="alg4.l11.m1.1.1.3.1.cmml" xref="alg4.l11.m1.1.1.3">subscript</csymbol><ci id="alg4.l11.m1.1.1.3.2.cmml" xref="alg4.l11.m1.1.1.3.2">𝐻</ci><ci id="alg4.l11.m1.1.1.3.3.cmml" xref="alg4.l11.m1.1.1.3.3">𝑖</ci></apply><apply id="alg4.l11.m1.1.1.1.cmml" xref="alg4.l11.m1.1.1.1"><times id="alg4.l11.m1.1.1.1.2.cmml" xref="alg4.l11.m1.1.1.1.2"></times><ci id="alg4.l11.m1.1.1.1.3.cmml" xref="alg4.l11.m1.1.1.1.3">ℎ</ci><apply id="alg4.l11.m1.1.1.1.1.1.1.cmml" xref="alg4.l11.m1.1.1.1.1.1"><csymbol cd="ambiguous" id="alg4.l11.m1.1.1.1.1.1.1.1.cmml" xref="alg4.l11.m1.1.1.1.1.1">subscript</csymbol><ci id="alg4.l11.m1.1.1.1.1.1.1.2.cmml" xref="alg4.l11.m1.1.1.1.1.1.1.2">𝑆</ci><ci id="alg4.l11.m1.1.1.1.1.1.1.3.cmml" xref="alg4.l11.m1.1.1.1.1.1.1.3">𝑖</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg4.l11.m1.1c">H_{i}\leftarrow h(S_{i})</annotation></semantics></math>

</div>
<div id="alg4.l12" class="ltx_listingline">     <math id="alg4.l12.m1.6" class="ltx_Math" alttext="S_{q,i}\leftarrow\arg\max_{S\subseteq S_{i}^{\prime},|S|=b}H_{i}(s)" display="inline"><semantics id="alg4.l12.m1.6a"><mrow id="alg4.l12.m1.6.7" xref="alg4.l12.m1.6.7.cmml"><msub id="alg4.l12.m1.6.7.2" xref="alg4.l12.m1.6.7.2.cmml"><mi id="alg4.l12.m1.6.7.2.2" xref="alg4.l12.m1.6.7.2.2.cmml">S</mi><mrow id="alg4.l12.m1.2.2.2.4" xref="alg4.l12.m1.2.2.2.3.cmml"><mi id="alg4.l12.m1.1.1.1.1" xref="alg4.l12.m1.1.1.1.1.cmml">q</mi><mo id="alg4.l12.m1.2.2.2.4.1" xref="alg4.l12.m1.2.2.2.3.cmml">,</mo><mi id="alg4.l12.m1.2.2.2.2" xref="alg4.l12.m1.2.2.2.2.cmml">i</mi></mrow></msub><mo stretchy="false" id="alg4.l12.m1.6.7.1" xref="alg4.l12.m1.6.7.1.cmml">←</mo><mrow id="alg4.l12.m1.6.7.3" xref="alg4.l12.m1.6.7.3.cmml"><mrow id="alg4.l12.m1.6.7.3.2" xref="alg4.l12.m1.6.7.3.2.cmml"><mi id="alg4.l12.m1.6.7.3.2.1" xref="alg4.l12.m1.6.7.3.2.1.cmml">arg</mi><mo lspace="0.167em" id="alg4.l12.m1.6.7.3.2a" xref="alg4.l12.m1.6.7.3.2.cmml">⁡</mo><mrow id="alg4.l12.m1.6.7.3.2.2" xref="alg4.l12.m1.6.7.3.2.2.cmml"><msub id="alg4.l12.m1.6.7.3.2.2.1" xref="alg4.l12.m1.6.7.3.2.2.1.cmml"><mi id="alg4.l12.m1.6.7.3.2.2.1.2" xref="alg4.l12.m1.6.7.3.2.2.1.2.cmml">max</mi><mrow id="alg4.l12.m1.5.5.3.3" xref="alg4.l12.m1.5.5.3.4.cmml"><mrow id="alg4.l12.m1.4.4.2.2.1" xref="alg4.l12.m1.4.4.2.2.1.cmml"><mi id="alg4.l12.m1.4.4.2.2.1.2" xref="alg4.l12.m1.4.4.2.2.1.2.cmml">S</mi><mo id="alg4.l12.m1.4.4.2.2.1.1" xref="alg4.l12.m1.4.4.2.2.1.1.cmml">⊆</mo><msubsup id="alg4.l12.m1.4.4.2.2.1.3" xref="alg4.l12.m1.4.4.2.2.1.3.cmml"><mi id="alg4.l12.m1.4.4.2.2.1.3.2.2" xref="alg4.l12.m1.4.4.2.2.1.3.2.2.cmml">S</mi><mi id="alg4.l12.m1.4.4.2.2.1.3.2.3" xref="alg4.l12.m1.4.4.2.2.1.3.2.3.cmml">i</mi><mo id="alg4.l12.m1.4.4.2.2.1.3.3" xref="alg4.l12.m1.4.4.2.2.1.3.3.cmml">′</mo></msubsup></mrow><mo id="alg4.l12.m1.5.5.3.3.3" xref="alg4.l12.m1.5.5.3.4a.cmml">,</mo><mrow id="alg4.l12.m1.5.5.3.3.2" xref="alg4.l12.m1.5.5.3.3.2.cmml"><mrow id="alg4.l12.m1.5.5.3.3.2.2.2" xref="alg4.l12.m1.5.5.3.3.2.2.1.cmml"><mo stretchy="false" id="alg4.l12.m1.5.5.3.3.2.2.2.1" xref="alg4.l12.m1.5.5.3.3.2.2.1.1.cmml">|</mo><mi id="alg4.l12.m1.3.3.1.1" xref="alg4.l12.m1.3.3.1.1.cmml">S</mi><mo stretchy="false" id="alg4.l12.m1.5.5.3.3.2.2.2.2" xref="alg4.l12.m1.5.5.3.3.2.2.1.1.cmml">|</mo></mrow><mo id="alg4.l12.m1.5.5.3.3.2.1" xref="alg4.l12.m1.5.5.3.3.2.1.cmml">=</mo><mi id="alg4.l12.m1.5.5.3.3.2.3" xref="alg4.l12.m1.5.5.3.3.2.3.cmml">b</mi></mrow></mrow></msub><mo lspace="0.167em" id="alg4.l12.m1.6.7.3.2.2a" xref="alg4.l12.m1.6.7.3.2.2.cmml">⁡</mo><msub id="alg4.l12.m1.6.7.3.2.2.2" xref="alg4.l12.m1.6.7.3.2.2.2.cmml"><mi id="alg4.l12.m1.6.7.3.2.2.2.2" xref="alg4.l12.m1.6.7.3.2.2.2.2.cmml">H</mi><mi id="alg4.l12.m1.6.7.3.2.2.2.3" xref="alg4.l12.m1.6.7.3.2.2.2.3.cmml">i</mi></msub></mrow></mrow><mo lspace="0em" rspace="0em" id="alg4.l12.m1.6.7.3.1" xref="alg4.l12.m1.6.7.3.1.cmml">​</mo><mrow id="alg4.l12.m1.6.7.3.3.2" xref="alg4.l12.m1.6.7.3.cmml"><mo stretchy="false" id="alg4.l12.m1.6.7.3.3.2.1" xref="alg4.l12.m1.6.7.3.cmml">(</mo><mi id="alg4.l12.m1.6.6" xref="alg4.l12.m1.6.6.cmml">s</mi><mo stretchy="false" id="alg4.l12.m1.6.7.3.3.2.2" xref="alg4.l12.m1.6.7.3.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg4.l12.m1.6b"><apply id="alg4.l12.m1.6.7.cmml" xref="alg4.l12.m1.6.7"><ci id="alg4.l12.m1.6.7.1.cmml" xref="alg4.l12.m1.6.7.1">←</ci><apply id="alg4.l12.m1.6.7.2.cmml" xref="alg4.l12.m1.6.7.2"><csymbol cd="ambiguous" id="alg4.l12.m1.6.7.2.1.cmml" xref="alg4.l12.m1.6.7.2">subscript</csymbol><ci id="alg4.l12.m1.6.7.2.2.cmml" xref="alg4.l12.m1.6.7.2.2">𝑆</ci><list id="alg4.l12.m1.2.2.2.3.cmml" xref="alg4.l12.m1.2.2.2.4"><ci id="alg4.l12.m1.1.1.1.1.cmml" xref="alg4.l12.m1.1.1.1.1">𝑞</ci><ci id="alg4.l12.m1.2.2.2.2.cmml" xref="alg4.l12.m1.2.2.2.2">𝑖</ci></list></apply><apply id="alg4.l12.m1.6.7.3.cmml" xref="alg4.l12.m1.6.7.3"><times id="alg4.l12.m1.6.7.3.1.cmml" xref="alg4.l12.m1.6.7.3.1"></times><apply id="alg4.l12.m1.6.7.3.2.cmml" xref="alg4.l12.m1.6.7.3.2"><arg id="alg4.l12.m1.6.7.3.2.1.cmml" xref="alg4.l12.m1.6.7.3.2.1"></arg><apply id="alg4.l12.m1.6.7.3.2.2.cmml" xref="alg4.l12.m1.6.7.3.2.2"><apply id="alg4.l12.m1.6.7.3.2.2.1.cmml" xref="alg4.l12.m1.6.7.3.2.2.1"><csymbol cd="ambiguous" id="alg4.l12.m1.6.7.3.2.2.1.1.cmml" xref="alg4.l12.m1.6.7.3.2.2.1">subscript</csymbol><max id="alg4.l12.m1.6.7.3.2.2.1.2.cmml" xref="alg4.l12.m1.6.7.3.2.2.1.2"></max><apply id="alg4.l12.m1.5.5.3.4.cmml" xref="alg4.l12.m1.5.5.3.3"><csymbol cd="ambiguous" id="alg4.l12.m1.5.5.3.4a.cmml" xref="alg4.l12.m1.5.5.3.3.3">formulae-sequence</csymbol><apply id="alg4.l12.m1.4.4.2.2.1.cmml" xref="alg4.l12.m1.4.4.2.2.1"><subset id="alg4.l12.m1.4.4.2.2.1.1.cmml" xref="alg4.l12.m1.4.4.2.2.1.1"></subset><ci id="alg4.l12.m1.4.4.2.2.1.2.cmml" xref="alg4.l12.m1.4.4.2.2.1.2">𝑆</ci><apply id="alg4.l12.m1.4.4.2.2.1.3.cmml" xref="alg4.l12.m1.4.4.2.2.1.3"><csymbol cd="ambiguous" id="alg4.l12.m1.4.4.2.2.1.3.1.cmml" xref="alg4.l12.m1.4.4.2.2.1.3">superscript</csymbol><apply id="alg4.l12.m1.4.4.2.2.1.3.2.cmml" xref="alg4.l12.m1.4.4.2.2.1.3"><csymbol cd="ambiguous" id="alg4.l12.m1.4.4.2.2.1.3.2.1.cmml" xref="alg4.l12.m1.4.4.2.2.1.3">subscript</csymbol><ci id="alg4.l12.m1.4.4.2.2.1.3.2.2.cmml" xref="alg4.l12.m1.4.4.2.2.1.3.2.2">𝑆</ci><ci id="alg4.l12.m1.4.4.2.2.1.3.2.3.cmml" xref="alg4.l12.m1.4.4.2.2.1.3.2.3">𝑖</ci></apply><ci id="alg4.l12.m1.4.4.2.2.1.3.3.cmml" xref="alg4.l12.m1.4.4.2.2.1.3.3">′</ci></apply></apply><apply id="alg4.l12.m1.5.5.3.3.2.cmml" xref="alg4.l12.m1.5.5.3.3.2"><eq id="alg4.l12.m1.5.5.3.3.2.1.cmml" xref="alg4.l12.m1.5.5.3.3.2.1"></eq><apply id="alg4.l12.m1.5.5.3.3.2.2.1.cmml" xref="alg4.l12.m1.5.5.3.3.2.2.2"><abs id="alg4.l12.m1.5.5.3.3.2.2.1.1.cmml" xref="alg4.l12.m1.5.5.3.3.2.2.2.1"></abs><ci id="alg4.l12.m1.3.3.1.1.cmml" xref="alg4.l12.m1.3.3.1.1">𝑆</ci></apply><ci id="alg4.l12.m1.5.5.3.3.2.3.cmml" xref="alg4.l12.m1.5.5.3.3.2.3">𝑏</ci></apply></apply></apply><apply id="alg4.l12.m1.6.7.3.2.2.2.cmml" xref="alg4.l12.m1.6.7.3.2.2.2"><csymbol cd="ambiguous" id="alg4.l12.m1.6.7.3.2.2.2.1.cmml" xref="alg4.l12.m1.6.7.3.2.2.2">subscript</csymbol><ci id="alg4.l12.m1.6.7.3.2.2.2.2.cmml" xref="alg4.l12.m1.6.7.3.2.2.2.2">𝐻</ci><ci id="alg4.l12.m1.6.7.3.2.2.2.3.cmml" xref="alg4.l12.m1.6.7.3.2.2.2.3">𝑖</ci></apply></apply></apply><ci id="alg4.l12.m1.6.6.cmml" xref="alg4.l12.m1.6.6">𝑠</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg4.l12.m1.6c">S_{q,i}\leftarrow\arg\max_{S\subseteq S_{i}^{\prime},|S|=b}H_{i}(s)</annotation></semantics></math>

</div>
<div id="alg4.l13" class="ltx_listingline">     <math id="alg4.l13.m1.3" class="ltx_Math" alttext="\text{SoS += }(S_{q,i})" display="inline"><semantics id="alg4.l13.m1.3a"><mrow id="alg4.l13.m1.3.3" xref="alg4.l13.m1.3.3.cmml"><mtext id="alg4.l13.m1.3.3.3" xref="alg4.l13.m1.3.3.3a.cmml">SoS += </mtext><mo lspace="0em" rspace="0em" id="alg4.l13.m1.3.3.2" xref="alg4.l13.m1.3.3.2.cmml">​</mo><mrow id="alg4.l13.m1.3.3.1.1" xref="alg4.l13.m1.3.3.1.1.1.cmml"><mo stretchy="false" id="alg4.l13.m1.3.3.1.1.2" xref="alg4.l13.m1.3.3.1.1.1.cmml">(</mo><msub id="alg4.l13.m1.3.3.1.1.1" xref="alg4.l13.m1.3.3.1.1.1.cmml"><mi id="alg4.l13.m1.3.3.1.1.1.2" xref="alg4.l13.m1.3.3.1.1.1.2.cmml">S</mi><mrow id="alg4.l13.m1.2.2.2.4" xref="alg4.l13.m1.2.2.2.3.cmml"><mi id="alg4.l13.m1.1.1.1.1" xref="alg4.l13.m1.1.1.1.1.cmml">q</mi><mo id="alg4.l13.m1.2.2.2.4.1" xref="alg4.l13.m1.2.2.2.3.cmml">,</mo><mi id="alg4.l13.m1.2.2.2.2" xref="alg4.l13.m1.2.2.2.2.cmml">i</mi></mrow></msub><mo stretchy="false" id="alg4.l13.m1.3.3.1.1.3" xref="alg4.l13.m1.3.3.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg4.l13.m1.3b"><apply id="alg4.l13.m1.3.3.cmml" xref="alg4.l13.m1.3.3"><times id="alg4.l13.m1.3.3.2.cmml" xref="alg4.l13.m1.3.3.2"></times><ci id="alg4.l13.m1.3.3.3a.cmml" xref="alg4.l13.m1.3.3.3"><mtext id="alg4.l13.m1.3.3.3.cmml" xref="alg4.l13.m1.3.3.3">SoS += </mtext></ci><apply id="alg4.l13.m1.3.3.1.1.1.cmml" xref="alg4.l13.m1.3.3.1.1"><csymbol cd="ambiguous" id="alg4.l13.m1.3.3.1.1.1.1.cmml" xref="alg4.l13.m1.3.3.1.1">subscript</csymbol><ci id="alg4.l13.m1.3.3.1.1.1.2.cmml" xref="alg4.l13.m1.3.3.1.1.1.2">𝑆</ci><list id="alg4.l13.m1.2.2.2.3.cmml" xref="alg4.l13.m1.2.2.2.4"><ci id="alg4.l13.m1.1.1.1.1.cmml" xref="alg4.l13.m1.1.1.1.1">𝑞</ci><ci id="alg4.l13.m1.2.2.2.2.cmml" xref="alg4.l13.m1.2.2.2.2">𝑖</ci></list></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg4.l13.m1.3c">\text{SoS += }(S_{q,i})</annotation></semantics></math>

</div>
<div id="alg4.l14" class="ltx_listingline">
<span id="alg4.l14.1" class="ltx_text ltx_font_bold">end</span> <span id="alg4.l14.2" class="ltx_text ltx_font_bold">for</span>
</div>
<div id="alg4.l15" class="ltx_listingline">
<span id="alg4.l15.1" class="ltx_text ltx_font_bold">return</span> SoS

</div>
</div>
</figure>
<figure id="A2.F7" class="ltx_figure"><svg id="A2.F7.pic1" class="ltx_picture ltx_centering" height="165.89" overflow="visible" version="1.1" width="600"><g transform="translate(0,165.89) matrix(1 0 0 -1 0 0)" fill="#000000" stroke="#000000" stroke-width="0.4pt"><g fill="#DFDFDF" fill-opacity="1.0"><path d="M 0 12.03 L 0 153.87 C 0 160.51 5.38 165.89 12.03 165.89 L 587.97 165.89 C 594.62 165.89 600 160.51 600 153.87 L 600 12.03 C 600 5.38 594.62 0 587.97 0 L 12.03 0 C 5.38 0 0 5.38 0 12.03 Z" style="stroke:none"></path></g><g fill="#F2F2F2" fill-opacity="1.0"><path d="M 4.15 12.03 L 4.15 136.99 L 595.85 136.99 L 595.85 12.03 C 595.85 7.68 592.32 4.15 587.97 4.15 L 12.03 4.15 C 7.68 4.15 4.15 7.68 4.15 12.03 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 22.14 145.29)"><foreignObject width="555.72" height="12.3" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#808080">
<span id="A2.F7.pic1.1.1.1.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:401.6pt;">
<span id="A2.F7.pic1.1.1.1.1.1.1" class="ltx_p"><span id="A2.F7.pic1.1.1.1.1.1.1.1" class="ltx_text ltx_font_bold">Optimal Path Solution</span></span>
</span></foreignObject></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 22.14 22.14)"><foreignObject width="555.72" height="96.86" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#000000">
<span id="A2.F7.pic1.2.2.2.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:401.6pt;">
<span id="A2.F7.pic1.2.2.2.1.1.1" class="ltx_p"><span id="A2.F7.pic1.2.2.2.1.1.1.1" class="ltx_text ltx_font_typewriter">Current State: 18:[74, 24, 36, 44], Operations: []</span></span>
<span id="A2.F7.pic1.2.2.2.1.1.2" class="ltx_p"><span id="A2.F7.pic1.2.2.2.1.1.2.1" class="ltx_text ltx_font_typewriter">Exploring Operation: 74+24=98, Resulting Numbers: [36, 44, 98]</span></span>
<span id="A2.F7.pic1.2.2.2.1.1.3" class="ltx_p"><span id="A2.F7.pic1.2.2.2.1.1.3.1" class="ltx_text ltx_font_typewriter">Generated Node #2: [36, 44, 98] from Operation: 74+24=98</span></span>
<span id="A2.F7.pic1.2.2.2.1.1.4" class="ltx_p"><span id="A2.F7.pic1.2.2.2.1.1.4.1" class="ltx_text ltx_font_typewriter">Current State: 18:[36, 44, 98], Operations: [’74+24=98’]</span></span>
<span id="A2.F7.pic1.2.2.2.1.1.5" class="ltx_p"><span id="A2.F7.pic1.2.2.2.1.1.5.1" class="ltx_text ltx_font_typewriter">Exploring Operation: 36+44=80, Resulting Numbers: [98, 80]</span></span>
<span id="A2.F7.pic1.2.2.2.1.1.6" class="ltx_p"><span id="A2.F7.pic1.2.2.2.1.1.6.1" class="ltx_text ltx_font_typewriter">Generated Node #3: [98, 80] from Operation: 36+44=80</span></span>
<span id="A2.F7.pic1.2.2.2.1.1.7" class="ltx_p"><span id="A2.F7.pic1.2.2.2.1.1.7.1" class="ltx_text ltx_font_typewriter">Current State: 18:[98, 80], Operations: [’74+24=98’, ’36+44=80’]</span></span>
<span id="A2.F7.pic1.2.2.2.1.1.8" class="ltx_p"><span id="A2.F7.pic1.2.2.2.1.1.8.1" class="ltx_text ltx_font_typewriter">Exploring Operation: 98-80=18, Resulting Numbers: [18]</span></span>
<span id="A2.F7.pic1.2.2.2.1.1.9" class="ltx_p"><span id="A2.F7.pic1.2.2.2.1.1.9.1" class="ltx_text ltx_font_typewriter">18,18 equal: Goal Reached</span></span>
</span></foreignObject></g></g></svg>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 7: </span>An example of an Optimal Path to the solution.</figcaption>
</figure>
<figure id="A2.F8" class="ltx_figure"><svg id="A2.F8.pic1" class="ltx_picture ltx_centering" height="827.38" overflow="visible" version="1.1" width="600"><g transform="translate(0,827.38) matrix(1 0 0 -1 0 0)" fill="#000000" stroke="#000000" stroke-width="0.4pt"><g fill="#DFDFDF" fill-opacity="1.0"><path d="M 0 12.03 L 0 815.35 C 0 821.99 5.38 827.38 12.03 827.38 L 587.97 827.38 C 594.62 827.38 600 821.99 600 815.35 L 600 12.03 C 600 5.38 594.62 0 587.97 0 L 12.03 0 C 5.38 0 0 5.38 0 12.03 Z" style="stroke:none"></path></g><g fill="#F2F2F2" fill-opacity="1.0"><path d="M 4.15 12.03 L 4.15 801.16 L 595.85 801.16 L 595.85 12.03 C 595.85 7.68 592.32 4.15 587.97 4.15 L 12.03 4.15 C 7.68 4.15 4.15 7.68 4.15 12.03 Z" style="stroke:none"></path></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 22.14 809.46)"><foreignObject width="555.72" height="9.61" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#808080">
<span id="A2.F8.pic1.1.1.1.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:401.6pt;">
<span id="A2.F8.pic1.1.1.1.1.1.1" class="ltx_p"><span id="A2.F8.pic1.1.1.1.1.1.1.1" class="ltx_text ltx_font_bold">Stream of Search Solution</span></span>
</span></foreignObject></g><g fill-opacity="1.0" transform="matrix(1.0 0.0 0.0 1.0 22.14 22.14)"><foreignObject width="555.72" height="761.04" transform="matrix(1 0 0 -1 0 16.6)" overflow="visible" color="#000000">
<span id="A2.F8.pic1.2.2.2.1.1" class="ltx_inline-block ltx_minipage ltx_align_bottom" style="width:401.6pt;">
<span id="A2.F8.pic1.2.2.2.1.1.1" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.1.1" class="ltx_text ltx_font_typewriter">Current State: 18:[74, 24, 36, 44], Operations: []</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.2" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.2.1" class="ltx_text ltx_font_typewriter">Exploring Operation: 74-44=30, Resulting Numbers: [24, 36, 30]</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.3" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.3.1" class="ltx_text ltx_font_typewriter">Generated Node #0,0: 18:[24, 36, 30] Operation: 74-44=30</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.4" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.4.1" class="ltx_text ltx_font_typewriter">Exploring Operation: 44-36=8, Resulting Numbers: [74, 24, 8]</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.5" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.5.1" class="ltx_text ltx_font_typewriter">Generated Node #0,1: 18:[74, 24, 8] Operation: 44-36=8</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.6" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.6.1" class="ltx_text ltx_font_typewriter">Exploring Operation: 44-24=20, Resulting Numbers: [74, 36, 20]</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.7" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.7.1" class="ltx_text ltx_font_typewriter">Generated Node #0,2: 18:[74, 36, 20] Operation: 44-24=20</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.8" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.8.1" class="ltx_text ltx_font_typewriter">Exploring Operation: 74-24=50, Resulting Numbers: [36, 44, 50]</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.9" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.9.1" class="ltx_text ltx_font_typewriter">Generated Node #0,3: 18:[36, 44, 50] Operation: 74-24=50</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.10" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.10.1" class="ltx_text ltx_font_typewriter">Exploring Operation: 74-36=38, Resulting Numbers: [24, 44, 38]</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.11" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.11.1" class="ltx_text ltx_font_typewriter">Generated Node #0,4: 18:[24, 44, 38] Operation: 74-36=38</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.12" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.12.1" class="ltx_text ltx_font_typewriter">Moving to Node #0,0</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.13" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.13.1" class="ltx_text ltx_font_typewriter">Current State: 18:[24, 36, 30], Operations: [’74-44=30’]</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.14" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.14.1" class="ltx_text ltx_font_typewriter">Exploring Operation: 36-24=12, Resulting Numbers: [30, 12]</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.15" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.15.1" class="ltx_text ltx_font_typewriter">Generated Node #0,0,0: 18:[30, 12] Operation: 36-24=12</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.16" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.16.1" class="ltx_text ltx_font_typewriter">Exploring Operation: 36-30=6, Resulting Numbers: [24, 6]</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.17" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.17.1" class="ltx_text ltx_font_typewriter">Generated Node #0,0,1: 18:[24, 6] Operation: 36-30=6</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.18" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.18.1" class="ltx_text ltx_font_typewriter">Exploring Operation: 24+30=54, Resulting Numbers: [36, 54]</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.19" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.19.1" class="ltx_text ltx_font_typewriter">Generated Node #0,0,2: 18:[36, 54] Operation: 24+30=54</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.20" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.20.1" class="ltx_text ltx_font_typewriter">Exploring Operation: 30-24=6, Resulting Numbers: [36, 6]</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.21" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.21.1" class="ltx_text ltx_font_typewriter">Generated Node #0,0,3: 18:[36, 6] Operation: 30-24=6</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.22" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.22.1" class="ltx_text ltx_font_typewriter">Exploring Operation: 24+36=60, Resulting Numbers: [30, 60]</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.23" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.23.1" class="ltx_text ltx_font_typewriter">Generated Node #0,0,4: 18:[30, 60] Operation: 24+36=60</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.24" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.24.1" class="ltx_text ltx_font_typewriter">Moving to Node #0,4</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.25" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.25.1" class="ltx_text ltx_font_typewriter">Current State: 18:[24, 44, 38], Operations: [’74-36=38’]</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.26" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.26.1" class="ltx_text ltx_font_typewriter">Exploring Operation: 44-38=6, Resulting Numbers: [24, 6]</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.27" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.27.1" class="ltx_text ltx_font_typewriter">Generated Node #0,4,0: 18:[24, 6] Operation: 44-38=6</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.28" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.28.1" class="ltx_text ltx_font_typewriter">Exploring Operation: 44-24=20, Resulting Numbers: [38, 20]</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.29" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.29.1" class="ltx_text ltx_font_typewriter">Generated Node #0,4,1: 18:[38, 20] Operation: 44-24=20</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.30" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.30.1" class="ltx_text ltx_font_typewriter">Exploring Operation: 24+38=62, Resulting Numbers: [44, 62]</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.31" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.31.1" class="ltx_text ltx_font_typewriter">Generated Node #0,4,2: 18:[44, 62] Operation: 24+38=62</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.32" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.32.1" class="ltx_text ltx_font_typewriter">Exploring Operation: 24+44=68, Resulting Numbers: [38, 68]</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.33" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.33.1" class="ltx_text ltx_font_typewriter">Generated Node #0,4,3: 18:[38, 68] Operation: 24+44=68</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.34" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.34.1" class="ltx_text ltx_font_typewriter">Exploring Operation: 38-24=14, Resulting Numbers: [44, 14]</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.35" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.35.1" class="ltx_text ltx_font_typewriter">Generated Node #0,4,4: 18:[44, 14] Operation: 38-24=14</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.36" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.36.1" class="ltx_text ltx_font_typewriter">Moving to Node #0,1</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.37" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.37.1" class="ltx_text ltx_font_typewriter">Current State: 18:[74, 24, 8], Operations: [’44-36=8’]</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.38" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.38.1" class="ltx_text ltx_font_typewriter">Exploring Operation: 74-8=66, Resulting Numbers: [24, 66]</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.39" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.39.1" class="ltx_text ltx_font_typewriter">Generated Node #0,1,0: 18:[24, 66] Operation: 74-8=66</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.40" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.40.1" class="ltx_text ltx_font_typewriter">Exploring Operation: 24-8=16, Resulting Numbers: [74, 16]</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.41" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.41.1" class="ltx_text ltx_font_typewriter">Generated Node #0,1,1: 18:[74, 16] Operation: 24-8=16</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.42" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.42.1" class="ltx_text ltx_font_typewriter">Exploring Operation: 24+8=32, Resulting Numbers: [74, 32]</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.43" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.43.1" class="ltx_text ltx_font_typewriter">Generated Node #0,1,2: 18:[74, 32] Operation: 24+8=32</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.44" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.44.1" class="ltx_text ltx_font_typewriter">Exploring Operation: 74+8=82, Resulting Numbers: [24, 82]</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.45" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.45.1" class="ltx_text ltx_font_typewriter">Generated Node #0,1,3: 18:[24, 82] Operation: 74+8=82</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.46" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.46.1" class="ltx_text ltx_font_typewriter">Exploring Operation: 74-24=50, Resulting Numbers: [8, 50]</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.47" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.47.1" class="ltx_text ltx_font_typewriter">Generated Node #0,1,4: 18:[8, 50] Operation: 74-24=50</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.48" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.48.1" class="ltx_text ltx_font_typewriter">Moving to Node #0,2</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.49" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.49.1" class="ltx_text ltx_font_typewriter">Current State: 18:[74, 36, 20], Operations: [’44-24=20’]</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.50" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.50.1" class="ltx_text ltx_font_typewriter">Exploring Operation: 74+20=94, Resulting Numbers: [36, 94]</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.51" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.51.1" class="ltx_text ltx_font_typewriter">Generated Node #0,2,0: 18:[36, 94] Operation: 74+20=94</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.52" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.52.1" class="ltx_text ltx_font_typewriter">Exploring Operation: 74-20=54, Resulting Numbers: [36, 54]</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.53" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.53.1" class="ltx_text ltx_font_typewriter">Generated Node #0,2,1: 18:[36, 54] Operation: 74-20=54</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.54" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.54.1" class="ltx_text ltx_font_typewriter">Exploring Operation: 36-20=16, Resulting Numbers: [74, 16]</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.55" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.55.1" class="ltx_text ltx_font_typewriter">Generated Node #0,2,2: 18:[74, 16] Operation: 36-20=16</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.56" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.56.1" class="ltx_text ltx_font_typewriter">Exploring Operation: 74+36=110, Resulting Numbers: [20, 110]</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.57" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.57.1" class="ltx_text ltx_font_typewriter">Generated Node #0,2,3: 18:[20, 110] Operation: 74+36=110</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.58" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.58.1" class="ltx_text ltx_font_typewriter">Exploring Operation: 74-36=38, Resulting Numbers: [20, 38]</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.59" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.59.1" class="ltx_text ltx_font_typewriter">Generated Node #0,2,4: 18:[20, 38] Operation: 74-36=38</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.60" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.60.1" class="ltx_text ltx_font_typewriter">Moving to Node #0,3</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.61" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.61.1" class="ltx_text ltx_font_typewriter">Current State: 18:[36, 44, 50], Operations: [’74-24=50’]</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.62" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.62.1" class="ltx_text ltx_font_typewriter">Exploring Operation: 44-36=8, Resulting Numbers: [50, 8]</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.63" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.63.1" class="ltx_text ltx_font_typewriter">Generated Node #0,3,0: 18:[50, 8] Operation: 44-36=8</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.64" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.64.1" class="ltx_text ltx_font_typewriter">Exploring Operation: 36+44=80, Resulting Numbers: [50, 80]</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.65" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.65.1" class="ltx_text ltx_font_typewriter">Generated Node #0,3,1: 18:[50, 80] Operation: 36+44=80</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.66" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.66.1" class="ltx_text ltx_font_typewriter">Exploring Operation: 50-36=14, Resulting Numbers: [44, 14]</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.67" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.67.1" class="ltx_text ltx_font_typewriter">Generated Node #0,3,2: 18:[44, 14] Operation: 50-36=14</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.68" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.68.1" class="ltx_text ltx_font_typewriter">Exploring Operation: 50-44=6, Resulting Numbers: [36, 6]</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.69" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.69.1" class="ltx_text ltx_font_typewriter">Generated Node #0,3,3: 18:[36, 6] Operation: 50-44=6</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.70" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.70.1" class="ltx_text ltx_font_typewriter">Exploring Operation: 36+50=86, Resulting Numbers: [44, 86]</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.71" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.71.1" class="ltx_text ltx_font_typewriter">Generated Node #0,3,4: 18:[44, 86] Operation: 36+50=86</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.72" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.72.1" class="ltx_text ltx_font_typewriter">Moving to Node #0,0,0</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.73" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.73.1" class="ltx_text ltx_font_typewriter">Current State: 18:[30, 12], Operations: [’74-44=30’, ’36-24=12’]</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.74" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.74.1" class="ltx_text ltx_font_typewriter">Exploring Operation: 30-12=18, Resulting Numbers: [18]</span></span>
<span id="A2.F8.pic1.2.2.2.1.1.75" class="ltx_p"><span id="A2.F8.pic1.2.2.2.1.1.75.1" class="ltx_text ltx_font_typewriter">18,18 equal: Goal Reached</span></span>
</span></foreignObject></g></g></svg>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 8: </span>An example of a Stream of Search trajectory.</figcaption>
</figure>
</section>
<section id="A3" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix C </span>Training Details</h2>

<div id="A3.p1" class="ltx_para ltx_noindent">
<p id="A3.p1.1" class="ltx_p">Models are trained with an Adam Optimizer on 4 x 80 GB A100 GPUs. For all methods, model selection is done based on accuracy on a validation set of 1,000 problems. All hyperparameters in this section are per GPU unless specified otherwise.</p>
</div>
<section id="A3.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">C.1 </span>Model Architecture</h3>

<div id="A3.SS1.p1" class="ltx_para ltx_noindent">
<p id="A3.SS1.p1.1" class="ltx_p">We train a GPT-Neo model <cite class="ltx_cite ltx_citemacro_citep">(Gao et al., <a href="#bib.bib9" title="" class="ltx_ref">2020</a>)</cite> with the following architecture:</p>
<ul id="A3.I1" class="ltx_itemize">
<li id="A3.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A3.I1.i1.p1" class="ltx_para">
<p id="A3.I1.i1.p1.1" class="ltx_p">Layers: 16 (all global attention)</p>
</div>
</li>
<li id="A3.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A3.I1.i2.p1" class="ltx_para">
<p id="A3.I1.i2.p1.1" class="ltx_p">Heads: 16</p>
</div>
</li>
<li id="A3.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A3.I1.i3.p1" class="ltx_para">
<p id="A3.I1.i3.p1.1" class="ltx_p">Context length: 4096</p>
</div>
</li>
<li id="A3.I1.i4" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A3.I1.i4.p1" class="ltx_para">
<p id="A3.I1.i4.p1.1" class="ltx_p">Hidden Size: 1024</p>
</div>
</li>
<li id="A3.I1.i5" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A3.I1.i5.p1" class="ltx_para">
<p id="A3.I1.i5.p1.1" class="ltx_p">Dropout: 0.1</p>
</div>
</li>
<li id="A3.I1.i6" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A3.I1.i6.p1" class="ltx_para">
<p id="A3.I1.i6.p1.1" class="ltx_p">Data Type: <span id="A3.I1.i6.p1.1.1" class="ltx_text ltx_font_typewriter">bf16</span></p>
</div>
</li>
<li id="A3.I1.i7" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A3.I1.i7.p1" class="ltx_para ltx_noindent">
<p id="A3.I1.i7.p1.1" class="ltx_p">Attention: Flash-Attention-2 <cite class="ltx_cite ltx_citemacro_citep">(Dao et al., <a href="#bib.bib6" title="" class="ltx_ref">2022</a>)</cite></p>
</div>
</li>
</ul>
</div>
</section>
<section id="A3.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">C.2 </span>Stream of Search Model</h3>

<div id="A3.SS2.p1" class="ltx_para ltx_noindent">
<ul id="A3.I2" class="ltx_itemize">
<li id="A3.I2.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A3.I2.i1.p1" class="ltx_para">
<p id="A3.I2.i1.p1.1" class="ltx_p">Number of training examples: <math id="A3.I2.i1.p1.1.m1.1" class="ltx_Math" alttext="5\times 10^{5}" display="inline"><semantics id="A3.I2.i1.p1.1.m1.1a"><mrow id="A3.I2.i1.p1.1.m1.1.1" xref="A3.I2.i1.p1.1.m1.1.1.cmml"><mn id="A3.I2.i1.p1.1.m1.1.1.2" xref="A3.I2.i1.p1.1.m1.1.1.2.cmml">5</mn><mo lspace="0.222em" rspace="0.222em" id="A3.I2.i1.p1.1.m1.1.1.1" xref="A3.I2.i1.p1.1.m1.1.1.1.cmml">×</mo><msup id="A3.I2.i1.p1.1.m1.1.1.3" xref="A3.I2.i1.p1.1.m1.1.1.3.cmml"><mn id="A3.I2.i1.p1.1.m1.1.1.3.2" xref="A3.I2.i1.p1.1.m1.1.1.3.2.cmml">10</mn><mn id="A3.I2.i1.p1.1.m1.1.1.3.3" xref="A3.I2.i1.p1.1.m1.1.1.3.3.cmml">5</mn></msup></mrow><annotation-xml encoding="MathML-Content" id="A3.I2.i1.p1.1.m1.1b"><apply id="A3.I2.i1.p1.1.m1.1.1.cmml" xref="A3.I2.i1.p1.1.m1.1.1"><times id="A3.I2.i1.p1.1.m1.1.1.1.cmml" xref="A3.I2.i1.p1.1.m1.1.1.1"></times><cn type="integer" id="A3.I2.i1.p1.1.m1.1.1.2.cmml" xref="A3.I2.i1.p1.1.m1.1.1.2">5</cn><apply id="A3.I2.i1.p1.1.m1.1.1.3.cmml" xref="A3.I2.i1.p1.1.m1.1.1.3"><csymbol cd="ambiguous" id="A3.I2.i1.p1.1.m1.1.1.3.1.cmml" xref="A3.I2.i1.p1.1.m1.1.1.3">superscript</csymbol><cn type="integer" id="A3.I2.i1.p1.1.m1.1.1.3.2.cmml" xref="A3.I2.i1.p1.1.m1.1.1.3.2">10</cn><cn type="integer" id="A3.I2.i1.p1.1.m1.1.1.3.3.cmml" xref="A3.I2.i1.p1.1.m1.1.1.3.3">5</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.I2.i1.p1.1.m1.1c">5\times 10^{5}</annotation></semantics></math></p>
</div>
</li>
<li id="A3.I2.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A3.I2.i2.p1" class="ltx_para">
<p id="A3.I2.i2.p1.1" class="ltx_p">Evaluation steps: 500</p>
</div>
</li>
<li id="A3.I2.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A3.I2.i3.p1" class="ltx_para">
<p id="A3.I2.i3.p1.1" class="ltx_p">Batch size: 24</p>
</div>
</li>
<li id="A3.I2.i4" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A3.I2.i4.p1" class="ltx_para">
<p id="A3.I2.i4.p1.1" class="ltx_p">Gradient accumulation steps: 1</p>
</div>
</li>
<li id="A3.I2.i5" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A3.I2.i5.p1" class="ltx_para">
<p id="A3.I2.i5.p1.1" class="ltx_p">Number of training steps: 50,000</p>
</div>
</li>
<li id="A3.I2.i6" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A3.I2.i6.p1" class="ltx_para">
<p id="A3.I2.i6.p1.1" class="ltx_p">Weight decay: 0.01</p>
</div>
</li>
<li id="A3.I2.i7" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A3.I2.i7.p1" class="ltx_para">
<p id="A3.I2.i7.p1.1" class="ltx_p">Warmup steps: 1</p>
</div>
</li>
<li id="A3.I2.i8" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A3.I2.i8.p1" class="ltx_para">
<p id="A3.I2.i8.p1.1" class="ltx_p">Learning rate scheduler type: Cosine</p>
</div>
</li>
<li id="A3.I2.i9" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A3.I2.i9.p1" class="ltx_para ltx_noindent">
<p id="A3.I2.i9.p1.1" class="ltx_p">Learning rate: <math id="A3.I2.i9.p1.1.m1.1" class="ltx_Math" alttext="1\times 10^{-5}" display="inline"><semantics id="A3.I2.i9.p1.1.m1.1a"><mrow id="A3.I2.i9.p1.1.m1.1.1" xref="A3.I2.i9.p1.1.m1.1.1.cmml"><mn id="A3.I2.i9.p1.1.m1.1.1.2" xref="A3.I2.i9.p1.1.m1.1.1.2.cmml">1</mn><mo lspace="0.222em" rspace="0.222em" id="A3.I2.i9.p1.1.m1.1.1.1" xref="A3.I2.i9.p1.1.m1.1.1.1.cmml">×</mo><msup id="A3.I2.i9.p1.1.m1.1.1.3" xref="A3.I2.i9.p1.1.m1.1.1.3.cmml"><mn id="A3.I2.i9.p1.1.m1.1.1.3.2" xref="A3.I2.i9.p1.1.m1.1.1.3.2.cmml">10</mn><mrow id="A3.I2.i9.p1.1.m1.1.1.3.3" xref="A3.I2.i9.p1.1.m1.1.1.3.3.cmml"><mo id="A3.I2.i9.p1.1.m1.1.1.3.3a" xref="A3.I2.i9.p1.1.m1.1.1.3.3.cmml">−</mo><mn id="A3.I2.i9.p1.1.m1.1.1.3.3.2" xref="A3.I2.i9.p1.1.m1.1.1.3.3.2.cmml">5</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="A3.I2.i9.p1.1.m1.1b"><apply id="A3.I2.i9.p1.1.m1.1.1.cmml" xref="A3.I2.i9.p1.1.m1.1.1"><times id="A3.I2.i9.p1.1.m1.1.1.1.cmml" xref="A3.I2.i9.p1.1.m1.1.1.1"></times><cn type="integer" id="A3.I2.i9.p1.1.m1.1.1.2.cmml" xref="A3.I2.i9.p1.1.m1.1.1.2">1</cn><apply id="A3.I2.i9.p1.1.m1.1.1.3.cmml" xref="A3.I2.i9.p1.1.m1.1.1.3"><csymbol cd="ambiguous" id="A3.I2.i9.p1.1.m1.1.1.3.1.cmml" xref="A3.I2.i9.p1.1.m1.1.1.3">superscript</csymbol><cn type="integer" id="A3.I2.i9.p1.1.m1.1.1.3.2.cmml" xref="A3.I2.i9.p1.1.m1.1.1.3.2">10</cn><apply id="A3.I2.i9.p1.1.m1.1.1.3.3.cmml" xref="A3.I2.i9.p1.1.m1.1.1.3.3"><minus id="A3.I2.i9.p1.1.m1.1.1.3.3.1.cmml" xref="A3.I2.i9.p1.1.m1.1.1.3.3"></minus><cn type="integer" id="A3.I2.i9.p1.1.m1.1.1.3.3.2.cmml" xref="A3.I2.i9.p1.1.m1.1.1.3.3.2">5</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.I2.i9.p1.1.m1.1c">1\times 10^{-5}</annotation></semantics></math></p>
</div>
</li>
</ul>
</div>
</section>
<section id="A3.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">C.3 </span>Optimal Paths Model</h3>

<div id="A3.SS3.p1" class="ltx_para ltx_noindent">
<ul id="A3.I3" class="ltx_itemize">
<li id="A3.I3.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A3.I3.i1.p1" class="ltx_para">
<p id="A3.I3.i1.p1.1" class="ltx_p">Number of training examples: <math id="A3.I3.i1.p1.1.m1.1" class="ltx_Math" alttext="5\times 10^{5}" display="inline"><semantics id="A3.I3.i1.p1.1.m1.1a"><mrow id="A3.I3.i1.p1.1.m1.1.1" xref="A3.I3.i1.p1.1.m1.1.1.cmml"><mn id="A3.I3.i1.p1.1.m1.1.1.2" xref="A3.I3.i1.p1.1.m1.1.1.2.cmml">5</mn><mo lspace="0.222em" rspace="0.222em" id="A3.I3.i1.p1.1.m1.1.1.1" xref="A3.I3.i1.p1.1.m1.1.1.1.cmml">×</mo><msup id="A3.I3.i1.p1.1.m1.1.1.3" xref="A3.I3.i1.p1.1.m1.1.1.3.cmml"><mn id="A3.I3.i1.p1.1.m1.1.1.3.2" xref="A3.I3.i1.p1.1.m1.1.1.3.2.cmml">10</mn><mn id="A3.I3.i1.p1.1.m1.1.1.3.3" xref="A3.I3.i1.p1.1.m1.1.1.3.3.cmml">5</mn></msup></mrow><annotation-xml encoding="MathML-Content" id="A3.I3.i1.p1.1.m1.1b"><apply id="A3.I3.i1.p1.1.m1.1.1.cmml" xref="A3.I3.i1.p1.1.m1.1.1"><times id="A3.I3.i1.p1.1.m1.1.1.1.cmml" xref="A3.I3.i1.p1.1.m1.1.1.1"></times><cn type="integer" id="A3.I3.i1.p1.1.m1.1.1.2.cmml" xref="A3.I3.i1.p1.1.m1.1.1.2">5</cn><apply id="A3.I3.i1.p1.1.m1.1.1.3.cmml" xref="A3.I3.i1.p1.1.m1.1.1.3"><csymbol cd="ambiguous" id="A3.I3.i1.p1.1.m1.1.1.3.1.cmml" xref="A3.I3.i1.p1.1.m1.1.1.3">superscript</csymbol><cn type="integer" id="A3.I3.i1.p1.1.m1.1.1.3.2.cmml" xref="A3.I3.i1.p1.1.m1.1.1.3.2">10</cn><cn type="integer" id="A3.I3.i1.p1.1.m1.1.1.3.3.cmml" xref="A3.I3.i1.p1.1.m1.1.1.3.3">5</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.I3.i1.p1.1.m1.1c">5\times 10^{5}</annotation></semantics></math></p>
</div>
</li>
<li id="A3.I3.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A3.I3.i2.p1" class="ltx_para">
<p id="A3.I3.i2.p1.1" class="ltx_p">Batch size: 24</p>
</div>
</li>
<li id="A3.I3.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A3.I3.i3.p1" class="ltx_para">
<p id="A3.I3.i3.p1.1" class="ltx_p">Gradient accumulation steps: 1</p>
</div>
</li>
<li id="A3.I3.i4" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A3.I3.i4.p1" class="ltx_para">
<p id="A3.I3.i4.p1.1" class="ltx_p">Number of training steps: 50,000</p>
</div>
</li>
<li id="A3.I3.i5" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A3.I3.i5.p1" class="ltx_para">
<p id="A3.I3.i5.p1.1" class="ltx_p">Weight decay: 0.01</p>
</div>
</li>
<li id="A3.I3.i6" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A3.I3.i6.p1" class="ltx_para">
<p id="A3.I3.i6.p1.1" class="ltx_p">Warmup steps: 1</p>
</div>
</li>
<li id="A3.I3.i7" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A3.I3.i7.p1" class="ltx_para">
<p id="A3.I3.i7.p1.1" class="ltx_p">Learning rate scheduler type: cosine</p>
</div>
</li>
<li id="A3.I3.i8" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A3.I3.i8.p1" class="ltx_para ltx_noindent">
<p id="A3.I3.i8.p1.1" class="ltx_p">Learning rate: <math id="A3.I3.i8.p1.1.m1.1" class="ltx_Math" alttext="1\times 10^{-5}" display="inline"><semantics id="A3.I3.i8.p1.1.m1.1a"><mrow id="A3.I3.i8.p1.1.m1.1.1" xref="A3.I3.i8.p1.1.m1.1.1.cmml"><mn id="A3.I3.i8.p1.1.m1.1.1.2" xref="A3.I3.i8.p1.1.m1.1.1.2.cmml">1</mn><mo lspace="0.222em" rspace="0.222em" id="A3.I3.i8.p1.1.m1.1.1.1" xref="A3.I3.i8.p1.1.m1.1.1.1.cmml">×</mo><msup id="A3.I3.i8.p1.1.m1.1.1.3" xref="A3.I3.i8.p1.1.m1.1.1.3.cmml"><mn id="A3.I3.i8.p1.1.m1.1.1.3.2" xref="A3.I3.i8.p1.1.m1.1.1.3.2.cmml">10</mn><mrow id="A3.I3.i8.p1.1.m1.1.1.3.3" xref="A3.I3.i8.p1.1.m1.1.1.3.3.cmml"><mo id="A3.I3.i8.p1.1.m1.1.1.3.3a" xref="A3.I3.i8.p1.1.m1.1.1.3.3.cmml">−</mo><mn id="A3.I3.i8.p1.1.m1.1.1.3.3.2" xref="A3.I3.i8.p1.1.m1.1.1.3.3.2.cmml">5</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="A3.I3.i8.p1.1.m1.1b"><apply id="A3.I3.i8.p1.1.m1.1.1.cmml" xref="A3.I3.i8.p1.1.m1.1.1"><times id="A3.I3.i8.p1.1.m1.1.1.1.cmml" xref="A3.I3.i8.p1.1.m1.1.1.1"></times><cn type="integer" id="A3.I3.i8.p1.1.m1.1.1.2.cmml" xref="A3.I3.i8.p1.1.m1.1.1.2">1</cn><apply id="A3.I3.i8.p1.1.m1.1.1.3.cmml" xref="A3.I3.i8.p1.1.m1.1.1.3"><csymbol cd="ambiguous" id="A3.I3.i8.p1.1.m1.1.1.3.1.cmml" xref="A3.I3.i8.p1.1.m1.1.1.3">superscript</csymbol><cn type="integer" id="A3.I3.i8.p1.1.m1.1.1.3.2.cmml" xref="A3.I3.i8.p1.1.m1.1.1.3.2">10</cn><apply id="A3.I3.i8.p1.1.m1.1.1.3.3.cmml" xref="A3.I3.i8.p1.1.m1.1.1.3.3"><minus id="A3.I3.i8.p1.1.m1.1.1.3.3.1.cmml" xref="A3.I3.i8.p1.1.m1.1.1.3.3"></minus><cn type="integer" id="A3.I3.i8.p1.1.m1.1.1.3.3.2.cmml" xref="A3.I3.i8.p1.1.m1.1.1.3.3.2">5</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.I3.i8.p1.1.m1.1c">1\times 10^{-5}</annotation></semantics></math></p>
</div>
</li>
</ul>
</div>
</section>
<section id="A3.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">C.4 </span>Stream of Search + STaR</h3>

<div id="A3.SS4.p1" class="ltx_para ltx_noindent">
<p id="A3.SS4.p1.1" class="ltx_p">For STaR, we use a temperature of 0.8 to sample trajectories for problems in the training set. We filter search trajectories so that only correct trajectories are used to finetune the next iteration. Similar to <cite class="ltx_cite ltx_citemacro_citet">Zelikman et al. (<a href="#bib.bib27" title="" class="ltx_ref">2022</a>)</cite> we reset the model to the base SoS model at each iteration of improvement.</p>
</div>
<div id="A3.SS4.p2" class="ltx_para ltx_noindent">
<ul id="A3.I4" class="ltx_itemize">
<li id="A3.I4.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A3.I4.i1.p1" class="ltx_para">
<p id="A3.I4.i1.p1.1" class="ltx_p">Number of training examples: <math id="A3.I4.i1.p1.1.m1.1" class="ltx_Math" alttext="1\times 10^{5}" display="inline"><semantics id="A3.I4.i1.p1.1.m1.1a"><mrow id="A3.I4.i1.p1.1.m1.1.1" xref="A3.I4.i1.p1.1.m1.1.1.cmml"><mn id="A3.I4.i1.p1.1.m1.1.1.2" xref="A3.I4.i1.p1.1.m1.1.1.2.cmml">1</mn><mo lspace="0.222em" rspace="0.222em" id="A3.I4.i1.p1.1.m1.1.1.1" xref="A3.I4.i1.p1.1.m1.1.1.1.cmml">×</mo><msup id="A3.I4.i1.p1.1.m1.1.1.3" xref="A3.I4.i1.p1.1.m1.1.1.3.cmml"><mn id="A3.I4.i1.p1.1.m1.1.1.3.2" xref="A3.I4.i1.p1.1.m1.1.1.3.2.cmml">10</mn><mn id="A3.I4.i1.p1.1.m1.1.1.3.3" xref="A3.I4.i1.p1.1.m1.1.1.3.3.cmml">5</mn></msup></mrow><annotation-xml encoding="MathML-Content" id="A3.I4.i1.p1.1.m1.1b"><apply id="A3.I4.i1.p1.1.m1.1.1.cmml" xref="A3.I4.i1.p1.1.m1.1.1"><times id="A3.I4.i1.p1.1.m1.1.1.1.cmml" xref="A3.I4.i1.p1.1.m1.1.1.1"></times><cn type="integer" id="A3.I4.i1.p1.1.m1.1.1.2.cmml" xref="A3.I4.i1.p1.1.m1.1.1.2">1</cn><apply id="A3.I4.i1.p1.1.m1.1.1.3.cmml" xref="A3.I4.i1.p1.1.m1.1.1.3"><csymbol cd="ambiguous" id="A3.I4.i1.p1.1.m1.1.1.3.1.cmml" xref="A3.I4.i1.p1.1.m1.1.1.3">superscript</csymbol><cn type="integer" id="A3.I4.i1.p1.1.m1.1.1.3.2.cmml" xref="A3.I4.i1.p1.1.m1.1.1.3.2">10</cn><cn type="integer" id="A3.I4.i1.p1.1.m1.1.1.3.3.cmml" xref="A3.I4.i1.p1.1.m1.1.1.3.3">5</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.I4.i1.p1.1.m1.1c">1\times 10^{5}</annotation></semantics></math></p>
</div>
</li>
<li id="A3.I4.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A3.I4.i2.p1" class="ltx_para">
<p id="A3.I4.i2.p1.1" class="ltx_p">Batch size: 24</p>
</div>
</li>
<li id="A3.I4.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A3.I4.i3.p1" class="ltx_para">
<p id="A3.I4.i3.p1.1" class="ltx_p">Gradient accumulation steps: 1</p>
</div>
</li>
<li id="A3.I4.i4" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A3.I4.i4.p1" class="ltx_para">
<p id="A3.I4.i4.p1.1" class="ltx_p">Number of training steps: 20,000</p>
</div>
</li>
<li id="A3.I4.i5" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A3.I4.i5.p1" class="ltx_para">
<p id="A3.I4.i5.p1.1" class="ltx_p">Weight decay: 0.01</p>
</div>
</li>
<li id="A3.I4.i6" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A3.I4.i6.p1" class="ltx_para">
<p id="A3.I4.i6.p1.1" class="ltx_p">Warmup steps: 100</p>
</div>
</li>
<li id="A3.I4.i7" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A3.I4.i7.p1" class="ltx_para">
<p id="A3.I4.i7.p1.1" class="ltx_p">Learning rate scheduler type: cosine</p>
</div>
</li>
<li id="A3.I4.i8" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A3.I4.i8.p1" class="ltx_para ltx_noindent">
<p id="A3.I4.i8.p1.1" class="ltx_p">Learning rate: <math id="A3.I4.i8.p1.1.m1.1" class="ltx_Math" alttext="1\times 10^{-5}" display="inline"><semantics id="A3.I4.i8.p1.1.m1.1a"><mrow id="A3.I4.i8.p1.1.m1.1.1" xref="A3.I4.i8.p1.1.m1.1.1.cmml"><mn id="A3.I4.i8.p1.1.m1.1.1.2" xref="A3.I4.i8.p1.1.m1.1.1.2.cmml">1</mn><mo lspace="0.222em" rspace="0.222em" id="A3.I4.i8.p1.1.m1.1.1.1" xref="A3.I4.i8.p1.1.m1.1.1.1.cmml">×</mo><msup id="A3.I4.i8.p1.1.m1.1.1.3" xref="A3.I4.i8.p1.1.m1.1.1.3.cmml"><mn id="A3.I4.i8.p1.1.m1.1.1.3.2" xref="A3.I4.i8.p1.1.m1.1.1.3.2.cmml">10</mn><mrow id="A3.I4.i8.p1.1.m1.1.1.3.3" xref="A3.I4.i8.p1.1.m1.1.1.3.3.cmml"><mo id="A3.I4.i8.p1.1.m1.1.1.3.3a" xref="A3.I4.i8.p1.1.m1.1.1.3.3.cmml">−</mo><mn id="A3.I4.i8.p1.1.m1.1.1.3.3.2" xref="A3.I4.i8.p1.1.m1.1.1.3.3.2.cmml">5</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="A3.I4.i8.p1.1.m1.1b"><apply id="A3.I4.i8.p1.1.m1.1.1.cmml" xref="A3.I4.i8.p1.1.m1.1.1"><times id="A3.I4.i8.p1.1.m1.1.1.1.cmml" xref="A3.I4.i8.p1.1.m1.1.1.1"></times><cn type="integer" id="A3.I4.i8.p1.1.m1.1.1.2.cmml" xref="A3.I4.i8.p1.1.m1.1.1.2">1</cn><apply id="A3.I4.i8.p1.1.m1.1.1.3.cmml" xref="A3.I4.i8.p1.1.m1.1.1.3"><csymbol cd="ambiguous" id="A3.I4.i8.p1.1.m1.1.1.3.1.cmml" xref="A3.I4.i8.p1.1.m1.1.1.3">superscript</csymbol><cn type="integer" id="A3.I4.i8.p1.1.m1.1.1.3.2.cmml" xref="A3.I4.i8.p1.1.m1.1.1.3.2">10</cn><apply id="A3.I4.i8.p1.1.m1.1.1.3.3.cmml" xref="A3.I4.i8.p1.1.m1.1.1.3.3"><minus id="A3.I4.i8.p1.1.m1.1.1.3.3.1.cmml" xref="A3.I4.i8.p1.1.m1.1.1.3.3"></minus><cn type="integer" id="A3.I4.i8.p1.1.m1.1.1.3.3.2.cmml" xref="A3.I4.i8.p1.1.m1.1.1.3.3.2">5</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.I4.i8.p1.1.m1.1c">1\times 10^{-5}</annotation></semantics></math></p>
</div>
</li>
</ul>
</div>
</section>
<section id="A3.SS5" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">C.5 </span>Stream of Search + APA</h3>

<div id="A3.SS5.p1" class="ltx_para ltx_noindent">
<ul id="A3.I5" class="ltx_itemize">
<li id="A3.I5.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A3.I5.i1.p1" class="ltx_para">
<p id="A3.I5.i1.p1.1" class="ltx_p">Number of rollouts: 32</p>
</div>
</li>
<li id="A3.I5.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A3.I5.i2.p1" class="ltx_para">
<p id="A3.I5.i2.p1.1" class="ltx_p">Temperature for sampling: 1.0</p>
</div>
</li>
<li id="A3.I5.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A3.I5.i3.p1" class="ltx_para">
<p id="A3.I5.i3.p1.1" class="ltx_p">Batch Size: 8</p>
</div>
</li>
<li id="A3.I5.i4" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A3.I5.i4.p1" class="ltx_para">
<p id="A3.I5.i4.p1.1" class="ltx_p">Online Epochs: 2</p>
</div>
</li>
<li id="A3.I5.i5" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A3.I5.i5.p1" class="ltx_para">
<p id="A3.I5.i5.p1.1" class="ltx_p">Gradient accumulation steps: 1</p>
</div>
</li>
<li id="A3.I5.i6" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A3.I5.i6.p1" class="ltx_para">
<p id="A3.I5.i6.p1.1" class="ltx_p">Scaling of reward: None</p>
</div>
</li>
<li id="A3.I5.i7" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A3.I5.i7.p1" class="ltx_para">
<p id="A3.I5.i7.p1.1" class="ltx_p">Value Loss Coefficient: 10</p>
</div>
</li>
<li id="A3.I5.i8" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A3.I5.i8.p1" class="ltx_para">
<p id="A3.I5.i8.p1.1" class="ltx_p"><math id="A3.I5.i8.p1.1.m1.1" class="ltx_Math" alttext="\gamma" display="inline"><semantics id="A3.I5.i8.p1.1.m1.1a"><mi id="A3.I5.i8.p1.1.m1.1.1" xref="A3.I5.i8.p1.1.m1.1.1.cmml">γ</mi><annotation-xml encoding="MathML-Content" id="A3.I5.i8.p1.1.m1.1b"><ci id="A3.I5.i8.p1.1.m1.1.1.cmml" xref="A3.I5.i8.p1.1.m1.1.1">𝛾</ci></annotation-xml><annotation encoding="application/x-tex" id="A3.I5.i8.p1.1.m1.1c">\gamma</annotation></semantics></math>: 1.0</p>
</div>
</li>
<li id="A3.I5.i9" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A3.I5.i9.p1" class="ltx_para">
<p id="A3.I5.i9.p1.1" class="ltx_p"><math id="A3.I5.i9.p1.1.m1.1" class="ltx_Math" alttext="\lambda" display="inline"><semantics id="A3.I5.i9.p1.1.m1.1a"><mi id="A3.I5.i9.p1.1.m1.1.1" xref="A3.I5.i9.p1.1.m1.1.1.cmml">λ</mi><annotation-xml encoding="MathML-Content" id="A3.I5.i9.p1.1.m1.1b"><ci id="A3.I5.i9.p1.1.m1.1.1.cmml" xref="A3.I5.i9.p1.1.m1.1.1">𝜆</ci></annotation-xml><annotation encoding="application/x-tex" id="A3.I5.i9.p1.1.m1.1c">\lambda</annotation></semantics></math>: 2.0</p>
</div>
</li>
<li id="A3.I5.i10" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A3.I5.i10.p1" class="ltx_para">
<p id="A3.I5.i10.p1.1" class="ltx_p">Learning rate scheduler type: None</p>
</div>
</li>
<li id="A3.I5.i11" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A3.I5.i11.p1" class="ltx_para ltx_noindent">
<p id="A3.I5.i11.p1.1" class="ltx_p">Learning rate: <math id="A3.I5.i11.p1.1.m1.1" class="ltx_Math" alttext="1\times 10^{-6}" display="inline"><semantics id="A3.I5.i11.p1.1.m1.1a"><mrow id="A3.I5.i11.p1.1.m1.1.1" xref="A3.I5.i11.p1.1.m1.1.1.cmml"><mn id="A3.I5.i11.p1.1.m1.1.1.2" xref="A3.I5.i11.p1.1.m1.1.1.2.cmml">1</mn><mo lspace="0.222em" rspace="0.222em" id="A3.I5.i11.p1.1.m1.1.1.1" xref="A3.I5.i11.p1.1.m1.1.1.1.cmml">×</mo><msup id="A3.I5.i11.p1.1.m1.1.1.3" xref="A3.I5.i11.p1.1.m1.1.1.3.cmml"><mn id="A3.I5.i11.p1.1.m1.1.1.3.2" xref="A3.I5.i11.p1.1.m1.1.1.3.2.cmml">10</mn><mrow id="A3.I5.i11.p1.1.m1.1.1.3.3" xref="A3.I5.i11.p1.1.m1.1.1.3.3.cmml"><mo id="A3.I5.i11.p1.1.m1.1.1.3.3a" xref="A3.I5.i11.p1.1.m1.1.1.3.3.cmml">−</mo><mn id="A3.I5.i11.p1.1.m1.1.1.3.3.2" xref="A3.I5.i11.p1.1.m1.1.1.3.3.2.cmml">6</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="A3.I5.i11.p1.1.m1.1b"><apply id="A3.I5.i11.p1.1.m1.1.1.cmml" xref="A3.I5.i11.p1.1.m1.1.1"><times id="A3.I5.i11.p1.1.m1.1.1.1.cmml" xref="A3.I5.i11.p1.1.m1.1.1.1"></times><cn type="integer" id="A3.I5.i11.p1.1.m1.1.1.2.cmml" xref="A3.I5.i11.p1.1.m1.1.1.2">1</cn><apply id="A3.I5.i11.p1.1.m1.1.1.3.cmml" xref="A3.I5.i11.p1.1.m1.1.1.3"><csymbol cd="ambiguous" id="A3.I5.i11.p1.1.m1.1.1.3.1.cmml" xref="A3.I5.i11.p1.1.m1.1.1.3">superscript</csymbol><cn type="integer" id="A3.I5.i11.p1.1.m1.1.1.3.2.cmml" xref="A3.I5.i11.p1.1.m1.1.1.3.2">10</cn><apply id="A3.I5.i11.p1.1.m1.1.1.3.3.cmml" xref="A3.I5.i11.p1.1.m1.1.1.3.3"><minus id="A3.I5.i11.p1.1.m1.1.1.3.3.1.cmml" xref="A3.I5.i11.p1.1.m1.1.1.3.3"></minus><cn type="integer" id="A3.I5.i11.p1.1.m1.1.1.3.3.2.cmml" xref="A3.I5.i11.p1.1.m1.1.1.3.3.2">6</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.I5.i11.p1.1.m1.1c">1\times 10^{-6}</annotation></semantics></math></p>
</div>
</li>
</ul>
</div>
</section>
</section>
<section id="A4" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix D </span>Analysis of generated trajectories</h2>

<div id="A4.p1" class="ltx_para">
<p id="A4.p1.1" class="ltx_p">In each evaluation of the trained model, we prompt the model to generate trajectories on 1) 10,000 tests with seen targets and unseen sets of input numbers, and 2) 10,000 tests with unseen or held-out targets. On top of improving accuracy, both policy improvement methods also reduced the average number of states or nodes explored before reaching the goal in correct trajectories, compared to the correct trajectories generated by the SoS LM. Under our framework, the number of states explored is correlated with the length of generation. This demonstrates that the policy improvement methods increase the efficiency of search.</p>
</div>
<figure id="A4.T1" class="ltx_table">
<table id="A4.T1.6.6" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A4.T1.6.6.7.1" class="ltx_tr">
<th id="A4.T1.6.6.7.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_t" style="padding-top:2.5pt;padding-bottom:2.5pt;"><span id="A4.T1.6.6.7.1.1.1" class="ltx_text ltx_font_bold">Model</span></th>
<th id="A4.T1.6.6.7.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-top:2.5pt;padding-bottom:2.5pt;"><span id="A4.T1.6.6.7.1.2.1" class="ltx_text ltx_font_bold">Target type</span></th>
<th id="A4.T1.6.6.7.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-top:2.5pt;padding-bottom:2.5pt;"><span id="A4.T1.6.6.7.1.3.1" class="ltx_text ltx_font_bold">Accuracy</span></th>
<th id="A4.T1.6.6.7.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-top:2.5pt;padding-bottom:2.5pt;"><span id="A4.T1.6.6.7.1.4.1" class="ltx_text ltx_font_bold">Num. states explored in correct trajectories</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A4.T1.1.1.1" class="ltx_tr">
<td id="A4.T1.1.1.1.2" class="ltx_td ltx_align_left ltx_border_t" style="padding-top:2.5pt;padding-bottom:2.5pt;">SoS</td>
<td id="A4.T1.1.1.1.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:2.5pt;padding-bottom:2.5pt;">Seen</td>
<td id="A4.T1.1.1.1.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:2.5pt;padding-bottom:2.5pt;">0.5127</td>
<td id="A4.T1.1.1.1.1" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:2.5pt;padding-bottom:2.5pt;"><math id="A4.T1.1.1.1.1.m1.1" class="ltx_Math" alttext="75.043\pm 0.640" display="inline"><semantics id="A4.T1.1.1.1.1.m1.1a"><mrow id="A4.T1.1.1.1.1.m1.1.1" xref="A4.T1.1.1.1.1.m1.1.1.cmml"><mn id="A4.T1.1.1.1.1.m1.1.1.2" xref="A4.T1.1.1.1.1.m1.1.1.2.cmml">75.043</mn><mo id="A4.T1.1.1.1.1.m1.1.1.1" xref="A4.T1.1.1.1.1.m1.1.1.1.cmml">±</mo><mn id="A4.T1.1.1.1.1.m1.1.1.3" xref="A4.T1.1.1.1.1.m1.1.1.3.cmml">0.640</mn></mrow><annotation-xml encoding="MathML-Content" id="A4.T1.1.1.1.1.m1.1b"><apply id="A4.T1.1.1.1.1.m1.1.1.cmml" xref="A4.T1.1.1.1.1.m1.1.1"><csymbol cd="latexml" id="A4.T1.1.1.1.1.m1.1.1.1.cmml" xref="A4.T1.1.1.1.1.m1.1.1.1">plus-or-minus</csymbol><cn type="float" id="A4.T1.1.1.1.1.m1.1.1.2.cmml" xref="A4.T1.1.1.1.1.m1.1.1.2">75.043</cn><cn type="float" id="A4.T1.1.1.1.1.m1.1.1.3.cmml" xref="A4.T1.1.1.1.1.m1.1.1.3">0.640</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.T1.1.1.1.1.m1.1c">75.043\pm 0.640</annotation></semantics></math></td>
</tr>
<tr id="A4.T1.2.2.2" class="ltx_tr">
<td id="A4.T1.2.2.2.2" class="ltx_td ltx_align_left" style="padding-top:2.5pt;padding-bottom:2.5pt;">SoS</td>
<td id="A4.T1.2.2.2.3" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;">Unseen</td>
<td id="A4.T1.2.2.2.4" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;">0.4900</td>
<td id="A4.T1.2.2.2.1" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;"><math id="A4.T1.2.2.2.1.m1.1" class="ltx_Math" alttext="81.747\pm 0.643" display="inline"><semantics id="A4.T1.2.2.2.1.m1.1a"><mrow id="A4.T1.2.2.2.1.m1.1.1" xref="A4.T1.2.2.2.1.m1.1.1.cmml"><mn id="A4.T1.2.2.2.1.m1.1.1.2" xref="A4.T1.2.2.2.1.m1.1.1.2.cmml">81.747</mn><mo id="A4.T1.2.2.2.1.m1.1.1.1" xref="A4.T1.2.2.2.1.m1.1.1.1.cmml">±</mo><mn id="A4.T1.2.2.2.1.m1.1.1.3" xref="A4.T1.2.2.2.1.m1.1.1.3.cmml">0.643</mn></mrow><annotation-xml encoding="MathML-Content" id="A4.T1.2.2.2.1.m1.1b"><apply id="A4.T1.2.2.2.1.m1.1.1.cmml" xref="A4.T1.2.2.2.1.m1.1.1"><csymbol cd="latexml" id="A4.T1.2.2.2.1.m1.1.1.1.cmml" xref="A4.T1.2.2.2.1.m1.1.1.1">plus-or-minus</csymbol><cn type="float" id="A4.T1.2.2.2.1.m1.1.1.2.cmml" xref="A4.T1.2.2.2.1.m1.1.1.2">81.747</cn><cn type="float" id="A4.T1.2.2.2.1.m1.1.1.3.cmml" xref="A4.T1.2.2.2.1.m1.1.1.3">0.643</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.T1.2.2.2.1.m1.1c">81.747\pm 0.643</annotation></semantics></math></td>
</tr>
<tr id="A4.T1.3.3.3" class="ltx_tr">
<td id="A4.T1.3.3.3.2" class="ltx_td ltx_align_left" style="padding-top:2.5pt;padding-bottom:2.5pt;">STaR</td>
<td id="A4.T1.3.3.3.3" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;">Seen</td>
<td id="A4.T1.3.3.3.4" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;">0.5559</td>
<td id="A4.T1.3.3.3.1" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;"><math id="A4.T1.3.3.3.1.m1.1" class="ltx_Math" alttext="66.176\pm 0.641" display="inline"><semantics id="A4.T1.3.3.3.1.m1.1a"><mrow id="A4.T1.3.3.3.1.m1.1.1" xref="A4.T1.3.3.3.1.m1.1.1.cmml"><mn id="A4.T1.3.3.3.1.m1.1.1.2" xref="A4.T1.3.3.3.1.m1.1.1.2.cmml">66.176</mn><mo id="A4.T1.3.3.3.1.m1.1.1.1" xref="A4.T1.3.3.3.1.m1.1.1.1.cmml">±</mo><mn id="A4.T1.3.3.3.1.m1.1.1.3" xref="A4.T1.3.3.3.1.m1.1.1.3.cmml">0.641</mn></mrow><annotation-xml encoding="MathML-Content" id="A4.T1.3.3.3.1.m1.1b"><apply id="A4.T1.3.3.3.1.m1.1.1.cmml" xref="A4.T1.3.3.3.1.m1.1.1"><csymbol cd="latexml" id="A4.T1.3.3.3.1.m1.1.1.1.cmml" xref="A4.T1.3.3.3.1.m1.1.1.1">plus-or-minus</csymbol><cn type="float" id="A4.T1.3.3.3.1.m1.1.1.2.cmml" xref="A4.T1.3.3.3.1.m1.1.1.2">66.176</cn><cn type="float" id="A4.T1.3.3.3.1.m1.1.1.3.cmml" xref="A4.T1.3.3.3.1.m1.1.1.3">0.641</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.T1.3.3.3.1.m1.1c">66.176\pm 0.641</annotation></semantics></math></td>
</tr>
<tr id="A4.T1.4.4.4" class="ltx_tr">
<td id="A4.T1.4.4.4.2" class="ltx_td ltx_align_left" style="padding-top:2.5pt;padding-bottom:2.5pt;">STaR</td>
<td id="A4.T1.4.4.4.3" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;">Unseen</td>
<td id="A4.T1.4.4.4.4" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;">0.5334</td>
<td id="A4.T1.4.4.4.1" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;"><math id="A4.T1.4.4.4.1.m1.1" class="ltx_Math" alttext="72.252\pm 0.647" display="inline"><semantics id="A4.T1.4.4.4.1.m1.1a"><mrow id="A4.T1.4.4.4.1.m1.1.1" xref="A4.T1.4.4.4.1.m1.1.1.cmml"><mn id="A4.T1.4.4.4.1.m1.1.1.2" xref="A4.T1.4.4.4.1.m1.1.1.2.cmml">72.252</mn><mo id="A4.T1.4.4.4.1.m1.1.1.1" xref="A4.T1.4.4.4.1.m1.1.1.1.cmml">±</mo><mn id="A4.T1.4.4.4.1.m1.1.1.3" xref="A4.T1.4.4.4.1.m1.1.1.3.cmml">0.647</mn></mrow><annotation-xml encoding="MathML-Content" id="A4.T1.4.4.4.1.m1.1b"><apply id="A4.T1.4.4.4.1.m1.1.1.cmml" xref="A4.T1.4.4.4.1.m1.1.1"><csymbol cd="latexml" id="A4.T1.4.4.4.1.m1.1.1.1.cmml" xref="A4.T1.4.4.4.1.m1.1.1.1">plus-or-minus</csymbol><cn type="float" id="A4.T1.4.4.4.1.m1.1.1.2.cmml" xref="A4.T1.4.4.4.1.m1.1.1.2">72.252</cn><cn type="float" id="A4.T1.4.4.4.1.m1.1.1.3.cmml" xref="A4.T1.4.4.4.1.m1.1.1.3">0.647</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.T1.4.4.4.1.m1.1c">72.252\pm 0.647</annotation></semantics></math></td>
</tr>
<tr id="A4.T1.5.5.5" class="ltx_tr">
<td id="A4.T1.5.5.5.2" class="ltx_td ltx_align_left" style="padding-top:2.5pt;padding-bottom:2.5pt;">APA</td>
<td id="A4.T1.5.5.5.3" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;">Seen</td>
<td id="A4.T1.5.5.5.4" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;">0.5652</td>
<td id="A4.T1.5.5.5.1" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;"><math id="A4.T1.5.5.5.1.m1.1" class="ltx_Math" alttext="64.733\pm 0.652" display="inline"><semantics id="A4.T1.5.5.5.1.m1.1a"><mrow id="A4.T1.5.5.5.1.m1.1.1" xref="A4.T1.5.5.5.1.m1.1.1.cmml"><mn id="A4.T1.5.5.5.1.m1.1.1.2" xref="A4.T1.5.5.5.1.m1.1.1.2.cmml">64.733</mn><mo id="A4.T1.5.5.5.1.m1.1.1.1" xref="A4.T1.5.5.5.1.m1.1.1.1.cmml">±</mo><mn id="A4.T1.5.5.5.1.m1.1.1.3" xref="A4.T1.5.5.5.1.m1.1.1.3.cmml">0.652</mn></mrow><annotation-xml encoding="MathML-Content" id="A4.T1.5.5.5.1.m1.1b"><apply id="A4.T1.5.5.5.1.m1.1.1.cmml" xref="A4.T1.5.5.5.1.m1.1.1"><csymbol cd="latexml" id="A4.T1.5.5.5.1.m1.1.1.1.cmml" xref="A4.T1.5.5.5.1.m1.1.1.1">plus-or-minus</csymbol><cn type="float" id="A4.T1.5.5.5.1.m1.1.1.2.cmml" xref="A4.T1.5.5.5.1.m1.1.1.2">64.733</cn><cn type="float" id="A4.T1.5.5.5.1.m1.1.1.3.cmml" xref="A4.T1.5.5.5.1.m1.1.1.3">0.652</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.T1.5.5.5.1.m1.1c">64.733\pm 0.652</annotation></semantics></math></td>
</tr>
<tr id="A4.T1.6.6.6" class="ltx_tr">
<td id="A4.T1.6.6.6.2" class="ltx_td ltx_align_left ltx_border_b" style="padding-top:2.5pt;padding-bottom:2.5pt;">APA</td>
<td id="A4.T1.6.6.6.3" class="ltx_td ltx_align_center ltx_border_b" style="padding-top:2.5pt;padding-bottom:2.5pt;">Unseen</td>
<td id="A4.T1.6.6.6.4" class="ltx_td ltx_align_center ltx_border_b" style="padding-top:2.5pt;padding-bottom:2.5pt;">0.5423</td>
<td id="A4.T1.6.6.6.1" class="ltx_td ltx_align_center ltx_border_b" style="padding-top:2.5pt;padding-bottom:2.5pt;"><math id="A4.T1.6.6.6.1.m1.1" class="ltx_Math" alttext="70.849\pm 0.655" display="inline"><semantics id="A4.T1.6.6.6.1.m1.1a"><mrow id="A4.T1.6.6.6.1.m1.1.1" xref="A4.T1.6.6.6.1.m1.1.1.cmml"><mn id="A4.T1.6.6.6.1.m1.1.1.2" xref="A4.T1.6.6.6.1.m1.1.1.2.cmml">70.849</mn><mo id="A4.T1.6.6.6.1.m1.1.1.1" xref="A4.T1.6.6.6.1.m1.1.1.1.cmml">±</mo><mn id="A4.T1.6.6.6.1.m1.1.1.3" xref="A4.T1.6.6.6.1.m1.1.1.3.cmml">0.655</mn></mrow><annotation-xml encoding="MathML-Content" id="A4.T1.6.6.6.1.m1.1b"><apply id="A4.T1.6.6.6.1.m1.1.1.cmml" xref="A4.T1.6.6.6.1.m1.1.1"><csymbol cd="latexml" id="A4.T1.6.6.6.1.m1.1.1.1.cmml" xref="A4.T1.6.6.6.1.m1.1.1.1">plus-or-minus</csymbol><cn type="float" id="A4.T1.6.6.6.1.m1.1.1.2.cmml" xref="A4.T1.6.6.6.1.m1.1.1.2">70.849</cn><cn type="float" id="A4.T1.6.6.6.1.m1.1.1.3.cmml" xref="A4.T1.6.6.6.1.m1.1.1.3">0.655</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.T1.6.6.6.1.m1.1c">70.849\pm 0.655</annotation></semantics></math></td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 1: </span>Details on statistics for each trained model (average over 10000 tests). Error margins represent 95% confidence intervals.</figcaption>
</figure>
<div id="A4.p2" class="ltx_para">
<p id="A4.p2.1" class="ltx_p">To further study the factors contributing to the correctness of trajectories, an error analysis was conducted for each one. Errors are grouped into four main types: 1) arithmetic (hallucinating invalid operations, dividing by zero, etc.), 2) formatting, 3) exploration (jumping to a node that has no parents), and 4) other (a child node has the wrong set of numbers associated with it). We can see that the APA training method suppressed arithmetic errors significantly and preserved a low rate of exploration errors. Meanwhile, the STaR method, while boosting accuracy, had an increased rate of exploration anomalies and other memory-based discrepancies.</p>
</div>
<figure id="A4.T2" class="ltx_table">
<table id="A4.T2.24.24" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A4.T2.24.24.25.1" class="ltx_tr">
<th id="A4.T2.24.24.25.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_t" style="padding-top:2.5pt;padding-bottom:2.5pt;"><span id="A4.T2.24.24.25.1.1.1" class="ltx_text ltx_font_bold">Model</span></th>
<th id="A4.T2.24.24.25.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-top:2.5pt;padding-bottom:2.5pt;"><span id="A4.T2.24.24.25.1.2.1" class="ltx_text ltx_font_bold">Target type</span></th>
<th id="A4.T2.24.24.25.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-top:2.5pt;padding-bottom:2.5pt;"><span id="A4.T2.24.24.25.1.3.1" class="ltx_text ltx_font_bold">Arithmetic</span></th>
<th id="A4.T2.24.24.25.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-top:2.5pt;padding-bottom:2.5pt;"><span id="A4.T2.24.24.25.1.4.1" class="ltx_text ltx_font_bold">Formatting</span></th>
<th id="A4.T2.24.24.25.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-top:2.5pt;padding-bottom:2.5pt;"><span id="A4.T2.24.24.25.1.5.1" class="ltx_text ltx_font_bold">Exploration</span></th>
<th id="A4.T2.24.24.25.1.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="padding-top:2.5pt;padding-bottom:2.5pt;"><span id="A4.T2.24.24.25.1.6.1" class="ltx_text ltx_font_bold">Other</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A4.T2.4.4.4" class="ltx_tr">
<th id="A4.T2.4.4.4.5" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" style="padding-top:2.5pt;padding-bottom:2.5pt;">SoS</th>
<td id="A4.T2.4.4.4.6" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:2.5pt;padding-bottom:2.5pt;">Seen</td>
<td id="A4.T2.1.1.1.1" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:2.5pt;padding-bottom:2.5pt;">2.247 <math id="A4.T2.1.1.1.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A4.T2.1.1.1.1.m1.1a"><mo id="A4.T2.1.1.1.1.m1.1.1" xref="A4.T2.1.1.1.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A4.T2.1.1.1.1.m1.1b"><csymbol cd="latexml" id="A4.T2.1.1.1.1.m1.1.1.cmml" xref="A4.T2.1.1.1.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A4.T2.1.1.1.1.m1.1c">\pm</annotation></semantics></math> 0.097</td>
<td id="A4.T2.2.2.2.2" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:2.5pt;padding-bottom:2.5pt;">0.014 <math id="A4.T2.2.2.2.2.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A4.T2.2.2.2.2.m1.1a"><mo id="A4.T2.2.2.2.2.m1.1.1" xref="A4.T2.2.2.2.2.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A4.T2.2.2.2.2.m1.1b"><csymbol cd="latexml" id="A4.T2.2.2.2.2.m1.1.1.cmml" xref="A4.T2.2.2.2.2.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A4.T2.2.2.2.2.m1.1c">\pm</annotation></semantics></math> 0.003</td>
<td id="A4.T2.3.3.3.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:2.5pt;padding-bottom:2.5pt;">0.008 <math id="A4.T2.3.3.3.3.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A4.T2.3.3.3.3.m1.1a"><mo id="A4.T2.3.3.3.3.m1.1.1" xref="A4.T2.3.3.3.3.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A4.T2.3.3.3.3.m1.1b"><csymbol cd="latexml" id="A4.T2.3.3.3.3.m1.1.1.cmml" xref="A4.T2.3.3.3.3.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A4.T2.3.3.3.3.m1.1c">\pm</annotation></semantics></math> 0.003</td>
<td id="A4.T2.4.4.4.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:2.5pt;padding-bottom:2.5pt;">0.018 <math id="A4.T2.4.4.4.4.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A4.T2.4.4.4.4.m1.1a"><mo id="A4.T2.4.4.4.4.m1.1.1" xref="A4.T2.4.4.4.4.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A4.T2.4.4.4.4.m1.1b"><csymbol cd="latexml" id="A4.T2.4.4.4.4.m1.1.1.cmml" xref="A4.T2.4.4.4.4.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A4.T2.4.4.4.4.m1.1c">\pm</annotation></semantics></math> 0.005</td>
</tr>
<tr id="A4.T2.8.8.8" class="ltx_tr">
<th id="A4.T2.8.8.8.5" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding-top:2.5pt;padding-bottom:2.5pt;">SoS</th>
<td id="A4.T2.8.8.8.6" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;">Unseen</td>
<td id="A4.T2.5.5.5.1" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;">2.547 <math id="A4.T2.5.5.5.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A4.T2.5.5.5.1.m1.1a"><mo id="A4.T2.5.5.5.1.m1.1.1" xref="A4.T2.5.5.5.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A4.T2.5.5.5.1.m1.1b"><csymbol cd="latexml" id="A4.T2.5.5.5.1.m1.1.1.cmml" xref="A4.T2.5.5.5.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A4.T2.5.5.5.1.m1.1c">\pm</annotation></semantics></math> 0.102</td>
<td id="A4.T2.6.6.6.2" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;">0.014 <math id="A4.T2.6.6.6.2.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A4.T2.6.6.6.2.m1.1a"><mo id="A4.T2.6.6.6.2.m1.1.1" xref="A4.T2.6.6.6.2.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A4.T2.6.6.6.2.m1.1b"><csymbol cd="latexml" id="A4.T2.6.6.6.2.m1.1.1.cmml" xref="A4.T2.6.6.6.2.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A4.T2.6.6.6.2.m1.1c">\pm</annotation></semantics></math> 0.002</td>
<td id="A4.T2.7.7.7.3" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;">0.005 <math id="A4.T2.7.7.7.3.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A4.T2.7.7.7.3.m1.1a"><mo id="A4.T2.7.7.7.3.m1.1.1" xref="A4.T2.7.7.7.3.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A4.T2.7.7.7.3.m1.1b"><csymbol cd="latexml" id="A4.T2.7.7.7.3.m1.1.1.cmml" xref="A4.T2.7.7.7.3.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A4.T2.7.7.7.3.m1.1c">\pm</annotation></semantics></math> 0.002</td>
<td id="A4.T2.8.8.8.4" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;">0.011 <math id="A4.T2.8.8.8.4.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A4.T2.8.8.8.4.m1.1a"><mo id="A4.T2.8.8.8.4.m1.1.1" xref="A4.T2.8.8.8.4.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A4.T2.8.8.8.4.m1.1b"><csymbol cd="latexml" id="A4.T2.8.8.8.4.m1.1.1.cmml" xref="A4.T2.8.8.8.4.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A4.T2.8.8.8.4.m1.1c">\pm</annotation></semantics></math> 0.003</td>
</tr>
<tr id="A4.T2.12.12.12" class="ltx_tr">
<th id="A4.T2.12.12.12.5" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding-top:2.5pt;padding-bottom:2.5pt;">STaR</th>
<td id="A4.T2.12.12.12.6" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;">Seen</td>
<td id="A4.T2.9.9.9.1" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;">1.927 <math id="A4.T2.9.9.9.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A4.T2.9.9.9.1.m1.1a"><mo id="A4.T2.9.9.9.1.m1.1.1" xref="A4.T2.9.9.9.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A4.T2.9.9.9.1.m1.1b"><csymbol cd="latexml" id="A4.T2.9.9.9.1.m1.1.1.cmml" xref="A4.T2.9.9.9.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A4.T2.9.9.9.1.m1.1c">\pm</annotation></semantics></math> 0.084</td>
<td id="A4.T2.10.10.10.2" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;">0.012 <math id="A4.T2.10.10.10.2.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A4.T2.10.10.10.2.m1.1a"><mo id="A4.T2.10.10.10.2.m1.1.1" xref="A4.T2.10.10.10.2.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A4.T2.10.10.10.2.m1.1b"><csymbol cd="latexml" id="A4.T2.10.10.10.2.m1.1.1.cmml" xref="A4.T2.10.10.10.2.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A4.T2.10.10.10.2.m1.1c">\pm</annotation></semantics></math> 0.002</td>
<td id="A4.T2.11.11.11.3" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;">0.024 <math id="A4.T2.11.11.11.3.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A4.T2.11.11.11.3.m1.1a"><mo id="A4.T2.11.11.11.3.m1.1.1" xref="A4.T2.11.11.11.3.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A4.T2.11.11.11.3.m1.1b"><csymbol cd="latexml" id="A4.T2.11.11.11.3.m1.1.1.cmml" xref="A4.T2.11.11.11.3.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A4.T2.11.11.11.3.m1.1c">\pm</annotation></semantics></math> 0.004</td>
<td id="A4.T2.12.12.12.4" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;">0.069 <math id="A4.T2.12.12.12.4.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A4.T2.12.12.12.4.m1.1a"><mo id="A4.T2.12.12.12.4.m1.1.1" xref="A4.T2.12.12.12.4.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A4.T2.12.12.12.4.m1.1b"><csymbol cd="latexml" id="A4.T2.12.12.12.4.m1.1.1.cmml" xref="A4.T2.12.12.12.4.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A4.T2.12.12.12.4.m1.1c">\pm</annotation></semantics></math> 0.009</td>
</tr>
<tr id="A4.T2.16.16.16" class="ltx_tr">
<th id="A4.T2.16.16.16.5" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding-top:2.5pt;padding-bottom:2.5pt;">STaR</th>
<td id="A4.T2.16.16.16.6" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;">Unseen</td>
<td id="A4.T2.13.13.13.1" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;">2.179 <math id="A4.T2.13.13.13.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A4.T2.13.13.13.1.m1.1a"><mo id="A4.T2.13.13.13.1.m1.1.1" xref="A4.T2.13.13.13.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A4.T2.13.13.13.1.m1.1b"><csymbol cd="latexml" id="A4.T2.13.13.13.1.m1.1.1.cmml" xref="A4.T2.13.13.13.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A4.T2.13.13.13.1.m1.1c">\pm</annotation></semantics></math> 0.084</td>
<td id="A4.T2.14.14.14.2" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;">0.013 <math id="A4.T2.14.14.14.2.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A4.T2.14.14.14.2.m1.1a"><mo id="A4.T2.14.14.14.2.m1.1.1" xref="A4.T2.14.14.14.2.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A4.T2.14.14.14.2.m1.1b"><csymbol cd="latexml" id="A4.T2.14.14.14.2.m1.1.1.cmml" xref="A4.T2.14.14.14.2.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A4.T2.14.14.14.2.m1.1c">\pm</annotation></semantics></math> 0.002</td>
<td id="A4.T2.15.15.15.3" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;">0.022 <math id="A4.T2.15.15.15.3.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A4.T2.15.15.15.3.m1.1a"><mo id="A4.T2.15.15.15.3.m1.1.1" xref="A4.T2.15.15.15.3.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A4.T2.15.15.15.3.m1.1b"><csymbol cd="latexml" id="A4.T2.15.15.15.3.m1.1.1.cmml" xref="A4.T2.15.15.15.3.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A4.T2.15.15.15.3.m1.1c">\pm</annotation></semantics></math> 0.004</td>
<td id="A4.T2.16.16.16.4" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;">0.074 <math id="A4.T2.16.16.16.4.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A4.T2.16.16.16.4.m1.1a"><mo id="A4.T2.16.16.16.4.m1.1.1" xref="A4.T2.16.16.16.4.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A4.T2.16.16.16.4.m1.1b"><csymbol cd="latexml" id="A4.T2.16.16.16.4.m1.1.1.cmml" xref="A4.T2.16.16.16.4.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A4.T2.16.16.16.4.m1.1c">\pm</annotation></semantics></math> 0.009</td>
</tr>
<tr id="A4.T2.20.20.20" class="ltx_tr">
<th id="A4.T2.20.20.20.5" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding-top:2.5pt;padding-bottom:2.5pt;">APA</th>
<td id="A4.T2.20.20.20.6" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;">Seen</td>
<td id="A4.T2.17.17.17.1" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;">0.668 <math id="A4.T2.17.17.17.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A4.T2.17.17.17.1.m1.1a"><mo id="A4.T2.17.17.17.1.m1.1.1" xref="A4.T2.17.17.17.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A4.T2.17.17.17.1.m1.1b"><csymbol cd="latexml" id="A4.T2.17.17.17.1.m1.1.1.cmml" xref="A4.T2.17.17.17.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A4.T2.17.17.17.1.m1.1c">\pm</annotation></semantics></math> 0.036</td>
<td id="A4.T2.18.18.18.2" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;">0.014 <math id="A4.T2.18.18.18.2.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A4.T2.18.18.18.2.m1.1a"><mo id="A4.T2.18.18.18.2.m1.1.1" xref="A4.T2.18.18.18.2.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A4.T2.18.18.18.2.m1.1b"><csymbol cd="latexml" id="A4.T2.18.18.18.2.m1.1.1.cmml" xref="A4.T2.18.18.18.2.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A4.T2.18.18.18.2.m1.1c">\pm</annotation></semantics></math> 0.003</td>
<td id="A4.T2.19.19.19.3" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;">0.008 <math id="A4.T2.19.19.19.3.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A4.T2.19.19.19.3.m1.1a"><mo id="A4.T2.19.19.19.3.m1.1.1" xref="A4.T2.19.19.19.3.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A4.T2.19.19.19.3.m1.1b"><csymbol cd="latexml" id="A4.T2.19.19.19.3.m1.1.1.cmml" xref="A4.T2.19.19.19.3.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A4.T2.19.19.19.3.m1.1c">\pm</annotation></semantics></math> 0.003</td>
<td id="A4.T2.20.20.20.4" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;">0.022 <math id="A4.T2.20.20.20.4.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A4.T2.20.20.20.4.m1.1a"><mo id="A4.T2.20.20.20.4.m1.1.1" xref="A4.T2.20.20.20.4.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A4.T2.20.20.20.4.m1.1b"><csymbol cd="latexml" id="A4.T2.20.20.20.4.m1.1.1.cmml" xref="A4.T2.20.20.20.4.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A4.T2.20.20.20.4.m1.1c">\pm</annotation></semantics></math> 0.004</td>
</tr>
<tr id="A4.T2.24.24.24" class="ltx_tr">
<th id="A4.T2.24.24.24.5" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_b" style="padding-top:2.5pt;padding-bottom:2.5pt;">APA</th>
<td id="A4.T2.24.24.24.6" class="ltx_td ltx_align_center ltx_border_b" style="padding-top:2.5pt;padding-bottom:2.5pt;">Unseen</td>
<td id="A4.T2.21.21.21.1" class="ltx_td ltx_align_center ltx_border_b" style="padding-top:2.5pt;padding-bottom:2.5pt;">0.785 <math id="A4.T2.21.21.21.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A4.T2.21.21.21.1.m1.1a"><mo id="A4.T2.21.21.21.1.m1.1.1" xref="A4.T2.21.21.21.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A4.T2.21.21.21.1.m1.1b"><csymbol cd="latexml" id="A4.T2.21.21.21.1.m1.1.1.cmml" xref="A4.T2.21.21.21.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A4.T2.21.21.21.1.m1.1c">\pm</annotation></semantics></math> 0.039</td>
<td id="A4.T2.22.22.22.2" class="ltx_td ltx_align_center ltx_border_b" style="padding-top:2.5pt;padding-bottom:2.5pt;">0.013 <math id="A4.T2.22.22.22.2.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A4.T2.22.22.22.2.m1.1a"><mo id="A4.T2.22.22.22.2.m1.1.1" xref="A4.T2.22.22.22.2.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A4.T2.22.22.22.2.m1.1b"><csymbol cd="latexml" id="A4.T2.22.22.22.2.m1.1.1.cmml" xref="A4.T2.22.22.22.2.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A4.T2.22.22.22.2.m1.1c">\pm</annotation></semantics></math> 0.002</td>
<td id="A4.T2.23.23.23.3" class="ltx_td ltx_align_center ltx_border_b" style="padding-top:2.5pt;padding-bottom:2.5pt;">0.006 <math id="A4.T2.23.23.23.3.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A4.T2.23.23.23.3.m1.1a"><mo id="A4.T2.23.23.23.3.m1.1.1" xref="A4.T2.23.23.23.3.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A4.T2.23.23.23.3.m1.1b"><csymbol cd="latexml" id="A4.T2.23.23.23.3.m1.1.1.cmml" xref="A4.T2.23.23.23.3.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A4.T2.23.23.23.3.m1.1c">\pm</annotation></semantics></math> 0.002</td>
<td id="A4.T2.24.24.24.4" class="ltx_td ltx_align_center ltx_border_b" style="padding-top:2.5pt;padding-bottom:2.5pt;">0.017 <math id="A4.T2.24.24.24.4.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A4.T2.24.24.24.4.m1.1a"><mo id="A4.T2.24.24.24.4.m1.1.1" xref="A4.T2.24.24.24.4.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A4.T2.24.24.24.4.m1.1b"><csymbol cd="latexml" id="A4.T2.24.24.24.4.m1.1.1.cmml" xref="A4.T2.24.24.24.4.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A4.T2.24.24.24.4.m1.1c">\pm</annotation></semantics></math> 0.003</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 2: </span>Error analysis for trained models (average number of each error per trajectory, 10000 tests). Error margins are 95% Confidence Intervals.</figcaption>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
</article>
</div>
<div class="ar5iv-footer"><a href="/html/2404.03682" class="ar5iv-nav-button ar5iv-nav-button-prev">◄</a>
    <a class="ar5iv-home-button" href="/"><img height="40" alt="ar5iv homepage" src="/assets/ar5iv.png"></a>
    <a href="/feeling_lucky" class="ar5iv-text-button">Feeling<br>lucky?</a>
    <a href="/log/2404.03683" class="ar5iv-text-button ar5iv-severity-error">Conversion<br>report</a>
    <a class="ar5iv-text-button" target="_blank" href="https://github.com/dginev/ar5iv/issues/new?template=improve-article--arxiv-id-.md&title=Improve+article+2404.03683">Report<br>an issue</a>
    <a href="https://arxiv.org/abs/2404.03683" class="ar5iv-text-button arxiv-ui-theme">View&nbsp;original<br>on&nbsp;arXiv</a><a href="/html/2404.03684" class="ar5iv-nav-button ar5iv-nav-button-next">►</a>
</div><footer class="ltx_page_footer">
<a class="ar5iv-toggle-color-scheme" href="javascript:toggleColorScheme()" title="Toggle ar5iv color scheme"><span class="color-scheme-icon"></span></a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/license" target="_blank">Copyright</a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/policies/privacy_policy" target="_blank">Privacy Policy</a>

<div class="ltx_page_logo">Generated  on Sun May  5 19:04:41 2024 by <a target="_blank" href="http://dlmf.nist.gov/LaTeXML/" class="ltx_LaTeXML_logo"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg==" alt="Mascot Sammy"></a>
</div></footer>
</div>

    <script>
      var canMathML = typeof(MathMLElement) == "function";
      if (!canMathML) {
        var body = document.querySelector("body");
        body.firstElementChild.setAttribute('style', 'opacity: 0;');
        var loading = document.createElement("div");
        loading.setAttribute("id", "mathjax-loading-spinner");
        var message = document.createElement("div");
        message.setAttribute("id", "mathjax-loading-message");
        message.innerText = "Typesetting Equations...";
        body.prepend(loading);
        body.prepend(message);

        var el = document.createElement("script");
        el.src = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js";
        document.querySelector("head").appendChild(el);

        window.MathJax = {
          startup: {
            pageReady: () => {
              return MathJax.startup.defaultPageReady().then(() => {
                body.removeChild(loading);
                body.removeChild(message);
                body.firstElementChild.removeAttribute('style');
              }); } } };
      }
    </script>
    <script>
    // Auxiliary function, building the preview feature when
    // an inline citation is clicked
    function clicked_cite(e) {
      e.preventDefault();
      let cite = this.closest('.ltx_cite');
      let next = cite.nextSibling;
      if (next && next.nodeType == Node.ELEMENT_NODE && next.getAttribute('class') == "ar5iv-bibitem-preview") {
        next.remove();
        return; }
      // Before adding a preview modal,
      // cleanup older previews, in case they're still open
      document.querySelectorAll('span.ar5iv-bibitem-preview').forEach(function(node) {
        node.remove();
      })

      // Create the preview
      preview = document.createElement('span');
      preview.setAttribute('class','ar5iv-bibitem-preview');
      let target = document.getElementById(this.getAttribute('href').slice(1));
      target.childNodes.forEach(function (child) {
        preview.append(child.cloneNode(true));
      });
      let close_x = document.createElement('button');
      close_x.setAttribute("aria-label","Close modal for bibliography item preview");
      close_x.textContent = "×";
      close_x.setAttribute('class', 'ar5iv-button-close-preview');
      close_x.setAttribute('onclick','this.parentNode.remove()');
      preview.append(close_x);
      preview.querySelectorAll('.ltx_tag_bibitem').forEach(function(node) {
        node.remove();
      });
      cite.parentNode.insertBefore(preview, cite.nextSibling);
      return;
    }
    // Global Document initialization:
    // - assign the preview feature to all inline citation links
    document.querySelectorAll(".ltx_cite .ltx_ref").forEach(function (link) {
      link.addEventListener("click", clicked_cite);
    });
    </script>
    </body>
</html>

2404.03683
Language models are rarely shown fruitful mistakes while training. They then struggle to look beyond the next token, suffering from a snowballing of errors and struggling to predict the consequence of their actions several steps ahead. In this paper, we show how language models can be taught to search by representing the process of search in language, as a flattened string — astream of search (SoS) . We propose a unified language for search that captures an array of different symbolic search strategies.
We demonstrate our approach using the simple yet difficult game of Countdown, where the goal is to combine input numbers with arithmetic operations to reach a target number.
We pretrain a transformer-based language model from scratch on a dataset of streams of search generated by heuristic solvers. We find that SoS pretraining increases search accuracy by 25% over models trained to predict only the optimal search trajectory.
We further finetune this model with two policy improvement methods: Advantage-Induced Policy Alignment (APA)  and Self-Taught Reasoner (STaR) . The finetuned SoS models solve 36% of previously unsolved problems, including problems that cannot be solved by any of the heuristic solvers.
Our results indicate that language models can learn to solve problems via search, self-improve to flexibly use different search strategies, and potentially discover new ones.111Code Available Here:https://github.com/kanishkg/stream-of-search
abstract
Stream of Search (SoS): Learning to Search in Language
Abstract
2404.03683
1IntroductionImagine, only ever seeing the right solutions to problems, never a mistake or recovery from it.
You might learn that problems must be solved in one clean pass, rather than through exploration and error.
Most data used to train language models (LMs)  only reflects the outcome of a decision making process, not the process itself. LMs never learn to make mistakes. They never learn to search, plan or backtrack. Complex decision-making and reasoning requires search. In this paper we explore the impact of training a LM on the search process, including mistakes, and then allowing them to self-improve.Figure 1:Overview of the Stream of Search (SoS)  framework. (a)  A search problem in Countdown is instantiated with input numbers and a target number. The input numbers need to be combined with simple arithmetic operations to get to the target. (b)  The Stream of Search dataset contains search trajectories generated by diverse search strategies, including exploration and backtracking. (c)  The language model is first trained on the SoS dataset and then iteratively improved using policy improvement techniques such as APA and STaR.Transformer-based auto-regressive models have been shown to struggle with planning.
Recent work has highlighted this weakness in autoregressive models by identifying two main issues: 1)  the snowballing of errors, where a single mistake can compound and lead to increasingly poor performance in subsequent steps, and 2)  a difficulty in ‘lookahead tasks’, where the model must predict the consequences of its actions several steps ahead. Both of these issues can be attributed to limited ability to search and backtrack. While recent efforts have combined language models with symbolic search algorithmsto mitigate some of these problems, they are limited — only supplementing language models during inference —they leave open the question of whether language models could effectively carry out search themselves. The most consequential outcome from learning to search could be during thetrainingitself. If language models can learn to search during training, they may be able to discover more flexible search strategies through self-improvement. This could lead to models that are better equipped to handle the challenges posed by error compounding and lookahead tasks.In this paper, we demonstrate that language models can be taught to search and backtrack in language, representing the process as a serialized string, a Stream of Search (SoS) . We systematize the different components of search, such as exploration, backtracking, and pruning in a unified language. We instantiate this unified language in the context of a search problem inspired by the game of Countdown, a generalized version of the game of 24. A problem consists of input numbers and a target number. The goal is to combine the input numbers with arithmetic operations to reach the target (seeFig. 1a) .
Countdown presents a challenging search problem due to its high branching factor and the need to efficiently navigate the combinatorial search space towards the target number.
We generate an initial training dataset of search trajectories or streams of search using different symbolic planners and simple heuristic functions, then train a language model on this diverse dataset. This can be contrasted with training on the optimal path once a solution is found, without the process of search and backtracking.
We find that the Stream of Search LM substantially outperforms models trained to predict the optimal steps. Moreover, we observe improvements in the search and planning abilities of the Stream of Search model when finetuned to optimize for correctness using Advantage-Induced Policy Alignmentand expert iteration with STaR.Our results indicate that transformer-based language models, when shown how to recover from mistakes and search through different options, can learn to solve problems by searching. More importantly, our results indicate that these models can self-improve to autonomously use different search strategies, solving previously unsolved problems. Finally, we see some evidence that they discover new search strategies when trained to optimize for accuracy.
introduction
Stream of Search (SoS): Learning to Search in Language
1Introduction
2404.03683
2Related WorksLanguage models as components of a search system.A line of recent work integrates language models as part of larger search and planning systems. In these methods, LMs typically play two roles: (1)  to generate candidate actions or successor states in the reasoning process, and (2)  to evaluate proposed actions or states, by determining validity and/or assigning a heuristic value. A symbolic search algorithm such as BFS or DFS dictates the strategy for exploration, and how the steps or evaluators are called. While these methods have been shown to improve search accuracy on certain problems, the LM components are typically used only for inference, so their reasoning ability is not improved. In contrast, our work focuses on training LMs that are capable of exploration, backtracking, and other critical components of reasoning. Relative to these “extrinsic” methods, which use fixed search strategies, our method learns an “intrinsic” policy that allows the LM to autonomously search the solution space. In doing so, we avoid the high inference costsrequired by tree-of-thoughts style approaches.In-context demonstrations of search.In contrast to using separate symbolic search algorithms to guide search, in-context demonstrations can also be used to demonstrate search procedures in language. These methods allow the language model to perform tree search based on the demonstrated search procedures. While improving efficiency in search, these methods are restricted by the choices of the search procedure demonstrated in the in-context examples, such as the demonstrated exploration strategy or the search heuristic.Process supervision.Process supervision of LMs is another method where an external verifier model is trained to learn the rewards of each intermediate reasoning step and the LM is then trained on this detailed feedback. Process supervision was shown to outperform outcome supervision in mathematical reasoning tasks. However, the training of the verifier requires a very large labelled dataset with human-generated annotations for each intermediate step, which may not scale well for other, more complex domains. Comparatively, our method directly improves the model’s planning and search abilities and removes the need to train a verifier reward model.Learning from search trajectories.Similar to our work,train transformer models on search trajectories to mimic search strategies such as Monte Carlo Tree Search or BFS. While their approach focuses on mimicking a search procedure with a fixed heuristic, we are interested in autonomous usage of different search procedures and the discovery of new ones. In recent concurrent work,employ A* search traces to train a base transformer model, where each trace includes the state and fixed A* heuristic value and search procedure. Their work aims to train transformers to closely imitate A* and enhance its efficiency. In contrast, our method emphasizesdiscoveryof new search strategies and tackling challenging problems that symbolic algorithms struggle to solve.
body
Stream of Search (SoS): Learning to Search in Language
2Related Works
2404.03683
3A Language for SearchThe problem space can be modeled as a Markov Decision Process (MDP) , with a set of states,, representing the steps in solving a problem; a set of actions, representing operations that the model can perform on states or to transition between them; a transition function, defining the transition from one state to another based on an action; and a reward function, assigning a reward for reaching the goal state.The process of search can be modeled as follows. The query for search has an initial stateand a goal state.
The search tree is defined by the problem itself. It contains all possible explorations of all actions from the initial state(containing the input)  to each possible child state of, and so on until a leaf state (a state with no children)  is reached. A correct path to the solutionis a sequence of states and actionsin the search tree, where each successive stateis obtained by applying a valid actionto the previous state, i.e.,, and the final state in the sequence is the goal state(seeFig. 2) .Figure 2:A visualization of how a search process is translated into a stream of search with a language for search. (left)  The search process represented as a tree, with different states and operations. The colored states represent the search trajectory, the green states represent the correct path to the goaland the arrows represent transitions between the states. The black circles represent unexplored states. (center)  The search process serialized as text to create a stream of search. The labels specify the different components of the process. SeeFig. 1b for how this is realized in Countdown. (right)  The optimal path,, to the goal state. Backtracking, exploration and the messy process of search are excluded.We are interested in the process of search, which corresponds to moving around the search tree until a solution is found. To represent this process, we propose a vocabulary of primitive operations222This is a non-exhaustive list, we choose to represent the most salient operations here.that can be used to define components of different search algorithms. This is in spirit similar to defining a domain-specific language for planning.•Current State:, The statethat is being explored.•Goal State:, The statethat is the target.•State Queue:The states at the ‘frontier’ of the the the trajectory that haven’t been explored yet.•State Expansion Function:, Explore a state adjacent to the current statebased on a transition function.•Exploration Choice:Choosing the order of states to explore following the state expansion. Eg: Should the breadth be explored first (BFS)  or the depth (DFS)  or any of the frontier states () .•Pruning:An action discarding states or subtrees that are unlikely to lead to a solution.•Backtracking:An action to move between nodes that have been explored. This allows the algorithm to choose which state should be expanded next.•Goal Check:An action to check if the current state is the goal state.•Heuristic:A functionthat approximates the distance of the current statefrom the goal. This can be used to decide which states should be explored or pruned. The heuristic function serves as an approximation of the value function, guiding the search or decision-making process by estimating the desirability of states in terms of their distance to the goal.Each of these operations can be left implicit, affecting how the trajectory unfolds, or made explicit in language as part of the search trajectory. When operations are implicit, a model is more likely to internalize abstract representations for them that can be improved with training. Explicit operations will turn into explicit reasoning moves made by the LM. We choose to represent the current state, the goal state, the backtracking operations, the goal checks and the exploration choices explicitly in the trajectory (written using language,Fig. 2) . We choose to keep the heuristic functions, values of states and the pruning strategy implicit.
body
Stream of Search (SoS): Learning to Search in Language
3A Language for Search
2404.03683
4Problem SetupTask Description: CountdownTo show the utility of using streams of search, we look at a generalization of the 24 Gamecalled Countdown. Countdown is a game where a set of input numbers need to be combined with simple arithmetic operations to reach a target number (seeFig. 1a) . We choose this task, since it has a high branching factor (for a depth that has N inputs)  and thus requires planning, search and backtracking to be solved. We consider problems with 4 input numbers since these problems are challenging enough to have long search traces without the search traces exceeding a standard LM context window (for example, the search trajectories for games with 5 input numbers can be 60,000 tokens long) . The range of target numbers in our problems is from 10 to 100. We randomly hold out 10% of the targets for an ‘out-of-distribution’ evaluation.Training dataTo train a model on streams of search for Countdown, we construct a synthetic dataset using set of diverse, and suboptimal symbolic search strategies.
To construct our Stream of Search dataset, we define 12 search strategies based on breadth first search (BFS)  and depth first search (DFS)  (see App.Alg. 4,Alg. 3)  that rely on two simple and interpretable heuristic functions. The heuristics we use to guide search are 1)  the absolute difference between the sum of the remaining options and the target and 2)  distance to the factors of the target (seeApp. B) . We generate a stream of search dataset of 500,000 search trajectories and corresponding trajectories with just the optimal solution. Of these 500,000 trajectories, only about 57% of the trajectories lead to the solution.
In our dataset, a search trajectory or a stream of search is serialized as a string that represents a list of tree nodes / states in the order of traversal (either generation or exploration) .
We hold out two kinds of generalization cases: 1)  seen targets with new sets of inputs, and 2)  new targets with new sets of inputs.MetricsTo measure accuracy in Countdown, we evaluate the percentage of problems for which the model is able to generate a correct solution trajectory. More formally, we define correctness as a binary function,, if the correct path to the solution,, is present in the generated trajectory,.To quantitatively understand the search strategies used by the trained models, we define two ways to measure alignment between different search strategies:1.Alignment of Correctness: This measures whether two search strategies solve the same set of problems correctly and incorrectly. We calculate this as the Pearson correlation between the solved and unsolved problems for the two search strategies.2.Alignment of States Visited: This measures the overlap in the states visited by two search strategies. To calculate this, we parse the search trajectories,and, into their constituent states and count the number of states that are common between them. We normalize this count by dividing it by the maximum number of states in the two trajectories. More formally:whereanddenote the number of states in trajectoriesand, respectively. To measure alignment between two models, we calculate the state alignment score for each problem and then compute the mean of these scores.
body
Stream of Search (SoS): Learning to Search in Language
4Problem Setup
2404.03683
5Learning from Suboptimal Search StrategiesIs it more useful to learn from clean, optimal solutions, or from messy, and sometimes unsuccessful, search trajectories? In this section we train LMs from scratch on one or the other and evaluate there performance at solving held out Countdown problems.Figure 3:(a)  Average accuracy of the symbolic search strategies used to construct the Stream of Search dataset compared to the model trained with optimal paths and the Stream of Search model. Error bars represent 95% binomial confidence intervals. (b)  Alignment of Correctness: The correlations of performance on different problems for the Stream of Search model and different symbolic search strategies. (c)  Alignment of States Visited: Number of times each search strategy visited the same states, normalized by the maximum number of states in the two trajectories. We find that the alignment scores for states visited for the SoS model are not highly correlated with any single symbolic strategy.Experiment Setup.For all experiments, we train a GPT-Neo modelwith 250M parameters and a context length of 4096 tokens (see appendix for more details) .
We train models with a causal language modeling objective on the 500,000 Countdown problems in two conditions: 1)  Optimal Paths (OP) : the model is trained to predict the correct, and optimal pathfor all problems in the dataset. 2)  Stream of Search (SoS) : The model is trained on search trajectoriessampled from different search strategies. The SoS dataset has 285,000 correct solutions while the OP dataset has 500,000 correct solutions. Both models are trained up to the same number of gradient steps.Results.We find that the model trained on streams of search outperforms the model trained on the optimal solutions (seeFig. 3a) . The SoS model achieves an accuracy of 51.27% on held-out inputs compared to 25.73% of the OP model. A similar pattern is seen for the test set with held-out targets. This shows that despite having fewer ‘correct’ examples for training, the SoS model outperforms the OP model.
The SoS model has a slightly lower accuracy compared to the average accuracy of the symbolic search methods used to construct the SoS training dataset. However, it is worth noting that SoS is solving a harder problem: symbolic search relies on an environment model to expose state transitions, while the SoS LM simulates those transitions itself, effectively learning to search with an internal ‘world model’; the language model must also learn how to perform arithmetic operations. Despite these challenges, the SoS model generates valid trajectories with a low error rate in state exploration (0.8%)  and only about 2 arithmetic errors per trajectory on average (see App.Tab. 2) .To understand the strategies that the trained SoS model uses, we measure the alignment of the model generated search trajectories with symbolic strategies. We find that the alignment scores for states visited for the SoS model are not highly correlated with any single symbolic strategy (seeFig. 3c) . The highest correlation of SoS is with DFS using the sum heuristic (0.57) , and the lowest correlation is with BFS using a breadth size of 5 and the sum heuristic (0.27) . A similar pattern is observed when we measure the alignments of correctness (seeFig. 3b)  for the SoS model and the symbolic search strategies. Overall, the SoS model has higher scores for alignment with strategies that use the sum heuristic, but does not seem to predominantly use any one strategy from its training data.
body
Stream of Search (SoS): Learning to Search in Language
5Learning from Suboptimal Search Strategies
2404.03683
6Policy Improvement with Stream of SearchThe SoS LM has learned to use search to solve new problems.
Can it learn to improve upon the symbolic strategies seen in its training data? In this section, we explore if the model can self-improve with feedback based on correctness and efficiency. We measure ability to solve previously unsolved problems (by the symbolic search strategies)  from the training dataset and solve difficult problems from the training set that none of the symbolic search strategies can solve. To improve the model, we use two RL strategies, expert iteration using STaR, and Advantage-Induced Policy Alignment.Experiment Setup.To improve the models with STaR (Alg. 1) , we use problems in the training dataset to generate 100,000 correct trajectories. We sample with a temperature of. These trajectories are then used to finetune the model. We repeat this process until we see a convergence in performance on the validation set.Algorithm 1Expert Iteration with STaR1:InputThe SoS model trained on Stream of Search Dataset2:Initialize the SoS model3:forindo4:Perform trajectory generation5:Filter trajectories based on correctness6:Finetune the model on correct solutions7:endforAlternatively, we can use advantage-induced policy alignment (APA;Alg. 2) . APA is an Actor-Critic reinforcement learning technique that involves creating a copy of the language model to serve as a value network that then used to enhance the policy, the original language model. We define a straightforward reward function that takes into account the correctness and length of the generated trajectory. We try APA, as we wanted to see if using a separate value network would improve exploration and the path-stitching abilities of the language model. We chose APA over other methods like Proximal Policy Optimization (PPO)  due to its stability and robustness to changes in hyperparameters.APA uses a reference policy,, to prevent the policy from drifting from its initial state. We observe that updating the reference policy whenever the validation reward converges results in further policy improvement (seeFig. 4b) . This shifting of the reference distribution can be interpreted as a means to reduce the weight assigned to staying close to the reference distribution (theparameter in the APA objective) . In practice, we found the strategy of shifting the reference distribution to be more stable for training when compared to designing a schedule for reducingover training.Algorithm 2Advantage Indued Policy-Alignment (APA) 1:Input:An initial policy parameter, a given reward function, Advantage coefficient.2:3:Copy the SoS model to create a reference network.4:Copy the SoS model to create a value network.5:forindo6:Roll outto produce dataset7:Update policy function according to8:.We omit the critic loss for simplicity9:where10:.11:If validation reward converges, update12:13:endforFigure 4:(a)  Improvement in accuracy with different policy improvement methods. (b)  Accuracy on the validation set of problems during APA training. We reset the reference policy after the validation accuracy converges. Different colors represent training after resetting the reference policy to the current model. We stop seeing improvements after 3 such resets. (c)  Improvements in test accuracies with each STaR iteration. Error bars represent 95% Binomial Confidence Intervals.Results.The SoS models converge after 3 iterations of STaR finetuning (Fig. 4) . We see that after three iterations, the finetuned SoS+STaR model solves an additional 5% of the held-out inputs test set beyond the base SoS model. A similar pattern is seen for the held-out targets test set. When SoS models are finetuned with APA, we see that the validation accuracy stops improving after about 4000 training steps. We reset the reference policy 3 times, when the validation accuracy stops improving — seeFig. 4b; different colors represent training after resetting the reference policy. Overall, we see an improvement of about 6% over the base SoS model.When we analyze the difference between the base and finetuned models in terms of the alignment of the states visited, we observe (Fig. 5a)  that both the STaR and APA models visit more states associated with the ‘multiply’ heuristic, which measures distance to the factors of the target. Further, we note that the APA model is less aligned with the symbolic strategies compared to the base SoS model, indicating that it diverges more from the symbolic strategies and employs different strategies for searching. These state visitation metrics provide insights into how the SoS+STaR and SoS+APA models can flexibly utilize various search strategies, potentially discovering novel heuristics and search methods.To further evaluate the performance of the improved models, we select 10,000 problems from the SoS training set that were unsolved by symbolic strategies when the dataset was generated, and 10,000 difficult problems that none of the symbolic strategies used to train the SoS models can solve. Remarkably, the models are able to solve approximately 36% of the previously unsolved problems (Fig. 5b)  and about 4% of the difficult problems (Fig. 5c) . Finally, SoS + APA and SoS + STaR models also have better models of the environment, making fewer errors while searching (Fig. 6; App.Tab. 2) , and finding the solution more quickly (Fig. 6) .Figure 5:(a)  Difference in the states visited compared to the base Stream of Search Model after finetuning with APA and STaR. We see that the model prefers the states visited by certain symbolic strategies after improvement. (b)  The percentage of previously unsolved problems from the training set that the trained models were able to solve. (c)  The percentage of difficult problems, defined as those not solvable by any symbolic algorithms, that the trained SoS models successfully solved. Error bars are 95% Binomial Confidence Intervals.Figure 6:(left)  Average number of arithmetic errors made per search trajectory by different models. Policy improvement leads to fewer errors. (right)  Average number of states visited per trajectory for correct solutions. Policy improvement leads to more efficient solutions.
body
Stream of Search (SoS): Learning to Search in Language
6Policy Improvement with Stream of Search
2404.03683
7DiscussionWe have introduced the Stream of Search (SoS)  framework enabling language models to learn to solve problems by searching in language, without any external structure or components. By systematizing the elements of search into a unified language, we are able to represent various search strategies in a common format to construct a dataset with diverse streams of search. Our experiments demonstrate that training language models to search leads to superior performance compared to models trained solely on optimal trajectories. This highlights the importance of exposing models to the messy process of problem solving, with exploration and backtracking, instead of only the ideal solution steps. SoS models can then self-improve by optimizing for correctness, using STaR and APA.The SoS framework may address criticismsof language models for planning and problem solving.
The problem of snowballing errors is addressed by teaching a model to backtrack. Search allows models to explore alternative paths, overcoming failures in lookahead tasks by considering multiple possible outcomes before committing to a course of action. Crucially, SoS leads language models to learn an internal ’world model’ for search. Unlike symbolic search that relies on an explicit environment model, SoS models simulate state transitions themselves.
Using a learned world models allows more adaptable and generalizable searchand addresses a key criticism of pretrained LMs.Our empirical results were restricted to the game of Countdown. Yet Countdown, with a high branching factor and variable goal states, captures the characteristics of complex planning problems. We are optimistic that SoS extends to more challenging, real-world tasks. In the short-term, externally-structured search methods such as Tree of Thoughtare likely to be more efficient for these tasks; in the longer run the increased flexibility and learnability of internally-structured search (i.e. SoS)  may prevail. It is also plausible these approaches can be combined, for sinstance, with with ToT providing initial training data for SoS.While we leave the evaluation of states to be done implicitly by the network in our current work, explicitly representing state evaluationsand introducing other formalizable operations such as limits, summarization, cycle checks, and subgoal setting could enhance the SoS framework.
Future research could explore integrating subgoals and hierarchical planning, as well as incorporating reflection and self-evaluation to enable models to discover and improve novel search strategies.
Generating the initial SoS dataset can be challenging, as it is not always feasible to create symbolic search algorithms to solve problems.
An important question is how well search abilities transfer between domains and between formal and informal domains.In conclusion, we have shown that hallmarks of symbolic reasoning—structured search with backtracking, heuristic state evaluation and world modeling—can be achieved within a sequence modeling paradigm.
To do so requires showing language models examples of productive mistakes, not only the optimal final solutions.
By embracing the diversity of search strategies and iteratively refining models, we can unlock the potential of language models to tackle complex problems and discover new ways to solve them.
body
Stream of Search (SoS): Learning to Search in Language
7Discussion
2404.03683
AcknowledgementsWe would like to thank Gabriel Poesia, Jacob Andreas, Joy He-Yueya, Dongwei Jiang, Eric Zelikman, Jan-Philipp Fränken and Ced Zhang for their discussions and support. This worked was supported by the Stanford Human-Centered Artifical Intelligence (HAI)  - Google grant, and the NSF Expeditions Grant, Award Number (FAIN)  1918771.
conclusion
Stream of Search (SoS): Learning to Search in Language
Acknowledgements