title: Detect the final page of a paginated cURL response in a Bash loop
body: <h2>Goal</h2>
  
  <p>Include all objects from a paginated REST API call in a single JSON file using cURL and Bash. This combined list will be fed into a Power BI report.</p>
  
  <h2>Details</h2>
  
  <p>A request returns a maximum of 100 objects. There are 400+ objects total. The total grows over time. I don't want to maintain a script that includes something like <code>for set in 0 100 200 300 400 ; do</code> because it requires me to manually match the sets to the current number of objects. To save labor, I would like a script to auto-detect when the final page has been processed and then break. </p>
  
  <p>To achieve my goal, the plan I have come up with so far is to extract each incremental set of 100 items into its own JSON file, then assemble them with <code>cat</code> and extract relevant JSON keys/values using JQ. The reason for Bash is that it is the only programming I know.</p>
  
  <h2>Attempt</h2>
  
  <p>(Based on <a href="https://stackoverflow.com/questions/42866665/bash-scripting-curl-request-with-twist">this question and answer</a>)</p>
  
  <pre><code>for ((i=0; ; i+=100)); do
      contents=$(curl -u "username:password" -H "Content-Type: application/json" "https://&lt;url&gt;/api/core/v3/places?count=100&amp;startIndex=$i")
      echo "$contents" &gt; $i.json
      if [[ $contents =~ 'list" : [ ]' ]]
      then break
      fi
  done
  </code></pre>
  
  <h3>Result</h3>
  
  <p>All pages export as expected except the first and last pages:</p>
  
  <ul>
  <li>The first <code>startIndex</code> should be <code>0</code> but the code makes the <code>startIndex</code> <code>100</code>. I have tried a number of variations with <code>i</code> but continue to fail.</li>
  <li>[edit: solved, thank you @weirdan] Neither <code>list":null</code> nor <code>next":null</code> end the loop. The script exports incremented JSON files indefinitely.~</li>
  </ul>
  
  <h2>Reference</h2>
  
  <h3>First page of returned paginated JSON</h3>
  
  <pre><code>{
    "itemsPerPage" : 100,
    "links" : {
      "next" : "https://&lt;url&gt;/api/core/v3/places?sort=titleAsc&amp;count=100&amp;startIndex=0" &lt;--- with my script, startIndex is erroneously 100
    },
    "list" : [ {
  ...
  </code></pre>
  
  <h3>Intermediate pages</h3>
  
  <pre><code>{
    "itemsPerPage" : 100,
    "links" : {
      "previous" : "https://&lt;url&gt;/api/core/v3/places?sort=titleAsc&amp;count=100",
      "next" : "https://&lt;url&gt;/api/core/v3/places?sort=titleAsc&amp;count=100&amp;startIndex=200"
    },
    "list" : [ {
  ...
  </code></pre>
  
  <h3>Final page</h3>
  
  <pre><code>{
    "itemsPerPage" : 100,
    "links" : {
      "previous" : "https://&lt;url&gt;/api/core/v3/places?sort=titleAsc&amp;count=100&amp;startIndex=400"
    },
    "list" : [ {
  ...
  </code></pre>
  
  <h3>Empty page</h3>
  
  <pre><code>{
    "itemsPerPage" : 100,
    "list" : [ ],
    "startIndex" : 500
  }
  </code></pre>
  
  <p>Thank you for any advice or ideas.</p>