title: How to parse paginated JSON API response with complex nesting and unnamed array?
body: <p>I have built (with the help of @EmielZuurbier) an invoice template which places an API call to Quickbase. The API response is paginated. How can I parse the paginated response into a single table?</p>
  <ul>
  <li>Quickbase API Endpoint: <a href="https://developer.quickbase.com/operation/runQuery" rel="nofollow noreferrer">https://developer.quickbase.com/operation/runQuery</a></li>
  <li>Quickbase paginated metadata explanation: <a href="https://developer.quickbase.com/pagination" rel="nofollow noreferrer">https://developer.quickbase.com/pagination</a></li>
  </ul>
  <p><strong>Here is what a response from the API call looks like (I deleted most of the items under data or else it would have been to long to post on stackoverflow</strong></p>
  <pre><code>{
      &quot;data&quot;: [
          {
              &quot;15&quot;: {
                  &quot;value&quot;: &quot;F079427&quot;
              },
              &quot;19&quot;: {
                  &quot;value&quot;: 50.0
              },
              &quot;48&quot;: {
                  &quot;value&quot;: &quot;(S1)&quot;
              },
              &quot;50&quot;: {
                  &quot;value&quot;: &quot;2021-03-01&quot;
              },
              &quot;8&quot;: {
                  &quot;value&quot;: &quot;71 Wauregan Rd, Danielson, Connecticut 06239&quot;
              }
          },
          {
              &quot;15&quot;: {
                  &quot;value&quot;: &quot;F079430&quot;
              },
              &quot;19&quot;: {
                  &quot;value&quot;: 50.0
              },
              &quot;48&quot;: {
                  &quot;value&quot;: &quot;(S1)&quot;
              },
              &quot;50&quot;: {
                  &quot;value&quot;: &quot;2021-03-01&quot;
              },
              &quot;8&quot;: {
                  &quot;value&quot;: &quot;7 County Home Road, Thompson, Connecticut 06277&quot;
              }
          },
          {
              &quot;15&quot;: {
                  &quot;value&quot;: &quot;F079433&quot;
              },
              &quot;19&quot;: {
                  &quot;value&quot;: 50.0
              },
              &quot;48&quot;: {
                  &quot;value&quot;: &quot;(S1)&quot;
              },
              &quot;50&quot;: {
                  &quot;value&quot;: &quot;2021-03-16&quot;
              },
              &quot;8&quot;: {
                  &quot;value&quot;: &quot;12 Bentwood Street, Foxboro, Massachusetts 02035&quot;
              }
          }
      ],
      &quot;fields&quot;: [
          {
              &quot;id&quot;: 15,
              &quot;label&quot;: &quot;Project Number&quot;,
              &quot;type&quot;: &quot;text&quot;
          },
          {
              &quot;id&quot;: 8,
              &quot;label&quot;: &quot;Property Adress&quot;,
              &quot;type&quot;: &quot;address&quot;
          },
          {
              &quot;id&quot;: 50,
              &quot;label&quot;: &quot;Date Completed&quot;,
              &quot;type&quot;: &quot;text&quot;
          },
          {
              &quot;id&quot;: 48,
              &quot;label&quot;: &quot;Billing Codes&quot;,
              &quot;type&quot;: &quot;text&quot;
          },
          {
              &quot;id&quot;: 19,
              &quot;label&quot;: &quot;Total Job Price&quot;,
              &quot;type&quot;: &quot;currency&quot;
          }
      ],
      &quot;metadata&quot;: {
          &quot;numFields&quot;: 5,
          &quot;numRecords&quot;: 500,
          &quot;skip&quot;: 0,
          &quot;totalRecords&quot;: 766
      }
  }
  </code></pre>
  <p><strong>Below is the full javascript code i am using</strong></p>
  <pre><code>const urlParams = new URLSearchParams(window.location.search);
  //const dbid = urlParams.get('dbid');//
  //const fids = urlParams.get('fids');//
  let rid = urlParams.get('rid');
  //const sortLineItems1 = urlParams.get('sortLineItems1');//
  //const sortLineItems2 = urlParams.get('sortLineItems2');//
  let subtotalAmount = urlParams.get('subtotalAmount');
  let discountAmount = urlParams.get('discountAmount');
  let creditAmount = urlParams.get('creditAmount');
  let paidAmount = urlParams.get('paidAmount');
  let balanceAmount = urlParams.get('balanceAmount');
  let clientName = urlParams.get('clientName');
  let clientStreetAddress = urlParams.get('clientStreetAddress');
  let clientCityStatePostal = urlParams.get('clientCityStatePostal');
  let clientPhone = urlParams.get('clientPhone');
  let invoiceNumber = urlParams.get('invoiceNumber');
  let invoiceTerms = urlParams.get('invoiceTerms');
  let invoiceDate = urlParams.get('invoiceDate');
  let invoiceDueDate = urlParams.get('invoiceDueDate');
  let invoiceNotes = urlParams.get('invoiceNotes');
  
  
  const formatCurrencyUS = function (x) {
      return new Intl.NumberFormat('en-US', { style: 'currency', currency: 'USD' }).format(x);
  }
  
  
  let subtotalAmountFormatted = formatCurrencyUS(subtotalAmount);
  let discountAmountFormatted = formatCurrencyUS(discountAmount);
  let creditAmountFormatted = formatCurrencyUS(creditAmount);
  let paidAmountFormatted = formatCurrencyUS(paidAmount);
  let balanceAmountFormatted = formatCurrencyUS(balanceAmount);
  
  
  document.getElementById(&quot;subtotalAmount&quot;).innerHTML = `${subtotalAmountFormatted}`;
  document.getElementById(&quot;discountAmount&quot;).innerHTML = `${discountAmountFormatted}`;
  document.getElementById(&quot;creditAmount&quot;).innerHTML = `${creditAmountFormatted}`;
  document.getElementById(&quot;paidAmount&quot;).innerHTML = `${paidAmountFormatted}`;
  document.getElementById(&quot;balanceAmount&quot;).innerHTML = `${balanceAmountFormatted}`;
  document.getElementById(&quot;clientName&quot;).innerHTML = `${clientName}`;
  document.getElementById(&quot;clientStreetAddress&quot;).innerHTML = `${clientStreetAddress}`;
  document.getElementById(&quot;clientCityStatePostal&quot;).innerHTML = `${clientCityStatePostal}`;
  document.getElementById(&quot;clientPhone&quot;).innerHTML = `${clientPhone}`;
  document.getElementById(&quot;invoiceNumber&quot;).innerHTML = `${invoiceNumber}`;
  document.getElementById(&quot;invoiceTerms&quot;).innerHTML = `${invoiceTerms}`;
  document.getElementById(&quot;invoiceDate&quot;).innerHTML = `${invoiceDate}`;
  document.getElementById(&quot;invoiceDueDate&quot;).innerHTML = `${invoiceDueDate}`;
  document.getElementById(&quot;invoiceNotes&quot;).innerHTML = `${invoiceNotes}`;
  
  
  let headers = {
      'QB-Realm-Hostname': 'XXXXX',
      'User-Agent': 'Invoice',
      'Authorization': 'XXXXX',
      'Content-Type': 'application/json'
  }
  
  
  let body =
  
  {
      &quot;from&quot;: &quot;bq9dajvu5&quot;,
      &quot;select&quot;: [
          15,
          8,
          50,
          48,
          19
      ],
      &quot;where&quot;: `{25.EX.${rid}}`,
      &quot;sortBy&quot;: [
          {
              &quot;fieldId&quot;: 50,
              &quot;order&quot;: &quot;ASC&quot;
          },
          {
              &quot;fieldId&quot;: 8,
              &quot;order&quot;: &quot;ASC&quot;
          }
      ],
      &quot;options&quot;: {
          &quot;skip&quot;: 0
      }
  }
  
  
  const xmlHttp = new XMLHttpRequest();
  xmlHttp.open('POST', 'https://api.quickbase.com/v1/records/query', true);
  for (const key in headers) {
      xmlHttp.setRequestHeader(key, headers[key]);
  }
  
  
  xmlHttp.onreadystatechange = function () {
      if (xmlHttp.readyState === XMLHttpRequest.DONE) {
          console.log(xmlHttp.responseText);
  
  
          let line_items = JSON.parse(this.responseText);
          console.log(line_items);
  
  
  
  
  
          const transformResponseData = (line_items) =&gt; {
              const { data, fields } = line_items;
  
              //***Return a new array with objects based on the values of the data and fields arrays***//
              const revivedData = data.map(entry =&gt;
                  fields.reduce((object, { id, label }) =&gt; {
                      object[label] = entry[id].value;
                      return object;
                  }, {})
              );
  
              //***Combine the original object with the new data key***//
              return {
                  ...line_items,
                  data: revivedData
              };
          };
  
  
  
  
  
          const createTable = ({ data, fields }) =&gt; {
              const table = document.getElementById('line_items');                //const table = document.createElement('table'); 
              const tHead = document.getElementById('line_items_thead');      //const tHead = table.createTHead();
              const tBody = document.getElementById('line_items_tbody');      //const tBody = table.createTBody();
              //***Create a head for each label in the fields array***//
              const tHeadRow = tHead.insertRow();
  
  
  
              // ***Create the counts cell manually***//
              const tHeadRowCountCell = document.createElement('th');
              tHeadRowCountCell.textContent = 'Count';
              tHeadRow.append(tHeadRowCountCell);
  
  
  
              for (const { label } of fields) {
                  const tHeadRowCell = document.createElement('th');
                  tHeadRowCell.textContent = label;
                  tHeadRow.append(tHeadRowCell);
              }
  
  
              // Output all the values of the new data array//
              for (const [index, entry] of data.entries()) {
                  const tBodyRow = tBody.insertRow();
  
                  // Create a new array with the index and the values from the object//
                  const values = [
                      index + 1,
                      ...Object.values(entry)
                  ];
  
                  // Loop over the combined values array//
                  for (const [index, value] of values.entries()) {
                      const tBodyCell = tBodyRow.insertCell();
                      tBodyCell.textContent = index === 5 ?
                          Intl.NumberFormat('en-US', { style: 'currency', currency: 'USD' }).format(value) ://value.toFixed(2) :
                          value;
  
  
                  }
              }
              return table;
          };
  
  
  
  
  
          const data = transformResponseData(line_items);
          const table = createTable(data);
          document.getElementById(&quot;line_items_div&quot;).append(table)                 //.innerHTML = table &lt;-- this does not work//       //document.body.append(table); 
          console.log(data);
  
  
  
  
  
      }
  };
  
  xmlHttp.send(JSON.stringify(body));
  </code></pre>
  <p><strong>This is what I am trying to achieve (address is only shown as xxx so the table fits nicely on stackoverflow)</strong></p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th>Count</th>
  <th>Project Number</th>
  <th>Property Address</th>
  <th>Date Completed</th>
  <th>Billing Codes</th>
  <th>Total Job Price</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>1</td>
  <td>F079427</td>
  <td>xxx</td>
  <td>2021-03-01</td>
  <td>(S1)</td>
  <td>$50.00</td>
  </tr>
  <tr>
  <td>2</td>
  <td>F079430</td>
  <td>xxx</td>
  <td>2021-03-01</td>
  <td>(S1)</td>
  <td>$50.00</td>
  </tr>
  <tr>
  <td>3</td>
  <td>F079433</td>
  <td>xxx</td>
  <td>2021-03-16</td>
  <td>(S1)</td>
  <td>$50.00</td>
  </tr>
  </tbody>
  </table>
  </div>
  <p><strong>My thoughts on how to accomplish this</strong></p>
  <p>For the request formula we probably need a function that loops and it will skip an amount of records that is <code>=== to the sum of all the numRecords for every request made until skip + numRecords === totalRecords</code></p>
  <p>so for example if the totalRecords = 1700</p>
  <ol>
  <li>First request <code>{&quot;skip&quot;: 0}</code> returns numRecords=500</li>
  <li>Second request <code>{&quot;skip&quot;: 500}</code> returns numRecords=500</li>
  <li>Third request <code>{&quot;skip&quot;: 1000}</code> returns numRecords=500</li>
  <li>Fourth request <code>{&quot;skip&quot;: 1500}</code> returns numRecords=200</li>
  </ol>
  <p>On the fourth request <code>skip + numRecords = 1700</code> which is equal to the total records so the loop should stop.</p>
  <p>And after we have all of those arrays we somehow merge them together into a single table which is way more advanced javascript then what I am familiar with.</p>