title: Appending pandas dataframes after looping through layers
body: <p>I need some help with the final step of formatting API results for import to a PostgreSQL database. The structure of the data is:</p>
  <pre><code>[{  &quot;season&quot;: 0,
      &quot;seasonType&quot;: &quot;string&quot;,
      &quot;week&quot;: 0,
      &quot;polls&quot;: [
        {
          &quot;poll&quot;: &quot;string&quot;,
          &quot;ranks&quot;: [
            {
              &quot;rank&quot;: 0,
              &quot;school&quot;: &quot;string&quot;,
              &quot;conference&quot;: &quot;string&quot;,
              &quot;firstPlaceVotes&quot;: 0,
              &quot;points&quot;: 0 }]}]}]
  </code></pre>
  <p>Here's the code I'm using to unpack it (of course if there is a better, more efficient way to do this too, all ears):</p>
  <pre><code>year = list(range(2020,2021))
  req = []
  pbp = pd.DataFrame()
  headers = {
      &quot;Accept&quot;: &quot;application/json&quot;,
      &quot;Authorization&quot;: &quot;Bearer 1a2b3c4d&quot;}
  for year in tqdm(year, desc = 'fetch record'):
      parameters = {&quot;year&quot;:year, &quot;seasonType&quot;:&quot;regular&quot;}
      req = requests.get(&quot;https://api.collegefootballdata.com/rankings&quot;, headers=headers, params = parameters)
      r = req.json()
      print(type(r))
      df1 = pd.DataFrame(r, columns = ['season', 'seasonType', 'week'], dtype = int)
      pbp = pbp.append(json.loads(req.text))
      for polls in pbp[&quot;polls&quot;]:
          try:
              p1 = polls[1]
          except IndexError:
              continue
          df2 = pd.DataFrame.from_dict(p1)
          poll = df2[&quot;poll&quot;]
          y = df1.append(poll)
          for rank in df2[&quot;ranks&quot;]:
              df3 = pd.DataFrame(rank, index=[0])
              z = y.append(df3)
  </code></pre>
  <p>When I append, the data comes out like this:</p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th>year</th>
  <th>season</th>
  <th>week</th>
  <th>poll</th>
  <th>rank</th>
  <th>team</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>2020</td>
  <td>regular</td>
  <td>1</td>
  <td></td>
  <td></td>
  <td></td>
  </tr>
  <tr>
  <td>2020</td>
  <td>regular</td>
  <td>2</td>
  <td></td>
  <td></td>
  <td></td>
  </tr>
  <tr>
  <td></td>
  <td></td>
  <td></td>
  <td>AP</td>
  <td></td>
  <td></td>
  </tr>
  <tr>
  <td></td>
  <td></td>
  <td></td>
  <td></td>
  <td>1</td>
  <td>Alabama</td>
  </tr>
  <tr>
  <td>2020</td>
  <td>regular</td>
  <td>1</td>
  <td></td>
  <td></td>
  <td></td>
  </tr>
  <tr>
  <td>2020</td>
  <td>regular</td>
  <td>2</td>
  <td></td>
  <td></td>
  <td></td>
  </tr>
  <tr>
  <td></td>
  <td></td>
  <td></td>
  <td>AP</td>
  <td></td>
  <td></td>
  </tr>
  <tr>
  <td></td>
  <td></td>
  <td></td>
  <td></td>
  <td>2</td>
  <td>Clemson</td>
  </tr>
  </tbody>
  </table>
  </div>
  <p>And, I'd like it to look like this:</p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th>year</th>
  <th>season</th>
  <th>week</th>
  <th>poll</th>
  <th>rank</th>
  <th>team</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>2020</td>
  <td>regular</td>
  <td>1</td>
  <td>AP</td>
  <td>1</td>
  <td>Alabama</td>
  </tr>
  <tr>
  <td>2020</td>
  <td>regular</td>
  <td>1</td>
  <td>AP</td>
  <td>2</td>
  <td>Clemson</td>
  </tr>
  <tr>
  <td>2020</td>
  <td>regular</td>
  <td>2</td>
  <td>AP</td>
  <td>1</td>
  <td>Alabama</td>
  </tr>
  <tr>
  <td>2020</td>
  <td>regular</td>
  <td>2</td>
  <td>AP</td>
  <td>2</td>
  <td>Clemson</td>
  </tr>
  </tbody>
  </table>
  </div>