title: API capture all paginated data? (python)
body: <p>I'm using the requests package to hit an API (greenhouse.io). The API is paginated so I need to loop through the pages to get all the data I want. Using something like:</p>
  
  <pre><code>results = []
  for i in range(1,326+1):
      response = requests.get(url, 
                              auth=(username, password), 
                              params={'page':i,'per_page':100})
      if response.status_code == 200:
          results += response.json()
  </code></pre>
  
  <p>I know there are 326 pages by hitting the headers attribute:</p>
  
  <pre><code>In [8]:
  response.headers['link']
  Out[8]:
  '&lt;https://harvest.greenhouse.io/v1/applications?page=3&amp;per_page=100&gt;; rel="next",&lt;https://harvest.greenhouse.io/v1/applications?page=1&amp;per_page=100&gt;; rel="prev",&lt;https://harvest.greenhouse.io/v1/applications?page=326&amp;per_page=100&gt;; rel="last"'
  </code></pre>
  
  <p>Is there any way to extract this number automatically? Using the requests package? Or do I need to use regex or something?</p>
  
  <p>Alternatively, should I somehow use a while loop to get all this data? What is the best way? Any thoughts?</p>