title: Python - how to extract data by looping through paginated API (Harvest)
body: <p><br>
  First of all I have been working with Python for about a couple of days, so I don't necessarily know the best practices or all the terminology ... yet. I learn best by reverse engineering and my code below is based on the official documentation from Harvest and other bits I've found with google-fu </p>
  
  <p>My request is to download all the time entries records from Harvest and save as a JSON (or ideally a CSV file).</p>
  
  <p><a href="https://github.com/harvesthq/harvest_api_samples/blob/master/v2/harvest_api_sample.py/%22LINK%22" rel="nofollow noreferrer">Official Python Example from Harvest Git Hub</a></p>
  
  <p>This is my adapted code (including all outputs, which won't be neccessary in the final code but handy for my learning):</p>
  
  <pre><code>import requests, json, urllib.request
  
  #Set variables for authorisation
  AUTH = "REDACTED"
  ACCOUNT = "REDACTED"
  
  URL = "https://api.harvestapp.com/v2/time_entries"
  HEADERS = { "Authorization": AUTH,
              "Harvest-Account-ID": ACCOUNT}
  PAGENO = str("5")
  
  request = urllib.request.Request(url=URL+"?page="+PAGENO, headers=HEADERS)
  response = urllib.request.urlopen(request, timeout=5)
  responseBody = response.read().decode("utf-8")
  jsonResponse = json.loads(responseBody)
  
  # Find the values for pagination
  parsed = json.loads(responseBody)
  links_first = parsed["links"]["first"]
  links_last = parsed["links"]["last"]
  links_next = parsed["links"]["next"]
  links_previous = parsed["links"]["previous"]
  nextpage = parsed["next_page"]
  page = parsed["page"]
  perpage = parsed["per_page"]
  prevpage = parsed["previous_page"]
  totalentries = parsed["total_entries"]
  totalpages = parsed["total_pages"]
  
  #Print the output
  print(json.dumps(jsonResponse, sort_keys=True, indent=4))
  print("first link : " + links_first)
  print("last link : " + links_last)
  print("next page : " + str(nextpage))
  print("page : " + str(page))
  print("per page : " + str(perpage))
  print("total records : " + str(totalentries))
  print("total pages : " + str(totalpages))
  </code></pre>
  
  <p>The output response is
  <br>"Squeezed text (5816 lines)"
  <br>first link : <a href="https://api.harvestapp.com/v2/time_entries?page=1&amp;per_page=100" rel="nofollow noreferrer">https://api.harvestapp.com/v2/time_entries?page=1&amp;per_page=100</a>
  <br>last link : <a href="https://api.harvestapp.com/v2/time_entries?page=379&amp;per_page=100" rel="nofollow noreferrer">https://api.harvestapp.com/v2/time_entries?page=379&amp;per_page=100</a>
  <br>next page : 6
  <br>page : 5
  <br>per page : 100
  <br>total records : 37874
  <br>total pages : 379</p>
  
  <p>Please can someone advise the best way to loop through the pages to form one JSON file ?
  If you are also able to advise the best way then output that JSON file I would be very grateful. </p>