title: Python Requests Module Not Getting Latest Data from Web Server
body: <p>In the code snippet below, you can see that I am trying to scrape some data from the NCAA Men's Basketball website.</p>
  
  <pre><code>import requests
  
  url = "https://www.ncaa.com/scoreboard/basketball-men/d1/"
  
  response = requests.get(url)
  html = response.text
  
  print(html)
  print(response.headers)
  print("\n\n")
  print(response.request.headers)
  </code></pre>
  
  <p>The website has a listing of games and their scores. I figured out how to pull all the data I need using Python Requests for the HTTP request and then BeautifulSoup for extracting data from the HTML. <a href="https://repl.it/@joekreydt/MadnessTheBot07" rel="nofollow noreferrer">The full scraper is here if you'd like to take a look.</a></p>
  
  <p><strong>The problem:</strong> When Requests gets the response from the NCAA website, the data is much older (sometimes up to 30 or 40 minutes, at least) than the data on the actual website.</p>
  
  <p>I've been Googling this for hours. After reading through the <a href="http://docs.python-requests.org/en/master/" rel="nofollow noreferrer">Python Requests docs</a>, I believe I have discovered that the NCAA web server is sending outdated data. But I don't understand why it would send my program outdated data when it sends Google Chrome (or whatever web browser) the correct data.</p>
  
  <p>The reason I believe the server is sending outdated data is that when I print the response headers, one of the items is <em>'Last-Modified': 'Sat, 26 Jan 2019 17:49:13 GMT'</em> while another is <em>'Date': 'Sat, 26 Jan 2019 18:20:29 GMT'</em>, so it looks like the server gets the request at the right time, but provides data that hasn't been modified in a while.</p>
  
  <p><strong>My question:</strong> Do you know of any reason why this would happen? Is there something I need to add in my HTTP request that would get the server to send me data consistent with what is sends web browsers?</p>
  
  <p>P.S. I am so sorry for the long question. I tried to keep it concise, yet still explain things clearly.</p>