title: Getting ‘wrong’ page source when calling url from python
body: <p>Trying to retrieve the page source from a website, I get a completely different (and shorter) text than when viewing the same page source through a web browser. </p>
  
  <p><a href="https://stackoverflow.com/questions/24563601/python-getting-a-wrong-source-code-of-the-web-page-asp-net">https://stackoverflow.com/questions/24563601/python-getting-a-wrong-source-code-of-the-web-page-asp-net</a></p>
  
  <p>This fellow has a related issue, but obtained the home page source instead of the requested one - I am getting something completely alien.</p>
  
  <p>The code is:</p>
  
  <pre><code>from urllib import request
  
  def get_page_source(n):
      url = 'https://www.whoscored.com/Matches/' + str(n) + '/live'
      response = request.urlopen(url)
      return str(response.read())
  
  n = 1006233
  text = get_page_source(n)
  </code></pre>
  
  <p>This is the page I am targeting in this example:
  <a href="https://www.whoscored.com/Matches/1006233/live" rel="noreferrer">https://www.whoscored.com/Matches/1006233/live</a></p>
  
  <p>The url in question contains rich information in the page source, but I end up getting only the following when running the above code:</p>
  
  <p>text =</p>
  
  <pre><code>b'&lt;html style="height:100%"&gt;&lt;head&gt;&lt;META NAME="ROBOTS" CONTENT="NOINDEX,
  NOFOLLOW"&gt;&lt;meta name="format-detection" content="telephone=no"&gt;&lt;meta 
  name="viewport" content="initial-scale=1.0"&gt;&lt;meta http-equiv="X-
  UA-Compatible" content="IE=edge,chrome=1"&gt;&lt;/head&gt;&lt;body style="margin:0px;
  height:100%"&gt;&lt;iframe src="/_Incapsula_Resource?CWUDNSAI=24&amp;
  xinfo=0-12919260-0 0NNY RT(1462118673272 111) q(0 -1 -1 -1) r(0 -1) 
  B12(4,315,0) U2&amp;incident_id=276000100045095595-100029307305590944&amp;edet=12&amp;
  cinfo=04000000" frameborder=0 width="100%" height="100%" marginheight="0px" 
  marginwidth="0px"&gt;Request unsuccessful. Incapsula incident ID: 
  276000100045095595-100029307305590944&lt;/iframe&gt;&lt;/body&gt;&lt;/html&gt;'
  </code></pre>
  
  <p>What went wrong here? Can a server detect a robot even when it has not sent repetitive requests – if yes, how – and is there a way around?</p>