Unable to read HTML content
Question: I'm building a webCrawler which needs to read links inside a webpage. For
which I'm using urllib2 library of python to open and read the websites.

I found a website where I'm unable to fetch any data. The URL is
"<http://www.biography.com/people/michael-jordan-9358066>"

My code,

    
    
    import urllib2
    
    response = urllib2.urlopen("http://www.biography.com/people/michael-jordan-9358066")
    print response.read()
    

By running the above code, the content I get from the website, if I open it in
a browser and the content I get from the above code is very different. The
content from the above code does not include any data.

I thought it could be because of delay in reading the web page, so I
introduced a delay. Even after the delay, the response is the same.

    
    
    response = urllib2.urlopen("http://www.biography.com/people/michael-jordan-9358066")
    time.sleep(20)
    print response.read()
    

The web page opens perfectly fine in a browser.

However, the above code works fine for reading Wikipedia or some other
websites. I'm unable to find the reason behind this odd behaviour. Please
help, thanks in advance.


Answer: What you are experiencing is most likely to be the effect of [dynamic web
pages](https://en.wikipedia.org/wiki/Dynamic_web_page). These pages do not
have static content for `urllib` or `requests` to get. The data is loaded on
site. You can use Python's [`selenium`](http://selenium-
python.readthedocs.io/) to solve this.

