Python urllib2 - cannot read a page
Question: I am using `urllib2` in `Python` to scrape a webpage. However, the `read()`
method does not return.

Here is the code I am using:

    
    
    import urllib2
    
    url = 'http://edmonton.en.craigslist.ca/kid/'
    
    headers = {'User-Agent': 'Mozilla/5.0'}
    request = urllib2.Request(url, headers=headers)
    f_webpage = urllib2.urlopen(request)
    html = f_webpage.read()  # <- does not return
    

I last ran the script a month ago and it was working fine then.

Note that the same script runs well for webpages of other categories on
Edmonton Craigslist like `http://edmonton.en.craigslist.ca/act/` or
`http://edmonton.en.craigslist.ca/eve/`.


Answer: As requested in comments :)

Install `requests` by `$ pip install requests`

Use `requests` as the following:

    
    
    >>> import requests
    >>> url = 'http://edmonton.en.craigslist.ca/kid/'
    >>> headers = {'User-Agent': 'Mozilla/5.0'}
    >>> request = requests.get(url, headers=headers)
    >>> request.ok
    True
    >>> request.text  # content in string, similar to .read() in question
    ...
    ...
    

Disclaimer: this is not technically the answer to OP's question, but solves
OP's problem as `urllib2` is known to be problematic and `requests` library is
born to solve such problems.

