Urllib Python is not providing with the html code I see with with inspect element
Question: I'm trying to crawl the results in this link:

url =
"<http://topsy.com/trackback?url=http%3A%2F%2Fmashable.com%2F2014%2F08%2F27%2Faustralia-
retail-evolution-lab-aopen-shopping%2F>"

When I inspect it with firebug I can see the html code and I know what I need
to do to extract the tweets. The problem is when I get the response using
urlopen, i don't get the same html code. I only get tags. What am I missing?

Example code below:

    
    
       def get_tweets(section_url):
         html = urlopen(section_url).read()
         soup = BeautifulSoup(html, "lxml")
         tweets = soup.find("div", "results")
         category_links = [dd.a["href"] for tweet in tweets.findAll("div", "result-tweet")]
         return category_links
    
    url =  "http://topsy.com/trackback?url=http%3A%2F%2Fmashable.com%2F2014%2F08%2F27%2Faustralia-retail-evolution-lab-aopen-shopping%2F"
    cat_links = get_tweets(url)
    

Thanks, YB


Answer: The problem is that the content of `results` div is filled up with extra HTTP
call and javascript code being executed on the browser side. `urllib` only
"sees" the initial HTML page that doesn't contain the data you need.

One option would be to follow @Himal's suggestion and simulate the underlying
request to `trackbacks.js` that is sent for the data with tweets. The result
is in JSON format that you can
[`load()`](https://docs.python.org/2/library/json.html#json.load) using
[`json`](https://docs.python.org/2/library/json.html) module coming with
standard library:

    
    
    import json
    import urllib2
    
    url = 'http://otter.topsy.com/trackbacks.js?url=http%3A%2F%2Fmashable.com%2F2014%2F08%2F27%2Faustralia-retail-evolution-lab-aopen-shopping%2F&infonly=0&call_timestamp=1411090809443&apikey=09C43A9B270A470B8EB8F2946A9369F3'
    data = json.load(urllib2.urlopen(url))
    for tweet in data['response']['list']:
        print tweet['permalink_url']
    

Prints:

    
    
    http://twitter.com/Evonomie/status/512179917610835968
    http://twitter.com/abs_office/status/512054653723619329
    http://twitter.com/TKE_Global/status/511523709677756416
    http://twitter.com/trevinocreativo/status/510216232122200064
    http://twitter.com/TomCrouser/status/509730668814028800
    http://twitter.com/Evonomie/status/509703168062922753
    http://twitter.com/peterchaly/status/509592878491136000
    http://twitter.com/chandagarwala/status/509540405411840000
    http://twitter.com/Ayjay4650/status/509517948747526144
    http://twitter.com/Marketingccc/status/509131671900536832
    

This was "going down to metal" option.

* * *

Otherwise, you can take a "high-level" approach and don't bother about what is
there happening under-the-hood. Let the real browser load the page which you
would interact with through [selenium WebDriver](http://selenium-
python.readthedocs.org/):

    
    
    from selenium import webdriver
    
    driver = webdriver.Chrome()  # can be Firefox(), PhantomJS() and more
    driver.get("http://topsy.com/trackback?url=http%3A%2F%2Fmashable.com%2F2014%2F08%2F27%2Faustralia-retail-evolution-lab-aopen-shopping%2F")
    
    for tweet in driver.find_elements_by_class_name('result-tweet'):
        print tweet.find_element_by_xpath('.//div[@class="media-body"]//ul[@class="inline"]/li//a').get_attribute('href')
    
    driver.close()
    

Prints:

    
    
    http://twitter.com/Evonomie/status/512179917610835968
    http://twitter.com/abs_office/status/512054653723619329
    http://twitter.com/TKE_Global/status/511523709677756416
    http://twitter.com/trevinocreativo/status/510216232122200064
    http://twitter.com/TomCrouser/status/509730668814028800
    http://twitter.com/Evonomie/status/509703168062922753
    http://twitter.com/peterchaly/status/509592878491136000
    http://twitter.com/chandagarwala/status/509540405411840000
    http://twitter.com/Ayjay4650/status/509517948747526144
    http://twitter.com/Marketingccc/status/509131671900536832
    

* * *

This is how you can scale the second option to get all of tweets following
pagination:

    
    
    from selenium import webdriver
    from selenium.common.exceptions import NoSuchElementException
    from selenium.webdriver.common.by import By
    from selenium.webdriver.support.ui import WebDriverWait
    from selenium.webdriver.support import expected_conditions as EC
    
    BASE_URL = 'http://topsy.com/trackback?url=http%3A%2F%2Fmashable.com%2F2014%2F08%2F27%2Faustralia-retail-evolution-lab-aopen-shopping%2F&offset={offset}'
    
    driver = webdriver.Chrome()
    
    # get tweets count
    driver.get('http://topsy.com/trackback?url=http%3A%2F%2Fmashable.com%2F2014%2F08%2F27%2Faustralia-retail-evolution-lab-aopen-shopping%2F')
    tweets_count = int(driver.find_element_by_xpath('//li[@data-name="all"]/a/span').text)
    
    for x in xrange(0, tweets_count, 10):
        driver.get(BASE_URL.format(offset=x))
    
        # page header appears in case no more tweets found
        try:
            driver.find_element_by_xpath('//div[@class="page-header"]/h3')
        except NoSuchElementException:
            pass
        else:
            break
    
        # wait for results
        WebDriverWait(driver, 5).until(
            EC.presence_of_element_located((By.ID, "results"))
        )
    
        # get tweets
        for tweet in driver.find_elements_by_class_name('result-tweet'):
            print tweet.find_element_by_xpath('.//div[@class="media-body"]//ul[@class="inline"]/li//a').get_attribute('href')
    
    driver.close()
    

