Extracting from HTML with beautifulsoup
Question: I try to extract lotto numbers from
<https://www.lotto.de/de/ergebnisse/lotto-6aus49/archiv.html> (I know there is
an easier way, but it's rather for learning).

![These are the numbers I want to
extract:](http://i.stack.imgur.com/41I0D.jpg)

Tried with Python, beautifulsoup the following:

    
    
    from BeautifulSoup import BeautifulSoup
    import urllib2
    
    url="https://www.lotto.de/de/ergebnisse/lotto-6aus49/archiv.html"
    page=urllib2.urlopen(url)
    soup = BeautifulSoup(page.read())
    numbers=soup.findAll('li',{'class':'winning_numbers.boxRow.clearfix'})
    
    for number in numbers:
        print number['li']+","+number.string
    

Returns nothing, which I actually expected. I read the tutorial, but still
didn't understand the parsing totally. Could someone give me a hint?

Thank you!


Answer: As the data content is dynamically generated, one of the _EASIER_ solutions
you may use **[Selenium](https://selenium-python.readthedocs.org)** or alike
to simulate the action as a browser (I use
**[PhantomJS](http://phantomjs.org)** as webdriver), like so:

    
    
    from selenium import webdriver
    
    url="https://www.lotto.de/de/ergebnisse/lotto-6aus49/archiv.html"
    # I'm using PhantomJS, you may use your own...
    driver = webdriver.PhantomJS(executable_path='/usr/local/bin/phantomjs')
    driver.get(url)
    soup = BeautifulSoup(driver.page_source)
    # I just simply go through the div class and grab all number texts
    # without special number, like in the Sample
    for ul in soup.findAll('div', {'class': 'winning_numbers'}):
        n = ','.join(li for li in ul.text.split() if li.isdigit())
        if n:
            print 'number: {}'.format(n)
    
    number: 6,25,26,27,28,47
    

To also grab the special number:

    
    
    for ul in soup.findAll('div', {'class': 'winning_numbers'}):
        # grab only numeric chars, you may apply your own logic here
        n = ','.join(''.join(_ for _ in li if _.isdigit()) for li in ul.text.split())
        if n:
            print 'number: {}'.format(n)
    
    number: 6,25,26,27,28,47,5 # with special number
    

Hope this helps.

