Search a webpage and grab certain data in Python 3
Question: Im trying to get the text from a webpage with Python 3.3 and then search
through that text for certain strings. When I find a matching string I need to
save the following text. For example I take this page:
<http://gatherer.wizards.com/Pages/Card/Details.aspx?name=Dark%20Prophecy> and
I need to save the text after each category (card text, rarity, etc) in the
card info. Currently Im using beautiful Soup but get_text causes a
UnicodeEncodeError and doesnt return an iterable object. Here is the relevant
code:

    
    
    urlStr = urllib.request.urlopen(
        'http://gatherer.wizards.com/Pages/Card/Details.aspx?name=' + cardName
        ).read()
    
    htmlRaw = BeautifulSoup(urlStr)
    
    htmlText = htmlRaw.get_text
    
    for line in htmlText:
        line = line.strip()
        if "Converted Mana Cost:" in line:
            cmc = line.next()
            message += "*Converted Mana Cost: " + cmc +"* \n\n"
        elif "Types:" in line:
            type = line.next()
            message += "*Type: " + type +"* \n\n"
        elif "Card Text:" in line:
            rulesText = line.next()
            message += "*Rules Text: " + rulesText +"* \n\n"
        elif "Flavor Text:" in line:
            flavor = line.next()
            message += "*Flavor Text: " + flavor +"* \n\n"
        elif "Rarity:" in line:
            rarity == line.next()
            message += "*Rarity: " + rarity +"* \n\n"
    


Answer: I am not familiar with BeautifulSoup any more but I ran this code - not to
give you a complete answer but to point you in the right direction

    
    
    import urllib
    from lxml import html
    mypage = urllib.urlopen('http://gatherer.wizards.com/Pages/Card/Details.aspx?multiverseid=264')
    dir(mypage)
    ['__doc__', '__init__', '__iter__', '__module__', '__repr__', 'close', 'code', 'fileno', 'fp', 'getcode', 'geturl', 'headers', 'info', 'next', 'read', 'readline', 'readlines', 'url']
    page = mypage.readlines()
    len(page)
    526
    page[0]
    '<?xml version="1.0" encoding="utf-8" ?>\r\n'
    string = ''.join([apage for apage in page])
    tree = html.fromstring(string)
    elements = [e for e in tree.iter()]
    for e in elements:
        if 'cardtextbox' in e.values():
            e, e.text_content()
     (<Element div at 0x31a7ba0>, 'Enchant creature')
     (<Element div at 0x31a7bf8>, "Enchanted creature has protection from red. This effect doesn't remove Red Ward.")
    

I clearly do not know what I am doing but I was poking at it.

It seemed to me that the values you are trying to identify are values of
attribute dictionaries and so I knew enough to figure out this much. It would
take some more poking at it to list all of the attributes you want to identify
but I think this should get you started.

