Using Beautiful Soup to extract Norwegian text from HTML files, losing the Norwegian characters
Question: I have a Python script that uses Beautiful Soup to extract the text from HTML
files in a directory. However, I'm having trouble getting the encoding to work
properly. At first I though there may be a problem with the HTML files
themselves. However, when I view the source of an HTML file in Notepad.exe, I
for instance see this: `Vi er her for deg, og du må gjerne ta kontakt med oss
på 815 32 000 eller på Facebook om du har noen spørsmål.`

However, when I view the same HTML file in Internet Explorer, I see this: `Vi
er her for deg, og du mÃ¥ gjerne ta kontakt med oss pÃ¥ 815 32 000 eller pÃ¥
Facebook om du har noen spÃ¸rsmÃ¥l.`

And, the Internet Explorer text is the same text that my Python script appends
into my text file. So, obviously the encoding is detectable, and it's no
surprise that IE doesn't understand it, but I can't seem to figure out why
Python can't handle it. The encoding should be latin-1, which I thought
wouldn't be a problem. Here's the code I have:

    
    
    import os
    import glob
    from bs4 import BeautifulSoup
    
    path = "c:\\users\\me\\downloads\\"
    
    for infile in glob.glob(os.path.join(path, "*.html")):
        markup = (infile)
        soup = BeautifulSoup(open(markup, "r").read())
        with open("example.txt", "a") as myfile:
            myfile.write(soup.get_text())
            myfile.close()
    

Being as that seems to break the encoding, I thought I could pass latin-1
encoding, like so:

    
    
    soup = BeautifulSoup(open(markup, "r").read())
    soup = soup.prettify("latin-1")
    

But that gives me the error:

    
    
    Traceback (most recent call last):
      File "bsoup.py", line 12, in <module>
        myfile.write(soup.get_text())
    AttributeError: 'bytes' object has no attribute 'get_text'
    


Answer: `.prettify()` already returns bytes, so you just write that directly to the
file, but you have to open that file in _binary_ mode (note the `'ab'` mode
used below):

    
    
    soup = BeautifulSoup(open(markup, "r").read())
    with open("example.txt", "ab") as myfile:
        myfile.write(soup.prettify('latin-1'))
    

There is no need to call `myfile.close()`; the `with` statement already takes
care of that.

To save only the text, open the file in text mode (`'a'`) and specify the
encoding to use on save:

    
    
    soup = BeautifulSoup(open(markup, "r").read())
    with open("example.txt", "a", encoding='latin-1') as myfile:
        myfile.write(soup.get_text())
    

Now Python will automatically encode the unicode text to latin-1 for you.

When you see something like `Ã¥` instead of `å`, then you are interpreting
UTF-8 bytes as Latin-1.

You may want to read up on Python and Unicode:

  * [The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)](http://joelonsoftware.com/articles/Unicode.html) by Joel Spolsky

  * The [Python Unicode HOWTO](http://docs.python.org/3/howto/unicode.html)

  * [Pragmatic Unicode](http://nedbatchelder.com/text/unipain.html) by Ned Batchelder

