Utf-8 issues with python and nltk
Question: I have this little script, which is basically just a test I'm doing for a
larger program. I have a problem with the encoding. When I try to write to
file the utf-8 characters, such as øæå, are not encoded properly. Why is that,
and how can I solve this issue?

    
    
    #!/usr/bin/python
    # -*- coding: utf-8 -*-
    
    import nltk
    from nltk.collocations import *
    
    collocations = open('bam.txt', 'w')
    bigram_measures = nltk.collocations.BigramAssocMeasures()
    tokens = nltk.wordpunct_tokenize("Hei på deg øyerusk, du er meg en gammel dust, neida neida, det er ikke helt sant da."
                                     "Men du, hvorfor så brusk, ikke klut i din susk på en runkete lust")
    finder = BigramCollocationFinder.from_words(tokens)
    # finder.apply_freq_filter(3)
    scored = finder.score_ngrams(bigram_measures.raw_freq)
    for i in scored:
        print i[0][0] + ' ' + i[0][1] + ': ' + str(i[1]) + '\n'
        collocations.write(i[0][0] + ' ' + i[0][1] + ': ' + str(i[1]) + '\n')
    
    collocations.close()
    


Answer: The thing is `nltk.wordpunct_tokenize` doesn't work with non-ascii data. It is
better to use `PunktWordTokenizer` from `nltk.tokenize.punkt`. So import is
as:

    
    
    from nltk.tokenize.punkt import PunktWordTokenizer as PT
    

and replace:

    
    
    tokens = nltk.wordpunct_tokenize("Hei på deg øyerusk, du er meg en gammel dust, neida neida, det er ikke helt sant da."
                                 "Men du, hvorfor så brusk, ikke klut i din susk på en runkete lust")
    

with:

    
    
    tokens = PT().tokenize("Hei på deg øyerusk, du er meg en gammel dust, neida neida, det er ikke helt sant da." 
                                 "Men du, hvorfor så brusk, ikke klut i din susk på en runkete lust")
    

