Handling French text Python
Question: I am trying to read some French text and do some frequency analysis of words.
I want the characters with the umlauts and other diacritics to stay. So, I did
this for testing:

    
    
    >>> import codecs
    >>> f = codecs.open('file','r','utf-8')
    >>> for line in f:
    ...     print line
    ...
    
    Faites savoir à votre famille que vous êtes en sécurité.
    

So far, so good. But, I have a list of French files which I iterate over in
the following way:

    
    
    import codecs,sys,os
    
    path = sys.argv[1]
    for f in os.listdir(path):
        french = codecs.open(os.path.join(path,f),'r','utf-8')
        for line in french:
            print line
    

Here, it gives the following error:

    
    
    rdholaki74: python TestingCodecs.py ../frenchResources | more
    Traceback (most recent call last):
      File "TestingCodecs.py", line 7, in <module>
        print line
    UnicodeEncodeError: 'ascii' codec can't encode character u'\xe0' in position 14: ordinal not in range(128)
    

Why is it that the same file throws up an error when passed as an argument and
not when given explicitly in the code?

Thanks.


Answer: Because you're misinterpreting the cause. The fact that you're piping the
output means that Python can't detect what encoding to use. If
[stdout](http://docs.python.org/library/sys.html#sys.stdout) is [not a
TTY](http://docs.python.org/library/stdtypes.html#file.isatty) then you'll
need to encode as UTF-8 manually before outputting.

