Python 3 argument (semi)not UTF-8 when passed from Windows batch.cmd
Question: When I invoke a Python 3 script from a Windows batch.cmd, a UTF-8 arg is not
passed as "UTF-8", but as a series of bytes, each of which are interpreted by
Python as individual UTF-8 chars.

How can I convert the Python 3 arg string to its **intended** UTF-8 state?  
The calling **.cmd** and the called **.py** are shown below.

PS. As I mention in a comment below, calling u00FF.py "ÿ" **directly** from
the Windows console commandline works fine. It is only a problem when I invoke
u00FF.cmd via the **.cmd** , and I am looking for a `Python 3` way to convert
the double-encoded UTF-8 arg back to a "normally" encoded UTF-8 form.

I've now include here, the full (and latest) test code.. Its a bit long, but I
hope it explains the issue clearly enough.

Update: I've seen why the file read of "ÿ" was "double-encoding"... I was
reading the UTF-8 file in binary/byte mode... I should have used
`codecs.open('u00FF.arg', 'r', 'utf-8')` instead of just plain
`open('u00FF.arg','r')`... I've updated the offending code, and the output.
The codepage issues seems to be the only problem now...

Because the Python issue has been largely resolved, and the codepage issue is
quite independent of Python, I have posted another codepage specific question
at  
[Codepage 850 works, 65001 fails! There is NO response to “call foo.cmd”.
internal commands work
fine.](http://stackoverflow.com/questions/3401802/codepage-850-works-65001-fails-
there-is-no-response-to-call-foo-cmd-intern%29)

    
    
    :::::::::::::::::::   BEGIN .cmd BATCH FILE ::::::::::::::::::::
    :: Windows Batch file (UTF-8 encoded, no BOM): "u00FF.cmd" 
       @echo ÿ>u00FF.arg
       @u00FF.py "ÿ"  
       @goto :eof  
    :::::::::::::::::::   END OF .cmd BATCH FILE ::::::::::::::::::::
    
    ################### BEGIN .py SCRIPT #####################################  
        # -*- coding: utf-8 -*-
    
        import sys
        print ("""
        Unicode
        =======
            CodePoint U+00FF
            Character ÿ __Unicode Character 'LATIN SMALL LETTER Y WITH DIAERESIS'
    
        UTF-8 bytes
        ===========
            Hex: \\xC3 \\xBF
            Dec:  195  191
            Char:   Ã    ¿ __Unicode Character 'INVERTED QUESTION MARK'
                     \_______Unicode Character 'LATIN CAPITAL LETTER A WITH TILDE' 
    
        """)
        print("## ====================================================")
        print("## ÿ via hard-coding in this .py script itself ========")
        print("##")
        hard1s = "ÿ"
        hard1b = hard1s.encode('utf_8')
        print("hard1s: len", len(hard1s), " '" + hard1s + "'")
        print("hard1b: len", len(hard1b),        hard1b)
        for i in range(0,len(hard1s)):
            print("CodePoint[", i, "]", hard1s[i], "U+"+"{0:x}".upper().format(ord(hard1s[i])).zfill(4) )
        print('''         This is a single CodePoint for "ÿ" (as expected).''')
        print()
        print("## ====================================================")
        print("## ÿ read into this .py script from a UTF-8 file ======")
        print("##")
        import codecs
        file1 = codecs.open( 'u00FF.arg', 'r', 'utf-8' )
        file1s = file1.readline()
        file1s = file1s[:1] # remove \r
        file1b = file1s.encode('utf_8')
        print("file1s: len", len(file1s), " '" + file1s + "'")
        print("file1b: len", len(file1b),        file1b)
        for i in range(0,len(file1s)):
            print("CodePoint[", i, "]", file1s[i], "U+"+"{0:x}".upper().format(ord(file1s[i])).zfill(4) )
        print('''         This is a single CodePoint for "ÿ" (as expected).''')
        print()
        print("## ====================================================")
        print("## ÿ via sys.argv  from a call to .py  from a .cmd) ===")
        print("##")
        argv1s = sys.argv[1]
        argv1b = argv1s.encode('utf_8')
        print("argv1s: len", len(argv1s), " '" + argv1s + "'")
        print("argv1b: len", len(argv1b),        argv1b)
        for i in range(0,len(argv1s)):
            print("CodePoint[", i, "]", argv1s[i], "U+"+"{0:x}".upper().format(ord(argv1s[i])).zfill(4) )
        print('''         These 2 CodePoints are way off-beam,
                         even allowing for the "double-encoding" seen above.
                         The CodePoints are from an entirely different Unicode-Block.
                         This must be a Codepage issue.''')
        print()
    ################### END OF .py SCRIPT #####################################  
    

Here is the output from the above code.

    
    
    ========================== BEGIN OUTPUT ================================
        C:\>u00FF.cmd
        Unicode
        =======
            CodePoint U+00FF
            Character ÿ __Unicode Character 'LATIN SMALL LETTER Y WITH DIAERESIS'
    
        UTF-8 bytes
        ===========
            Hex: \xC3 \xBF
            Dec:  195  191
            Char:   Ã    ¿ __Unicode Character 'INVERTED QUESTION MARK'
                     \_______Unicode Character 'LATIN CAPITAL LETTER A WITH TILDE'
    
    
        ## ====================================================
        ## ÿ via hard-coding in this .py script itself ========
        ##
        hard1s: len 1  'ÿ'
        hard1b: len 2 b'\xc3\xbf'
        CodePoint[ 0 ] ÿ U+00FF
                         This is a single CodePoint for "ÿ" (as expected).
    
        ## ====================================================
        ## ÿ read into this .py script from a UTF-8 file ======
        ##
        file1s: len 1  'ÿ'
        file1b: len 2 b'\xc3\xbf'
        CodePoint[ 0 ] ÿ U+00FF
                         This is a single CodePoint for "ÿ" (as expected
    
        ## ====================================================
        ## ÿ via sys.argv  from a call to .py  from a .cmd) ===
        ##
        argv1s: len 2  '├┐'
        argv1b: len 6 b'\xe2\x94\x9c\xe2\x94\x90'
        CodePoint[ 0 ] ├ U+251C
        CodePoint[ 1 ] ┐ U+2510
                         These 2 CodePoints are way off-beam,
                         even allowing for the "double-encoding" seen above.
                         The CodePoints are from an entirely different Unicode-Block.
                         This must be a Codepage issue.
    ========================== END OF OUTPUT ================================
    


Answer: Batch files and encodings are a finicky issue. First of all: Batch files have
no direct way of specifying the encoding they're in and `cmd` does not really
support Unicode batch files. You can easily see that if you save a batch file
with a Unicode BOM or as UTF-16 – they will throw an error.

What you see when you put the `ÿ` directly into the command line is that when
running a command Windows will initially use the command line as Unicode (it
may have been converted from some legacy encoding beforehand, but in the end
what Windows uses is Unicode). So Python will (hopefully) always grab the
Unicode content of the arguments.

However, since `cmd` has its own opinions about the codepage (and you never
told it to use UTF-8) the UTF-8 string you put in the batch file won't be
interpreted as UTF-8 but instead in the default `cmd` codepage (850 or 437, in
your case).

You can force UTF-8 with `chcp`:

    
    
    chcp 65001 > nul
    

You can save the following file as UTF-8 and try it out:

    
    
    @echo off
    chcp 850 >nul
    echo ÿ
    chcp 65001 >nul
    echo ÿ
    

Keep in mind, though, that the `chcp` setting will persist in the shell if you
run the batch from there which may make things weird.

