Read Unicode characters from command-line arguments in Python 2.x on Windows
Question: I want my Python script to be able to read Unicode command line arguments in
Windows. But it appears that sys.argv is a string encoded in some local
encoding, rather than Unicode. How can I read the command line in full
Unicode?

Example code: `argv.py`

    
    
    import sys
    
    first_arg = sys.argv[1]
    print first_arg
    print type(first_arg)
    print first_arg.encode("hex")
    print open(first_arg)
    

On my PC set up for Japanese code page, I get:

    
    
    C:\temp>argv.py "PC・ソフト申請書08.09.24.doc"
    PC・ソフト申請書08.09.24.doc
    <type 'str'>
    50438145835c83748367905c90bf8f9130382e30392e32342e646f63
    <open file 'PC・ソフト申請書08.09.24.doc', mode 'r' at 0x00917D90>
    

That's Shift-JIS encoded I believe, and it "works" for that filename. But it
breaks for filenames with characters that aren't in the Shift-JIS character
set—the final "open" call fails:

    
    
    C:\temp>argv.py Jörgen.txt
    Jorgen.txt
    <type 'str'>
    4a6f7267656e2e747874
    Traceback (most recent call last):
      File "C:\temp\argv.py", line 7,
    in <module>
        print open(first_arg)
    IOError: [Errno 2] No such file or directory: 'Jorgen.txt'
    

Note—I'm talking about Python 2.x, not Python 3.0. I've found that Python 3.0
gives `sys.argv` as proper Unicode. But it's a bit early yet to transition to
Python 3.0 (due to lack of 3rd party library support).

**Update:**

A few answers have said I should decode according to whatever the `sys.argv`
is encoded in. The problem with that is that it's not full Unicode, so some
characters are not representable.

Here's the use case that gives me grief: I have [enabled drag-and-drop of
files onto .py files in Windows
Explorer](http://stackoverflow.com/q/142844/60075). I have file names with all
sorts of characters, including some not in the system default code page. My
Python script doesn't get the right Unicode filenames passed to it via
sys.argv in all cases, when the characters aren't representable in the current
code page encoding.

There is certainly some Windows API to read the command line with full Unicode
(and Python 3.0 does it). I assume the Python 2.x interpreter is not using it.


Answer: Here is a solution that is just what I'm looking for, making a call to the
Windows `GetCommandLineArgvW ` function:  
[Get sys.argv with Unicode characters under
Windows](http://code.activestate.com/recipes/572200/) (from ActiveState)

But I've made several changes, to simplify its usage and better handle certain
uses. Here is what I use:

**`win32_unicode_argv.py`**

    
    
    """
    win32_unicode_argv.py
    
    Importing this will replace sys.argv with a full Unicode form.
    Windows only.
    
    From this site, with adaptations:
          http://code.activestate.com/recipes/572200/
    
    Usage: simply import this module into a script. sys.argv is changed to
    be a list of Unicode strings.
    """
    
    
    import sys
    
    def win32_unicode_argv():
        """Uses shell32.GetCommandLineArgvW to get sys.argv as a list of Unicode
        strings.
    
        Versions 2.x of Python don't support Unicode in sys.argv on
        Windows, with the underlying Windows API instead replacing multi-byte
        characters with '?'.
        """
    
        from ctypes import POINTER, byref, cdll, c_int, windll
        from ctypes.wintypes import LPCWSTR, LPWSTR
    
        GetCommandLineW = cdll.kernel32.GetCommandLineW
        GetCommandLineW.argtypes = []
        GetCommandLineW.restype = LPCWSTR
    
        CommandLineToArgvW = windll.shell32.CommandLineToArgvW
        CommandLineToArgvW.argtypes = [LPCWSTR, POINTER(c_int)]
        CommandLineToArgvW.restype = POINTER(LPWSTR)
    
        cmd = GetCommandLineW()
        argc = c_int(0)
        argv = CommandLineToArgvW(cmd, byref(argc))
        if argc.value > 0:
            # Remove Python executable and commands if present
            start = argc.value - len(sys.argv)
            return [argv[i] for i in
                    xrange(start, argc.value)]
    
    sys.argv = win32_unicode_argv()
    

Now, the way I use it is simply to do:

    
    
    import sys
    import win32_unicode_argv
    

and from then on, `sys.argv` is a list of Unicode strings. The Python
`optparse` module seems happy to parse it, which is great.

