how to split a file of code into words, despite for string variables
Question: Using python, how can I split a file, containing for example a code with
methods, variables, etc. into words but leave the code's string variables as
one unit string?

For example: given the following python code inside a file:

    
    
    def example():
        a = 5
        b = "Hello World"
    

The result should be:

    
    
    ['def', 'example', '(', ')', ':', 'a', '=', '5', 'b', '=', '"Hello World"']
    

where "Hello World" is as one single token.

Thanks...


Answer: You can use the [shlex](https://docs.python.org/2/library/shlex.html) module.

Example, for the fule:

Take the following text:

    
    
    This string has embedded "double quotes" and 'single quotes' in it,
    and even "a 'nested example'".
    

Using the `shlex` library, we construct a simple lexical parser:

    
    
    import shlex
    import sys
    
    if len(sys.argv) != 2:
        print 'Please specify one filename on the command line.'
        sys.exit(1)
    
    filename = sys.argv[1]
    body = file(filename, 'rt').read()
    print 'ORIGINAL:', repr(body)
    print
    
    print 'TOKENS:'
    lexer = shlex.shlex(body)
    for token in lexer:
        print repr(token)
    

This generates the output:

    
    
    ORIGINAL: 'This string has embedded "double quotes" and \'single quotes\' in it,\nand even "a \'nested example\'".\n'
    
    TOKENS:
    'This'
    'string'
    'has'
    'embedded'
    '"double quotes"'
    'and'
    "'single quotes'"
    'in'
    'it'
    ','
    'and'
    'even'
    '"a \'nested example\'"'
    '.'
    

More information and a nice tutorial can be found
[here](https://pymotw.com/2/shlex/).

