How do I correctly pass double quotes to an awk subprocess in Python?
Question: I am trying to run simple awk shell command and capture its output (using
python2). Here is what I try to do:

    
    
    import subprocess as sb
    
    shell = ["awk '!/<tag>/ {print \"\\"\"$1\"\\"\", \"\\"\"$2\"\\"\"}' test.txt"]
    p = sb.check_output(shell, shell=True)
    print p
    

`test.txt` content:

    
    
    a, b, 5
    a, c, 3
    d, d, 1
    

I want the to get the following output with awk and store it into a variable:

    
    
    "a" "b"
    "a" "c"
    "d" "d"
    

However I obviously lack the knowledge of how to properly handle **double
quotes**. I tried escaping them with several backsplashes, it all didn't work.
How to correctly escape the double quotes so that the example above work?


Answer: When you use `shell=True` but pass a list, you're asking Python to merge your
list of strings together as if they were separate arguments. That means it may
do its own quoting, on top of whatever quoting you did, in hopes that the
shell will reverse things properly. This is going to be a nightmare to get
right. If you want to use `shell=True`, just pass a string.

But that raises the question of why you're using `shell=True` in the first
place. If you didn't use this, you could just pass a list of arguments,
without having to quote any of them to protect them from the shell. Much
easier to write, and easier to debug, and more efficient and more secure to
boot. Unless you actually need shell features, or you've got a command line
that you worked hard to get working and don't want to spend time breaking down
into separate arguments, never use the shell.

* * *

I'm not actually sure what `awk` command you're _trying_ to run here. If you
give it the double-quotes around `$1` and `$2` it's just going to print a
literal `"$1" "$2"`, because that's what quotes mean to awk. Maybe you wanted
something like this?

    
    
    awk '!/<tag>/ {print "\""$1"\"", "\""$2"\""}' test.txt
    

In which case:

    
    
    subprocess.check_output(['awk', r'!/<tag>/ {print "\""$1"\"", "\""$2"\""}', 
                             'test.txt'])
    

(Note that I used a raw string so I could pass the `"\""` literally, without
having to backslash the backslash.)

But this still doesn't provide your desired output, because `$1` is going to
be `a,`, so `"\""$1"\""` is going to be `"a,"`.

