Substitution of subprocess.PIPE in Python?
Question: I am using subprocess module to interact with output of the linux commands.
below is my code.

    
    
    import subprocess
    import sys
    
    file_name = 'myfile.txt'
    p = subprocess.Popen("grep \"SYSTEM CONTROLLER\" "+ file_name, stdout=subprocess.PIPE, shell=True)
    (output, err) = p.communicate()
    print output.strip()
    
    p = subprocess.Popen("grep \"controller\|worker\" "+ file_name, stdout=subprocess.PIPE, shell=True)
    (output, err) = p.communicate()
    lines = output.rstrip().split("\n")
    print lines
    

My program hangs while executing second subprocess i.e.

    
    
    p = subprocess.Popen("grep \"controller\|worker\""+ file_name,stdout=subprocess.PIPE, shell=True)
    

I got to know that the reason of process hang is buffer redirected to
subprocess.PIPE is getting filled, which blocks the process from writing
further.

I want to know if there is any way to avoid the buffer full situation so that
my program keeps on executing without any hang issue ?


Answer: The actual issue is that there is a whitespace missing between the pattern and
the filename and therefore `grep` waits for input on the standard input
(stdin).

_"buffer full"_ (`.communicate()` is not susceptible) or `p.stdout.read()` (it
fixes nothing: it loads the output into memory and unlike `.communicate()` it
fails if more than one pipe is used) are a red herring here.

Drop `shell=True` and use a list argument for the command:

    
    
    #!/usr/bin/env python
    from subprocess import Popen, PIPE
    
    p = Popen(["grep", r"controller\|worker", file_name], stdout=PIPE)
    output = p.communicate()[0]
    if p.returncode == 0:
        print('found')
    elif p.returncode == 1:
        print('not found')
    else: 
        print('error')
    

