Python how to direct subprocess to output file
Question: I have created a script that receives a bunch of log files as input in order
to do some pattern matching. However, my "processFiles" method is not working
properly. It should write all data to "fileDest". But the created file remains
empty. If I put a "print processFiles" statement under the function itself I
can see lines as throughput on my terminal.

To make matters more interesting, bunzip2 reports this error while I run the
script:

* * *

Proccessing /opt/syslog/app/applog-20140314.bz2. bunzip2: Can't open input
file > : No such file or directory. bunzip2: Compressed file ends
unexpectedly; perhaps it is corrupted? *Possible reason follows. bunzip2: No
such file or directory Input file =
/var/tmp/parsed_applog-20140314.decompressed, output file = (stdout)

* * *

It seems something in my code sends the output to stdout. And not to the file.

    
    
    def parse_log_files(self):
        sub_dir = os.listdir(self.base_path)
        for directory in sub_dir:
            if re.search('app\d+', directory):
                fileInput = self.base_path + '/' + directory + '/applog-' + str(self.date.strftime('%Y%m%d')) + '.bz2'
                fileDest = '/var/tmp/parsed_log_files-'  + str(self.date.strftime('%Y%m%d')) + '.decompressed'
                if not os.path.isfile(fileDest):
                    subprocess.Popen(['touch',fileDest]).communicate()[0]
                proccessFiles = subprocess.Popen(['/bin/bunzip2','-cd',fileInput,' > ',fileDest],stdout=subprocess.PIPE).communicate()[0]
                accessFileHandle =  open(self.file_out, 'r')
                readFileHandle = accessFileHandle.readlines()
                print "Proccessing %s." % fileInput
    


Answer: `' > '` is a shell redirection syntax. `Popen` doesn't spawn the shell unless
you ask (you shouldn't). If you want to redirect the output of a subprocess to
a file then use `stdout=file_object` parameter e.g.:

    
    
    from subprocess import check_call
    
    with open('/path/to/output', 'wb', 0) as output_file:
        check_call(['command', 'arg1', 'arg2'], stdout=output_file)
    

