Python File Concatenation
Question: I have a data folder, with subfolders for each subject that ran through a
program. So, for example, in the data folder, there are folders for Bob, Fred,
and Tom. Each one of those folders contains a variety of files and subfolders.
However, I am only interested in the 'summary.log' file contained in each
subject's folder.

I want to concatenate the 'summary.log' file from Bob, Fred, and Tom into a
single log file in the data folder. In addition, I want to add a column to
each log file that will list the subject number.

Is this possible to do in Python? Or is there an easier way to do it? I have
tried a number of different batches of code, but none of them get the job
done. For example,

    
    
    #!/usr/bin/python
    
    import sys, string, glob, os
    
    fls = glob.glob(r'/Users/slevclab/Desktop/Acceptability Judgement Task/data/*');
    outfile = open('summary.log','w');
    for x in fls:
             file=open(x,'r');
             data=file.read();
             file.close();
             outfile.write(data);
    outfile.close();
    

Gives me the error,

    
    
    Traceback (most recent call last):
      File "fileconcat.py", line 8, in <module>
        file=open(x,'r');
    IOError: [Errno 21] Is a directory
    

I think this has to do with the fact that the data folder contains subfolders,
but I don't know how to work around it. I also tried this, but to no avail:

    
    
    from glob import iglob
    import shutil
    import os
    
    PATH = r'/Users/slevclab/Desktop/Acceptability Judgement Task/data/*'
    
    destination = open('summary.log', 'wb')
    for filename in iglob(os.path.join(PATH, '*.log'))
        shutil.copyfileobj(open(filename, 'rb'), destination)
    destination.close()
    

This gives me an "invalid syntax" error at the "for filename" line, but I'm
not sure what to change.


Answer: The syntax is not related to the use of glob. You forget the ":" at the end of
the for statement:

    
    
    for filename in iglob(os.path.join(PATH, '*.log')):
                                                      ^--- missing
    

But the following pattern works :

    
    
    PATH = r'/Users/slevclab/Desktop/Acceptability Judgement Task/data/*/*.log'
    
    destination = open('summary.log', 'wb')
    for filename in iglob(PATH):
        shutil.copyfileobj(open(filename, 'rb'), destination)
    destination.close()
    

