Text processing program I have written a Python program to loop through a list of X files, open each one, read line by line, and write (append) to an output file. Being that these files are several GB each, it is taking very long.

I am looking for suggestions to improve the performance of this program. I have no formal CS training so it's likely I am missing the "obvious solution" to this problem; I have done some research but again, my limited knowledge (and other higher priority tasks) limits my ability to implement such.

for name in PR_files:
    with open(PR_path + name, 'r') as f:
        line = f.readline()
        while line:
            with open(PR_out_path, 'a') as g:
                g.write(line + '\n')
                line = f.readline()
    f.close()


The program will work but will have a blank line between each line in the output text file; this is because the first line of the next file began on the last line of the previous file (my solution to this problem was to add '\n' to each line being written to the output file. For that reason I wrote another block to remove all blank lines in the output file (yes I know, very inefficient).

# this removes all blank lines from out put file
with open(PR_out_path) as this, open(PR_out_path_fix, 'w') as that:
    for line in this:
        if not line.strip():
            continue
        that.write(line)


PLEASE NOTE: I attempted to do this, as oppose to reading line by line but I received a MemoryError.

with open(PR_out_path, 'a') as g:
    for name in PR_files:
        with open(PR_path + name, 'r') as f:
            g.write(f.read())
Your original code:

with open(PR_out_path, 'a') as g:
    for name in PR_files:
        with open(PR_path + name, 'r') as f:
            g.write(f.read())


works but, as you found, has problems if the entire file can't be read into memory.  The solution to that problem is to read the input file in chunks:

with open(PR_out_path, 'a') as g:
    for name in PR_files:
        with open(PR_path + name, 'r') as f:
            while True:
                data = f.read(ChunkSize)
                if not data:
                    break
                g.write(data)


where ChunkSize is something like 1GB.

But if speed is your only requirement why not use the tools offered by the operating system, as others have noted?