Python script searching for string in textfile and inserting another textfile before match I'm very new to Python, and far away from writing my own scripts. For my work with lilypond I needed a script that parses a text file, searches for a string and inserts another textfile before the match. I have been searching quite a lot for this kinda script and I did not find any. I ended up combining the snippets I found on here and other sites and came up with this script, which is working:

#!/usr/bin/env python

# usage:
# $ python thisfile.py text.txt searchstring insert.txt

import sys

f2 = open(sys.argv[3])
data = f2.read()
f2.close()

with open(sys.argv[1], "r+") as f1:
    a = [x.rstrip() for x in f1]
    index = 0
    for item in a:
        if item.startswith(sys.argv[2]):
            a.insert(index, data)
            break
        index += 1
    f1.seek(0)
    f1.truncate()
    for line in a:
        f1.write(line + "\n")


I also got a very detailed answer on Stack Overflow, telling what is actually going on in the code, before I was far away from understanding any detail.

What I got out of it so far is the following problem:


  If anything would go wrong with reading in the data from f1 or f2, f1.truncate() would delete the original content of f2, then not being able to (re)write the appropriate content the content would get lost. A much more secure way would be using some kind of temporary file, or at least moving the original content of f1 there before calling truncate().


I would be glad for any comments on this "problem", and any others if there are.
The problem is a general one in data processing, one that programmers have to think about all the time! When changing a file \$F\$ from \$A\$ to \$B\$ it's tempting to implement it like this:


Read \$A\$ from \$F\$.
Compute \$B\$ from \$A\$.
Delete \$F\$.
Write \$B\$ to \$F\$.


But we need to consider the possibility that something will go wrong. Maybe the user will type control-C on the keyboard and interrupt the program? Maybe there will be a power cut? Maybe the disk will not have enough room for \$B\$? If any of these things happened after step 3 and before step 4, then you would be left in a situation where \$F\$ contains neither \$A\$ nor \$B\$. So you have lost your data and can't get it back.

This is why we try to design systems so that operations are atomic — either they succeed completely or they fail completely. In this case we would use the following procedure:


Read \$A\$ from \$F\$.
Compute \$B\$ from \$A\$.
Write \$B\$ to a temporary location \$G\$.
Replace \$F\$ with \$G\$.


This works because operating systems (usually!) give us an atomic implementation of step 4. In Python we can use os.rename, where you can see that the documentation says:


  If successful, the renaming will be an atomic operation


In this design, if something goes wrong before step 4, the file \$F\$ still contains \$A\$, and so we haven't lost our data, and so we have a chance to fix the problem and try again.

So in this case, I'd write something like this (but this is not tested, so don't use it blindly!):

import os
import shutil
import sys
import tempfile

# usage:
# $ python thisfile.py text.txt searchstring insert.txt
text_file, searchstring, insert_file = sys.argv[1:]

with tempfile.NamedTemporaryFile('w', delete=False) as temp:
    with open(text_file) as f1:
        inserted = False # Have we inserted insert_file yet?
        for f1_line in f1:
            if not inserted and f1_line.startswith(searchstring):
                with open(insert_file) as f2:
                    for f2_line in f2:
                        temp.write(f2_line)
                inserted = True
            temp.write(f1_line)

os.rename(temp.name, text_file)


Here I've used the library function tempfile.NamedTemporaryFile to choose somewhere to put the temporary file.

 Update

So here's another reason why it's a good idea to make operations atomic — you might have made a programming error! The code I wrote above works correctly on my operating system (macOS) but as it says in the os.rename documentation:


  The operation may fail on some Unix flavors if src and dst are on different filesystems


So I'm guessing that you're on some kind of Linux system. On these systems you've got to ensure that the temporary file \$G\$ is on the same filesystem as \$F\$, and the only reliable way to do that is to put it in the same directory as \$F\$:

import os
import shutil
import sys
import tempfile

# usage:
# $ python thisfile.py text.txt searchstring insert.txt
text_file, searchstring, insert_file = sys.argv[1:]

# Directory and name of text_file.
dirname, basename = os.path.split(text_file)

# Create temporary file in same directory as text_file.
with tempfile.NamedTemporaryFile('w', dir=dirname, prefix=basename,
                                 delete=False) as temp:
    with open(text_file) as f1:
        inserted = False # Have we inserted insert_file yet?
        for f1_line in f1:
            if not inserted and f1_line.startswith(searchstring):
                with open(insert_file) as f2:
                    for f2_line in f2:
                        temp.write(f2_line)
                inserted = True
            temp.write(f1_line)

os.rename(temp.name, text_file)


Writing reliable code that works on different platforms is not easy!