<p>I want to loop over the contents of a text file and do a search and replace on some lines and write the result back to the file. I could first load the whole file in memory and then write it back, but that probably is not the best way to do it.</p>

<p>What is the best way to do this, within the following code?</p>

<pre><code>f = open(file)
for line in f:
    if line.contains('foo'):
        newline = line.replace('foo', 'bar')
        # how to write this newline back to the file
</code></pre>

[{'answer_id': 39089, 'author': 'Lasse V. Karlsen', 'author_id': 267, 'author_profile': 'https://Stackoverflow.com/users/267', 'pm_score': 3, 'selected': False, 'text': '<p>Create a new file, copy lines from the old to the new, and do the replacing before you write the lines to the new file.</p>\n'}, {'answer_id': 39110, 'author': 'Thomas Watnedal', 'author_id': 4059, 'author_profile': 'https://Stackoverflow.com/users/4059', 'pm_score': 9, 'selected': True, 'text': "<p>I guess something like this should do it. It basically writes the content to a new file and replaces the old file with the new file:</p>\n\n<pre><code>from tempfile import mkstemp\nfrom shutil import move, copymode\nfrom os import fdopen, remove\n\ndef replace(file_path, pattern, subst):\n    #Create temp file\n    fh, abs_path = mkstemp()\n    with fdopen(fh,'w') as new_file:\n        with open(file_path) as old_file:\n            for line in old_file:\n                new_file.write(line.replace(pattern, subst))\n    #Copy the file permissions from the old file to the new file\n    copymode(file_path, abs_path)\n    #Remove original file\n    remove(file_path)\n    #Move new file\n    move(abs_path, file_path)\n</code></pre>\n"}, {'answer_id': 39113, 'author': 'hamishmcn', 'author_id': 3590, 'author_profile': 'https://Stackoverflow.com/users/3590', 'pm_score': 4, 'selected': False, 'text': '<p>As lassevk suggests, write out the new file as you go, here is some example code:</p>\n\n<pre><code>fin = open("a.txt")\nfout = open("b.txt", "wt")\nfor line in fin:\n    fout.write( line.replace(\'foo\', \'bar\') )\nfin.close()\nfout.close()\n</code></pre>\n'}, {'answer_id': 290494, 'author': 'Eli Bendersky', 'author_id': 8206, 'author_profile': 'https://Stackoverflow.com/users/8206', 'pm_score': 8, 'selected': False, 'text': '<p>The shortest way would probably be to use the <a href="https://docs.python.org/3/library/fileinput.html#fileinput.input" rel="noreferrer">fileinput module</a>. For example, the following adds line numbers to a file, in-place:</p>\n\n<pre><code>import fileinput\n\nfor line in fileinput.input("test.txt", inplace=True):\n    print(\'{} {}\'.format(fileinput.filelineno(), line), end=\'\') # for Python 3\n    # print "%d: %s" % (fileinput.filelineno(), line), # for Python 2\n</code></pre>\n\n<p>What happens here is:</p>\n\n<ol>\n<li>The original file is moved to a backup file</li>\n<li>The standard output is redirected to the original file within the loop</li>\n<li>Thus any <code>print</code> statements write back into the original file</li>\n</ol>\n\n<p><code>fileinput</code> has more bells and whistles. For example, it can be used to automatically operate on all files in <code>sys.args[1:]</code>, without your having to iterate over them explicitly. Starting with Python 3.2 it also provides a convenient context manager for use in a <code>with</code> statement.</p>\n\n<hr>\n\n<p>While <code>fileinput</code> is great for throwaway scripts, I would be wary of using it in real code because admittedly it\'s not very readable or familiar. In real (production) code it\'s worthwhile to spend just a few more lines of code to make the process explicit and thus make the code readable.</p>\n\n<p>There are two options:</p>\n\n<ol>\n<li>The file is not overly large, and you can just read it wholly to memory. Then close the file, reopen it in writing mode and write the modified contents back.</li>\n<li>The file is too large to be stored in memory; you can move it over to a temporary file and open that, reading it line by line, writing back into the original file. Note that this requires twice the storage.</li>\n</ol>\n'}, {'answer_id': 315088, 'author': 'Jason', 'author_id': 26860, 'author_profile': 'https://Stackoverflow.com/users/26860', 'pm_score': 7, 'selected': False, 'text': '<p>Here\'s another example that was tested, and will match search &amp; replace patterns:</p>\n\n<pre><code>import fileinput\nimport sys\n\ndef replaceAll(file,searchExp,replaceExp):\n    for line in fileinput.input(file, inplace=1):\n        if searchExp in line:\n            line = line.replace(searchExp,replaceExp)\n        sys.stdout.write(line)\n</code></pre>\n\n<p>Example use:</p>\n\n<pre><code>replaceAll("/fooBar.txt","Hello\\sWorld!$","Goodbye\\sWorld.")\n</code></pre>\n'}, {'answer_id': 1388570, 'author': 'Kinlan', 'author_id': 1798387, 'author_profile': 'https://Stackoverflow.com/users/1798387', 'pm_score': 6, 'selected': False, 'text': '<p>This should work: (inplace editing)</p>\n\n<pre><code>import fileinput\n\n# Does a list of files, and\n# redirects STDOUT to the file in question\nfor line in fileinput.input(files, inplace = 1): \n      print line.replace("foo", "bar"),\n</code></pre>\n'}, {'answer_id': 11784227, 'author': 'loi', 'author_id': 1572353, 'author_profile': 'https://Stackoverflow.com/users/1572353', 'pm_score': 1, 'selected': False, 'text': "<p>if you remove the indent at the like below, it will search and replace in multiple line.\nSee below for example.</p>\n\n<pre><code>def replace(file, pattern, subst):\n    #Create temp file\n    fh, abs_path = mkstemp()\n    print fh, abs_path\n    new_file = open(abs_path,'w')\n    old_file = open(file)\n    for line in old_file:\n        new_file.write(line.replace(pattern, subst))\n    #close temp file\n    new_file.close()\n    close(fh)\n    old_file.close()\n    #Remove original file\n    remove(file)\n    #Move new file\n    move(abs_path, file)\n</code></pre>\n"}, {'answer_id': 13641746, 'author': 'Thijs', 'author_id': 1865688, 'author_profile': 'https://Stackoverflow.com/users/1865688', 'pm_score': 5, 'selected': False, 'text': "<p>Based on the answer by Thomas Watnedal. \nHowever, this does not answer the line-to-line part of the original question exactly. The function can still replace on a line-to-line basis </p>\n\n<p>This implementation replaces the file contents without using temporary files, as a consequence file permissions remain unchanged.</p>\n\n<p>Also re.sub instead of replace, allows regex replacement instead of plain text replacement only.</p>\n\n<p>Reading the file as a single string instead of line by line allows for multiline match and replacement.</p>\n\n<pre><code>import re\n\ndef replace(file, pattern, subst):\n    # Read contents from file as a single string\n    file_handle = open(file, 'r')\n    file_string = file_handle.read()\n    file_handle.close()\n\n    # Use RE package to allow for replacement (also allowing for (multiline) REGEX)\n    file_string = (re.sub(pattern, subst, file_string))\n\n    # Write contents to file.\n    # Using mode 'w' truncates the file.\n    file_handle = open(file, 'w')\n    file_handle.write(file_string)\n    file_handle.close()\n</code></pre>\n"}, {'answer_id': 18676598, 'author': 'Kiran', 'author_id': 959654, 'author_profile': 'https://Stackoverflow.com/users/959654', 'pm_score': 4, 'selected': False, 'text': '<p>A more pythonic way would be to use context managers like the code below:</p>\n\n<pre><code>from tempfile import mkstemp\nfrom shutil import move\nfrom os import remove\n\ndef replace(source_file_path, pattern, substring):\n    fh, target_file_path = mkstemp()\n    with open(target_file_path, \'w\') as target_file:\n        with open(source_file_path, \'r\') as source_file:\n            for line in source_file:\n                target_file.write(line.replace(pattern, substring))\n    remove(source_file_path)\n    move(target_file_path, source_file_path)\n</code></pre>\n\n<p>You can find the full snippet <a href="https://gist.github.com/kirang89/6478017" rel="noreferrer">here</a>.</p>\n'}, {'answer_id': 21857132, 'author': 'starryknight64', 'author_id': 1476057, 'author_profile': 'https://Stackoverflow.com/users/1476057', 'pm_score': 4, 'selected': False, 'text': '<p>If you\'re wanting a generic function that replaces <em>any</em> text with some other text, this is likely the best way to go, particularly if you\'re a fan of regex\'s:</p>\n\n<pre><code>import re\ndef replace( filePath, text, subs, flags=0 ):\n    with open( filePath, "r+" ) as file:\n        fileContents = file.read()\n        textPattern = re.compile( re.escape( text ), flags )\n        fileContents = textPattern.sub( subs, fileContents )\n        file.seek( 0 )\n        file.truncate()\n        file.write( fileContents )\n</code></pre>\n'}, {'answer_id': 23123426, 'author': 'Emmanuel', 'author_id': 3543537, 'author_profile': 'https://Stackoverflow.com/users/3543537', 'pm_score': 2, 'selected': False, 'text': '<p>Using hamishmcn\'s answer as a template I was able to search for a line in a file that match my regex and replacing it with empty string.</p>\n\n<pre><code>import re \n\nfin = open("in.txt", \'r\') # in file\nfout = open("out.txt", \'w\') # out file\nfor line in fin:\n    p = re.compile(\'[-][0-9]*[.][0-9]*[,]|[-][0-9]*[,]\') # pattern\n    newline = p.sub(\'\',line) # replace matching strings with empty string\n    print newline\n    fout.write(newline)\nfin.close()\nfout.close()\n</code></pre>\n'}, {'answer_id': 23426834, 'author': 'igniteflow', 'author_id': 343223, 'author_profile': 'https://Stackoverflow.com/users/343223', 'pm_score': 3, 'selected': False, 'text': "<p>Expanding on @Kiran's answer, which I agree is more succinct and Pythonic, this adds codecs to support the reading and writing of UTF-8:</p>\n\n<pre><code>import codecs \n\nfrom tempfile import mkstemp\nfrom shutil import move\nfrom os import remove\n\n\ndef replace(source_file_path, pattern, substring):\n    fh, target_file_path = mkstemp()\n\n    with codecs.open(target_file_path, 'w', 'utf-8') as target_file:\n        with codecs.open(source_file_path, 'r', 'utf-8') as source_file:\n            for line in source_file:\n                target_file.write(line.replace(pattern, substring))\n    remove(source_file_path)\n    move(target_file_path, source_file_path)\n</code></pre>\n"}, {'answer_id': 58217364, 'author': 'Akif', 'author_id': 950762, 'author_profile': 'https://Stackoverflow.com/users/950762', 'pm_score': 4, 'selected': False, 'text': '<p><a href="https://docs.python.org/3/library/fileinput.html" rel="nofollow noreferrer"><code>fileinput</code></a> is quite straightforward as mentioned on previous answers:</p>\n<pre><code>import fileinput\n\ndef replace_in_file(file_path, search_text, new_text):\n    with fileinput.input(file_path, inplace=True) as file:\n        for line in file:\n            new_line = line.replace(search_text, new_text)\n            print(new_line, end=\'\')\n</code></pre>\n<p>Explanation:</p>\n<ul>\n<li><code>fileinput</code> can accept multiple files, but I prefer to close each single file as soon as it is being processed. So placed single <code>file_path</code> in <code>with</code> statement.</li>\n<li><code>print</code> statement does not print anything when <code>inplace=True</code>, because <code>STDOUT</code> is being forwarded to the original file.</li>\n<li><code>end=\'\'</code> in <code>print</code> statement is to eliminate intermediate blank new lines.</li>\n</ul>\n<p>You can used it as follows:</p>\n<pre><code>file_path = \'/path/to/my/file\'\nreplace_in_file(file_path, \'old-text\', \'new-text\')\n</code></pre>\n'}]
['https://Stackoverflow.com/questions/39086', 'https://Stackoverflow.com', 'https://Stackoverflow.com/users/4166/']