Does python make a copy of opened files in memory?
Question: So I would like to search for filenames with os.walk() and write the resulting
list of names to a file. I would like to know what is more efficient : opening
the file and then writing each result as I find them or storing everything in
a list and then writing the whole list. That list could be big so I wonder if
the second solution would work.


Answer: See this example:

    
    
    import os
    fil = open('/tmp/stuff', 'w')
    fil.write('aaa')
    os.system('cat /tmp/stuff')
    

You may expect to see `aaa`, but instead you get nothing. This is because
Python has an internal buffer. Writing to disk is expensive, as it has to:

  * Tell the OS to write it.
  * Actually transfer the data to the disk (on a hard disk it may involve spinning it up, waiting for IO time, etc.).
  * Wait for the OS to report success on the writing.

If you want to write any small things, it can add up to quite some time.
Instead, what Python does is to keep a buffer and only _actually_ write from
time to time. You don't have to worry about the memory growth, as it will be
kept at a low value. From the
[docs](https://docs.python.org/2/library/functions.html#open):

"0 means unbuffered, 1 means line buffered, any other positive value means use
a buffer of (approximately) that size (in bytes). A negative buffering means
to use the system default, which is usually line buffered for tty devices and
fully buffered for other files. If omitted, the system default is used."

When you are done, make sure you do a `fil.close()`, or `fil.flush()` at any
point during the execution, or use the keyword `buffering=0` to disable
buffering.

Another thing to consider is what happens if, for some reason, the program
exits in the middle of the process. If you store everything in memory, it will
be lost. What you have on disk, will remain there (but unless you flush, there
is no guarantee of how much was actually saved).

