Django - make file I/O thread safe
Question: I want to read and write python-source-files from the file system in a thread-
safe way.

    
    
    open("n2.py","w").write(my_new_py_class)
    from myproject import n2
    #do something with n2
    

I assume that this is not thread-safe, since a request2 could modify the file
before request1 is loading and executing it. I would like to achieve something
like that one thread is waiting till the other thread wrote, loaded, executed
and closed the file.


Answer: Why are you making your application modify its own files? This is not only
incredibly evil, metaprogramming is orders of magnitude harder to understand
debug. Plus, python caches modules it imports, so it's not really easy to
reload that. And, last but not least, you don't have to writ the code to a
file to execute it, if you really must execute dynamically generated code.

To answer your question about writing files in a thread safe way, the general
convention is to:

  * Write your content to a temporary file on the same filesystem as your target file.
  * Rename that temporary file to your target file, overwritting it in the process.

This works, because rename is atomic on POSIX systems, when done on the same
device. So other threads/processes will either still have the old file opened,
which is now detached from the filesystem and will be deleted as soon as those
threads/processes are done with it, or they will open the new file with all of
its contents. You avoid having a file that is only half-written.

In practice, I like to make a temporary directory with python's tempfile
module, and write the file in there, then move it and remove the directory --
this way the file is being created with default umask.

Last but not least, rename is not really atomic on Windows, at least with
default settings, as it won't let you overwrite your old file -- you need to
do two renames, and that introduces a possibility of race condition. I don't
know a good solution for Windows.

