If process A copies files to some location loc and process B regularly copies the files from loc to some other location, can B read a file that is currently in the process of being copied to loc by A?I'm using Ubuntu Linux 12.04 if that's important.Background information: I want to continuously backup a PostgreSQL cluster. PostgreSQL provides WAL archiving for that. It works by having the database call a script that copies a completed WAL file to some backup location.I want another process to regularly copy the backed up WAL files to another server. If a WAL file is currently being copied by the database, can the second process still read the file without running into some EOF condition before the file is copied as a whole?In other words: Can I do the following with no synchronization between A and B?A                                   Bcp pg_xlog/some_wal_file /backup/   scp /backup/* user@remote-machine:/backups/
Can I safely read a file that is appended to by another process?
files;file copy;synchronization;lock
I think the best thing to is to ensure that process B only copies files which have been fully transferred by process A. One way to do this would be to use a combination of cp and mv in process A, since the mv process uses the rename system call (provided the files are on the same filesystem) which is atomic. This means that from the perspective of process B, the files appear in their fully formed state.One way to implement this would be to have a partial directory inside your /backup directory which is ignored by process B. For process A you could do something like:file=some_wal_filecp pg_xlog/$file /backup/partialmv /backup/partial/$file /backupAnd for process B (using bash):shopt -s extglobscp /backup/!(partial) user@remote-machine:/backups/Although the program that you probably want to look into, both for process A and process B,  is rsync. rsync creates partial files and atomically moves into place by default (although usually the partial files are hidden files rather than being in a specific directory). Rsync will also avoid transferring files that it doesn't need to and has a special delta algorithm for transferring only the relevant parts of files that need to be updated over the network (rsync must be installed in both locations, although transfers still go over ssh by default). Using rsync for process A:rsync -a --partial-dir=/backup/partial pg_xlog/some_wal_file /backup/For process B:rsync -a --exclude=/partial/ /backup/ user@remote-machine:/backups/