os.path.exists() lies
<p>I'm running a number of python scripts on a linux cluster, and the output from one job is generally the input to another script, potentially run on another node. I find that there is some not insignificant lag before python notices files that have been created on other nodes -- os.path.exists() returns false and open() fails as well. I can do a while not os.path.exists(mypath) loop until the file appears, and it can take upwards of a full minute, which is not optimal in a pipeline with many steps and potentially running many datasets in parallel.</p>

<p>The only workaround I've found so far is to call subprocess.Popen("ls %s"%(pathdir), shell=True), which magically fixes the problem. I figure this is probably a system problem, but any way python might be causing this? Some sort of cache or something? My sysadmin hasn't been much help so far.</p>
