title: "WindowsError: [Error 206] The filename or extension is too long" after running a program very many times with subprocess
body: <p>My python program prepares inputs, runs an external FORTRAN code, and processes the outputs in a Windows HPC 2008 environment. It works great, unless the code executes the external program between 1042-1045 times (Usually the problem converges earlier). In these situations, I get an exception:</p>
  
  <blockquote>
    <p>WindowsError: [Error 206] The filename or extension is too long</p>
  </blockquote>
  
  <p>However, the path to the filename is <em>not</em> growing with time. It's just cleaning the directory and running again. </p>
  
  <p>Here's the code: </p>
  
  <pre><code>inpF = open(inName)
  outF = open(localOutName,'w')
  p = subprocess.Popen(pathToExe,shell=False,stdin=inpF,stdout=outF,cwd=runPath)
  stdout, stderr = p.communicate()
  outF.close()
  inpF.close()
  </code></pre>
  
  <p>pathToExe is a constant string pointing to a UNC location (e.g. \\server\shared\program.exe), stdin is an open file in read-only mode on a local drive, stdout is an open file in write mode on a local drive, and cwd is a local path on the C:\ drive. I have confirmed that none of the arguments to subprocess are longer than 80 characters, even though the limit is supposed to be 32,768, according to <a href="https://stackoverflow.com/questions/2381241/what-is-the-subprocess-popen-max-length-of-the-args-parameter">this somewhat related post</a>. </p>
  
  <p>What am I doing wrong? Somehow something is accumulating that only becomes a problem when I run over a thousand times.</p>
  
  <p>UPDATE:</p>
  
  <p>To test the "too many files open" hypothesis, I made a very small example that runs very quickly with a different executable. The main difference here is that the stdin and stdout are just empty files here, whereas in the previous case, they're both large files. In this case, the code runs just fine for 2000 runs, whereas the earlier fails at ~1042. So it's not just that there are that many files. Maybe there are too many large files open?</p>
  
  <pre><code>import subprocess
  for i in range(nRuns):
      if not (i % (nRuns/10.0)):
          print('{0:.2}% complete'.format(i/float(nRuns)*100))
      inpF=open('in.txt')
      outF=open('out.txt','w')
      p = subprocess.Popen('isotxsmerge.exe',shell=False,stdin=inpF,
                                   stdout=outF,cwd='.')
      stdout, stderr = p.communicate()
      outF.close()
      inpF.close()
  </code></pre>