title: Python Multiprocessing (ForkAwareLocal, FileNotFoundError) - Main thread closed too early for many processes
body: <p>I'm experiencing some difficulties with multiprocessing in Python. Using the snippet below I'm getting two errors:</p>
  <pre><code>AttributeError: 'ForkAwareLocal' object has no attribute 'connection'
  </code></pre>
  <p>and</p>
  <pre><code>FileNotFoundError: [Errno 2] No such file or directory
  </code></pre>
  <p>I've found that this is because the main process closes earlier than the sub-processes causing the subprocesses to be discarded, throwing these errors. I'm using <code>p.join()</code> as this should keep the main process waiting until the current process is done (if I understand correctly).</p>
  <p>When using a small number of processes, the code works perfectly fine (<code>n_processes = 15</code>) but when <code>n_processes</code> becomes bigger than around 100 (differs per machine), the errors pop up.</p>
  <p>I'm asking this question because it seems quite arbitrary when the main process closes too soon, and why the <code>p.join()</code> function is not doing what I'm expecting. I've read a ton of similar SO posts, plus documentation, but can't seem to figure it out. Could someone help me with this?</p>
  <p>A minimal example is as follows:</p>
  <pre><code>&quot;&quot;&quot;Test for multiprocessing some large function.&quot;&quot;&quot;
  
  from multiprocessing import Process, Manager
  from timeit import default_timer as timer
  import time
  
  
  def fun(d, i):
      &quot;&quot;&quot;Simply add some data to the shared list.&quot;&quot;&quot;
      time.sleep(2)
      d.append((&quot;workerbee&quot; + str(i), i))
  
  
  def main(n_processes):
      &quot;&quot;&quot;Let's do some arbitrary thing to test overall logic.&quot;&quot;&quot;
      print(&quot;Starting!&quot;)
      with Manager() as manager:
          data = manager.list()  # &lt;-- can be shared between processes.
          procs = []
  
          # Create n processes
          # if n is small =&gt; everything is fine
          # if n is large (&gt;100) =&gt; ForkAwareLocal object has no attr connection
          for i in range(n_processes):
              procs.append(Process(target=fun, args=(data, i)))
              procs[-1].start()
  
          for p in procs:
              p.join()
  
          print(f'Result in main: {data}')
  
  
  if __name__ == '__main__':
      # Choose number of processes
      # n_processes = 15
      n_processes = 150
  
      start = timer()
      main(n_processes)
      end = timer()
      print(f'elapsed time: {end - start}')
  
  </code></pre>