title: is it possible to share subprocess pipe between processes (celery workers)?
body: <p>I have following problem:
  I run celery with many workers. During celery startup I create few subprocesses:</p>
  
  <pre><code>proc = subprocess.Popen("program", stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, cwd=cwd)
  </code></pre>
  
  <p>I need these subprocesses to start and be used later by celery workers (repeatedly). So I save subprocesses to multiprocessing.Manager().dict() - something like pool...</p>
  
  <pre><code>pool = multiprocessing.Manager().dict()
  pool[proc_id] = proc
  </code></pre>
  
  <p>All subprocesses are accessible from celery workers, but they don't work - I found out that pipes are broken in a moment, when subprocess is shared via pool.
  So first question: is there any chance to share subprocess pipes between another processes (celery workers)?</p>
  
  <p>I also tried to save pipes to separated regular dict. Then when worker gets subprocess from pool, these pipes are wired to subprocess:</p>
  
  <pre><code>proc.stdin = dict_of_pipes[proc_id]
  </code></pre>
  
  <p>This solution sometimes works, but sometimes pipe is not found in dictionary - I quess because sharing regular dictionary between processes is not ok?</p>
  
  <p>As "program" you can imagine /bin/bash. Locking is solved, dictionaries are never accessed by more than 1 process at a time...</p>
  
  <p>Second question - is it possible to open new pipe to subprocess? (from any celery worker?) Or other solution?</p>