title: How to execute a long running subprocess inside celery task?
body: <p>I have the following code where I am running a shell script using subprocess inside a celery task. It's not working as in I don't get an error or any forward progress, or any output from the celery task:</p>
  
  <p>The following is the code to execute the task:</p>
  
  <pre><code>def run_shell_command(command_line):
      command_line_args = shlex.split(command_line)
  
      logging.info('Subprocess: "' + command_line + '"')
  
      try:
          command_line_process = subprocess.Popen(
              command_line_args,
              shell=True,
              stdout=subprocess.PIPE,
              stderr=subprocess.STDOUT,
          )
  
          for l in iter(command_line_process.stdout.readline,b''):
              print l.strip()
  
          command_line_process.communicate()
          command_line_process.wait()
  
      except (OSError, subprocess.CalledProcessError) as exception:
          logging.info('Exception occured: ' + str(exception))
          logging.info('Subprocess failed')
          return False
      else:
          # no exception was raised
          logging.info('Subprocess finished')
  
      return True
  </code></pre>
  
  <p>It's called from within a task:</p>
  
  <pre><code>@app.task
  def execute(jsonConfig, projectName, tagName, stage, description):
  
       command = 'python ' + runScript + ' -c ' + fileName
       run_shell_command(command)
  </code></pre>
  
  <p>Here the python "runScript" is in itself calling subprocesses, and executes a long running task. What could be the problem</p>
  
  <p>The logging level has been set to INFO :</p>
  
  <pre><code>logging.basicConfig(filename='celery-execution.log',level=logging.INFO)
  </code></pre>
  
  <p>The celery worker is started as follow:</p>
  
  <pre><code>celery -A celery_worker worker --loglevel=info
  </code></pre>
  
  <p>I can see the subprocess being started:</p>
  
  <pre><code>[2016-05-03 01:08:55,126: INFO/Worker-2] Subprocess: "python runScript.py -c data/confs/Demo-demo14-1.conf"
  </code></pre>
  
  <p>I can also see the subprocess running in the background using a ps -ef, however this is a compute/memory intensive workload and it does not seem to be actually using any cpu or memory which makes me believe that nothing is really happening and it's stuck.</p>