title: AirflowException: Celery command failed - The recorded hostname does not match this instance's hostname
body: <p>I'm running Airflow on a clustered environment running on two AWS EC2-Instances. One for master and one for the worker. The worker node though periodically throws this error when running "$airflow worker":</p>
  
  <pre><code>[2018-08-09 16:15:43,553] {jobs.py:2574} WARNING - The recorded hostname ip-1.2.3.4 does not match this instance's hostname ip-1.2.3.4.eco.tanonprod.comanyname.io
  Traceback (most recent call last):
    File "/usr/bin/airflow", line 27, in &lt;module&gt;
      args.func(args)
    File "/usr/local/lib/python3.6/site-packages/airflow/bin/cli.py", line 387, in run
      run_job.run()
    File "/usr/local/lib/python3.6/site-packages/airflow/jobs.py", line 198, in run
      self._execute()
    File "/usr/local/lib/python3.6/site-packages/airflow/jobs.py", line 2527, in _execute
      self.heartbeat()
    File "/usr/local/lib/python3.6/site-packages/airflow/jobs.py", line 182, in heartbeat
      self.heartbeat_callback(session=session)
    File "/usr/local/lib/python3.6/site-packages/airflow/utils/db.py", line 50, in wrapper
      result = func(*args, **kwargs)
    File "/usr/local/lib/python3.6/site-packages/airflow/jobs.py", line 2575, in heartbeat_callback
      raise AirflowException("Hostname of job runner does not match")
  airflow.exceptions.AirflowException: Hostname of job runner does not match
  [2018-08-09 16:15:43,671] {celery_executor.py:54} ERROR - Command 'airflow run arl_source_emr_test_dag runEmrStep2WaiterTask 2018-08-07T00:00:00 --local -sd /var/lib/airflow/dags/arl_source_emr_test_dag.py' returned non-zero exit status 1.
  [2018-08-09 16:15:43,681: ERROR/ForkPoolWorker-30] Task airflow.executors.celery_executor.execute_command[875a4da9-582e-4c10-92aa-5407f3b46d5f] raised unexpected: AirflowException('Celery command failed',)
  Traceback (most recent call last):
    File "/usr/local/lib/python3.6/site-packages/airflow/executors/celery_executor.py", line 52, in execute_command
      subprocess.check_call(command, shell=True)
    File "/usr/lib64/python3.6/subprocess.py", line 291, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command 'airflow run arl_source_emr_test_dag runEmrStep2WaiterTask 2018-08-07T00:00:00 --local -sd /var/lib/airflow/dags/arl_source_emr_test_dag.py' returned non-zero exit status 1.
  
  During handling of the above exception, another exception occurred:
  
  Traceback (most recent call last):
    File "/usr/lib/python3.6/dist-packages/celery/app/trace.py", line 382, in trace_task
      R = retval = fun(*args, **kwargs)
    File "/usr/lib/python3.6/dist-packages/celery/app/trace.py", line 641, in __protected_call__
      return self.run(*args, **kwargs)
    File "/usr/local/lib/python3.6/site-packages/airflow/executors/celery_executor.py", line 55, in execute_command
      raise AirflowException('Celery command failed')
  airflow.exceptions.AirflowException: Celery command failed
  </code></pre>
  
  <p>When this error occurs the task is marked as failed on Airflow and thus fails my DAG when nothing actually went wrong in the task.</p>
  
  <p>I'm using Redis as my queue and postgreSQL as my meta-database. Both are external as AWS services. I'm running all of this on my company environment which is why the full name of the server is <code>ip-1.2.3.4.eco.tanonprod.comanyname.io</code>. It looks like it wants this full name somewhere but I have no idea where I need to fix this value so that it's getting <code>ip-1.2.3.4.eco.tanonprod.comanyname.io</code> instead of just <code>ip-1.2.3.4</code>.</p>
  
  <p><strong>The really weird thing about this issue is that it doesn't always happen.</strong> It seems to just randomly happen every once in a while when I run the DAG. It's also occurring on all of my DAGs sporadically so it's not just one DAG. I find it strange though how it's sporadic because that means other task runs are handling the IP address for whatever this is just fine.</p>
  
  <p><strong>Note:</strong> I've changed the real IP address to 1.2.3.4 for privacy reasons.</p>
  
  <p><strong>Answer:</strong></p>
  
  <p><a href="https://github.com/apache/incubator-airflow/pull/2484" rel="noreferrer">https://github.com/apache/incubator-airflow/pull/2484</a></p>
  
  <p>This is exactly the problem I am having and other Airflow users on AWS EC2-Instances are experiencing it as well.</p>