title: airflow cleared tasks not getting executed
body: <p><strong>Preamble</strong></p>
  
  <p>Yet another airflow tasks not getting executed question... </p>
  
  <p>Everything was going more or less fine in my airflow experience up until this weekend when things really went downhill.</p>
  
  <p>I have checked all the standard things e.g. as outlined in <a href="https://stackoverflow.com/questions/49021055/airflow-1-9-0-is-queuing-but-not-launching-tasks/49047832#49047832">this helpful post</a>.</p>
  
  <p>I have reset the whole instance multiple times trying to get it working properly but I am totally losing the battle here.</p>
  
  <p><strong>Environment</strong></p>
  
  <ul>
  <li>version: airflow 1.10.2 </li>
  <li>os: centos 7 </li>
  <li>python: python 3.6 </li>
  <li>virtualenv: yes </li>
  <li>executor: LocalExecutor </li>
  <li>backend db: mysql</li>
  </ul>
  
  <p><strong>The problem</strong></p>
  
  <p>Here's what happens in my troubleshooting infinite loop / recurring nightmare.</p>
  
  <ol>
  <li>I reset the metadata DB (or possibly the whole virtualenv and config etc) and re-enter connection information. </li>
  <li>Tasks will get executed once.  They may succeed.  If I missed something in the setup, a task may fail.</li>
  <li>When task fails, it goes to retry state. </li>
  <li>I fix the issue with (e.g. forgot to enter a connection) and manually clear the task instance. </li>
  <li>Cleared task instances do not run, but just sit in a "none" state</li>
  <li>Attempts to get dag running again fail.  </li>
  </ol>
  
  <p>Before I started having this trouble, after a cleared a task instance, it would always very quickly get picked up and executed again.</p>
  
  <p>But now, clearing the task instance usually results in the task instance getting stuck in a cleared state.  It just sits there. </p>
  
  <p>Worse, if I try failing the dag and all instances, and manually triggering the dag again, the task instances get created but stay in 'none' state.  Restarting scheduler doesn't help. </p>
  
  <p><strong>Other observation</strong></p>
  
  <p>This is probably a red herring, but one thing I have noticed only recently is that when I click on the icon representing the task instances stuck in the 'none' state, it takes me to a "task instances" view filter that has the wrong filter; the filter is set at "string equals null".  </p>
  
  <p>But you need to switch it to "string empty yes" in order to have it actually return the task instances that are stuck.</p>
  
  <p>I am assuming this is just an unrelated UI bug, a red herring as far as I am concerned, but I thought I'd mention it just in case.</p>
  
  <p><strong>Edit 1</strong></p>
  
  <p>I am noticing that there is some "null operator" going on:
  <a href="https://i.stack.imgur.com/HtAzo.png" rel="noreferrer"><img src="https://i.stack.imgur.com/HtAzo.png" alt="why is my operator null?  i will look into it"></a></p>
  
  <p><strong>Edit 2</strong></p>
  
  <p>Is <code>null</code> a valid value for task instance state?  Or is this an indicator that something is wrong.</p>
  
  <p><a href="https://i.stack.imgur.com/4VfQA.png" rel="noreferrer"><img src="https://i.stack.imgur.com/4VfQA.png" alt="Is it legit to have a null task instance state?"></a></p>
  
  <p><strong>Edit 3</strong> </p>
  
  <p>More <code>none</code> stuff.</p>
  
  <p>Here are some bits from the task instance details page.  Lots of attributes are <code>none</code>:</p>
  
  <pre><code>Task Instance Details
  Dependencies Blocking Task From Getting Scheduled
  Dependency  Reason
  Unknown All dependencies are met but the task instance is not running. In most cases this just means that the task will probably be scheduled soon unless:
  - The scheduler is down or under heavy load
  - The following configuration values may be limiting the number of queueable processes: parallelism, dag_concurrency, max_active_dag_runs_per_dag, non_pooled_task_slot_count
  - This task instance already ran and had its state changed manually (e.g. cleared in the UI)
  
  If this task instance does not start soon please contact your Airflow administrator for assistance.
  Task Instance Attributes
  Attribute   Value
  duration    None
  end_date    None
  is_premature    False
  job_id  None
  operator    None
  pid None
  queued_dttm None
  raw False
  run_as_user None
  start_date  None
  state   None
  </code></pre>
  
  <p><strong>Update</strong></p>
  
  <p>I may finally be on to something...</p>
  
  <p>After my <em>nightmarish</em>, marathon, stuck-in-twilight-zone troubleshooting session, I threw my hands up and resolved to use docker containers instead of running natively.  It was just too weird.  Things were just not making sense.  I needed to move to docker so that the environment could be completely controlled and reproduced.</p>
  
  <p>So I started working on the docker setup based on puckel/docker-airflow. This was no trivial task either, because I decided to use environment variables for all parameters and connections.  Not all hooks parse connection URIs the same way, so you have to be careful and look at the code and do some trial and error.</p>
  
  <p>So then I did that, I finally got my docker setup working locally.  But when I went to build the image on my EC2 instance, I found that the <strong>disk was full</strong>.  And it was in no small part due to airflow logs that it was full.</p>
  
  <p>So, my new theory is that lack of disk space may have had something to do with this.  I am not sure if I will be able to find a smoking gun in the logs, but I will look.</p>