title: Issues running airflow scheduler as a daemon process
body: <p>I have an EC2 instance that is running airflow 1.8.0 using <code>LocalExecutor</code>. Per the docs I would have expected that one of the following two commands would have raised the scheduler in daemon mode: </p>
  
  <p><code>airflow scheduler --daemon --num_runs=20</code></p>
  
  <p>or </p>
  
  <p><code>airflow scheduler --daemon=True --num_runs=5</code></p>
  
  <p>But that isn't the case. The first command seems like it's going to work, but it just returns the following output before returning to terminal without producing any background task: </p>
  
  <pre><code>[2017-09-28 18:15:02,794] {__init__.py:57} INFO - Using executor LocalExecutor
  [2017-09-28 18:15:03,064] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python3.5/lib2to3/Grammar.txt
  [2017-09-28 18:15:03,203] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python3.5/lib2to3/PatternGrammar.txt
  </code></pre>
  
  <p>The second command produces the error: </p>
  
  <pre><code>airflow scheduler: error: argument -D/--daemon: ignored explicit argument 'True'
  </code></pre>
  
  <p>Which is odd, because according to the <a href="https://airflow.incubator.apache.org/cli.html" rel="noreferrer">docs</a> <code>--daemon=True</code> should be a valid argument for the <code>airflow scheduler</code> call. </p>
  
  <p>Digging a little deeper took me to <a href="https://stackoverflow.com/questions/39383429/how-to-run-airflow-scheduler-as-a-daemon-process">this StackOverflow post</a>, where one of the responses recommends an implementation of <code>systemd</code> for handling the airflow scheduler as a background process according to the code available as <a href="https://github.com/apache/incubator-airflow/tree/master/scripts/systemd" rel="noreferrer">this repo</a>. </p>
  
  <p>My lightly-edited adaptations of the script are posted as the following Gists. I am using a vanilla m4.xlarge EC2 instance with Ubuntu 16.04.3: </p>
  
  <ul>
  <li><a href="https://gist.github.com/aaronpolhamus/7c06dcf844c6018d8ca185021b8cfb3f" rel="noreferrer">/etc/sysconfig/airflow</a></li>
  <li><a href="https://gist.github.com/aaronpolhamus/96a5840ceeb70aeaa206feef5de4f6c4" rel="noreferrer">/user/lib/systemd/system/airflow-scheduler.service</a></li>
  <li><a href="https://gist.github.com/aaronpolhamus/fa95f403d97ee0c5c3e8708985c5f265" rel="noreferrer">/etc/tmpfiles.d/airflow.conf</a></li>
  </ul>
  
  <p>From there I call: </p>
  
  <pre><code>sudo systemctl enable airflow-scheduler
  sudo systemctl start airflow-scheduler
  </code></pre>
  
  <p>And nothing happens. While I have much more complex DAGs running on this instance, <a href="https://gist.github.com/aaronpolhamus/4616738f353093d8bdd84b057e0fa1fe" rel="noreferrer">I am using this dummy case</a> to create a simple test that also serves as a listener to let me know when the scheduler is operating as planned. </p>
  
  <p>I've been using <code>journalctl -f</code> to debug. Here are a few lines of output from the scheduler process. There's no obvious problem, but my tasks aren't executing and no logs are being produced for the test DAG that would help me zoom in on the error. Is the problem in here somewhere? </p>
  
  <pre><code>Sep 28 18:39:30 ip-172-31-15-209 airflow[20603]: [2017-09-28 18:39:30,965] {dag_processing.py:627} INFO - Started a process (PID: 21822) to generate tasks for /home/ubuntu/airflow/dags/scheduler_test_dag.py - logging into /home/ubuntu/airflow/logs/scheduler/2017-09-28/scheduler_test_dag.py.log
  Sep 28 18:39:31 ip-172-31-15-209 airflow[20603]: [2017-09-28 18:39:31,016] {jobs.py:1002} INFO - No tasks to send to the executor
  Sep 28 18:39:31 ip-172-31-15-209 airflow[20603]: [2017-09-28 18:39:31,020] {jobs.py:1440} INFO - Heartbeating the executor
  Sep 28 18:39:32 ip-172-31-15-209 airflow[20603]: [2017-09-28 18:39:32,022] {jobs.py:1404} INFO - Heartbeating the process manager
  Sep 28 18:39:32 ip-172-31-15-209 airflow[20603]: [2017-09-28 18:39:32,023] {jobs.py:1440} INFO - Heartbeating the executor
  Sep 28 18:39:33 ip-172-31-15-209 airflow[20603]: [2017-09-28 18:39:33,024] {jobs.py:1404} INFO - Heartbeating the process manager
  Sep 28 18:39:33 ip-172-31-15-209 airflow[20603]: [2017-09-28 18:39:33,025] {dag_processing.py:559} INFO - Processor for /home/ubuntu/airflow/dags/capone_dash_dag.py finished
  Sep 28 18:39:33 ip-172-31-15-209 airflow[20603]: [2017-09-28 18:39:33,026] {dag_processing.py:559} INFO - Processor for /home/ubuntu/airflow/dags/scheduler_test_dag.py finished
  </code></pre>
  
  <p>When I run <code>airflow scheduler</code> manually this all works fine. Since my test DAG has a start date of September 9 it just keep backfilling every minute since then, producing a running time ticker. When I use <code>systemd</code> to run the scheduler as a deamon, however, it's totally quiet with no obvious source of the error. </p>
  
  <p>Any thoughts? </p>