title: How to run Spark code in Airflow?
body: <p>Hello people of the Earth!
  I'm using Airflow to schedule and run Spark tasks.
  All I found by this time is python DAGs that Airflow can manage.
  <br> DAG example:</p>
  
  <pre><code>spark_count_lines.py
  import logging
  
  from airflow import DAG
  from airflow.operators import PythonOperator
  
  from datetime import datetime
  
  args = {
    'owner': 'airflow'
    , 'start_date': datetime(2016, 4, 17)
    , 'provide_context': True
  }
  
  dag = DAG(
    'spark_count_lines'
    , start_date = datetime(2016, 4, 17)
    , schedule_interval = '@hourly'
    , default_args = args
  )
  
  def run_spark(**kwargs):
    import pyspark
    sc = pyspark.SparkContext()
    df = sc.textFile('file:///opt/spark/current/examples/src/main/resources/people.txt')
    logging.info('Number of lines in people.txt = {0}'.format(df.count()))
    sc.stop()
  
  t_main = PythonOperator(
    task_id = 'call_spark'
    , dag = dag
    , python_callable = run_spark
  )
  </code></pre>
  
  <p>The problem is I'm not good in Python code and have some tasks written in Java. My question is how to run Spark Java jar in python DAG? Or maybe there is other way yo do it? I found spark submit: <a href="http://spark.apache.org/docs/latest/submitting-applications.html" rel="noreferrer">http://spark.apache.org/docs/latest/submitting-applications.html</a>
  <br>But I don't know how to connect everything together. Maybe someone used it before and has working example. Thank you for your time!</p>