title: Pass packages to pyspark running on dataproc from airflow?
body: <p>We have an Airflow DAG that involves running a pyspark job on Dataproc. We need a jdbc driver during the job, which I'd normally pass to the dataproc submit command:</p>
  
  <pre><code>gcloud dataproc jobs submit pyspark \
  --cluster my-cluster \
  --properties spark.jars.packages=mysql:mysql-connector-java:6.0.6 \
  --py-files ...
  </code></pre>
  
  <p>But how can I do it with Airflow's DataProcPySparkOperator?</p>
  
  <p>For now we're adding this library to the cluster itself:</p>
  
  <pre><code>gcloud dataproc clusters create my-cluster \
    --region global \
    --zone europe-west1-d \
    ...
    --properties spark:spark.jars.packages=mysql:mysql-connector-java:6.0.6 \
    ...
  </code></pre>
  
  <p>This seems to be working fine, but it doesn't feel like the right way to do it. Is there another way?</p>