title: How to save a spark DataFrame back into a Google BigQuery project using pyspark?
body: <p>I am loading a dataset from BigQuery and after some transformations, I'd like to save the transformed DataFrame back into BigQuery. Is there a way of doing this?</p>
  
  <p>This is how I am loading the data:</p>
  
  <pre><code>df = spark.read \
    .format('bigquery') \
    .option('table', 'publicdata.samples.shakespeare') \
    .load()
  </code></pre>
  
  <p>Some transformations:</p>
  
  <pre><code> df_new = df.select("word")
  </code></pre>
  
  <p>And this I how I am trying to save the data as a new table into my project area:</p>
  
  <pre><code>df_new \
  .write \
  .mode('overwrite') \
  .format('bigquery') \
  .save('my_project.some_schema.df_new_table')
  </code></pre>
  
  <p>Is this even possible? Is there a way to save to BQ directly? </p>
  
  <p>ps: I know this works but this is not exactly what I am looking for:</p>
  
  <pre><code>df_new \
  .write \
  .mode('overwrite') \
  .format('csv') \
  .save('gs://my_bucket/df_new.csv')
  </code></pre>
  
  <p>Thanks!</p>