title: Pyspark: How to convert a spark dataframe to json and save it as json file?
body: <p>I am trying to convert my pyspark sql dataframe to json and then save as a file.</p>
  
  <pre><code>df_final = df_final.union(join_df)
  </code></pre>
  
  <p>df_final contains the value as such:</p>
  
  <p><a href="https://i.stack.imgur.com/aredn.png" rel="noreferrer"><img src="https://i.stack.imgur.com/aredn.png" alt="enter image description here"></a></p>
  
  <p>I tried something like this. But it created a invalid json.</p>
  
  <pre><code>df_final.coalesce(1).write.format('json').save(data_output_file+"createjson.json", overwrite=True)
  
  {"Variable":"Col1","Min":"20","Max":"30"}
  {"Variable":"Col2","Min":"25,"Max":"40"}
  </code></pre>
  
  <p>My expected file should have data as below:</p>
  
  <pre><code>[
  {"Variable":"Col1",
  "Min":"20",
  "Max":"30"},
  {"Variable":"Col2",
  "Min":"25,
  "Max":"40"}]
  </code></pre>