title: How to save a spark dataframe to csv on HDFS?
body: <p>Spark version: 1.6.1, I use pyspark API.</p>
  
  <p>DataFrame: df, which has two colume.</p>
  
  <p>I have tried:</p>
  
  <pre><code>1: df.write.format('csv').save("hdfs://path/bdt_sum_vol.csv")
  2: df.write.save('hdfs://path/bdt_sum_vol.csv', format='csv', mode='append')
  3: df.coalesce(1).write.format('com.databricks.spark.csv').options(header='true').save('hdfs://path/')
  4: df.write.format('com.databricks.spark.csv').save('hdfs://path/df.csv')
  
  (All above didn't work, Failed to find data source)
  </code></pre>
  
  <p>or:</p>
  
  <pre><code>def toCSVLine(data):
      return ','.join(str(d) for d in data)
  
  lines = df.rdd.map(toCSVLine)
  lines.saveAsTextFile('hdfs://path/df.csv')  
  
  (Permission denied)
  </code></pre>
  
  <p>Q: </p>
  
  <p>1, How to solve "Failed to find data source"?</p>
  
  <p>2, I used sudo to make the dictionary "/path" on hdfs, if I turn the dataframe to rdd, how to write the rdd to csv on hdfs?</p>
  
  <p>Thanks a lot!</p>