title: Very slow writing of a dataframe to file on Spark cluster
body: <p>I have a test program that writes a dataframe to file. The dataframe is generated by adding sequential numbers for each row, like</p>
  
  <pre><code>1,2,3,4,5,6,7.....11
  2,3,4,5,6,7,8.....12
  ......
  </code></pre>
  
  <p>There is 100000 rows in the dataframe, but I don't think it is too big.
  When I submit the Spark task, it takes almost 20 minutes to write the dataframe to file on HDFS. I am wondering why it is so slow, and how to improve the performance.</p>
  
  <pre><code>val sc = new SparkContext(conf)
  val sqlContext = new org.apache.spark.sql.SQLContext(sc)
  val numCol = 11
  
  val arraydataInt = 1 to 100000 toArray
  val arraydata = arraydataInt.map(x =&gt; x.toDouble)
  val slideddata = arraydata.sliding(numCol).toSeq
  val rows = arraydata.sliding(numCol).map { x =&gt; Row(x: _*) }
  val datasetsize = arraydataInt.size
  
  val myrdd = sc.makeRDD(rows.toSeq, arraydata.size - numCol).persist()
  
  val schemaString = "value1 value2 value3 value4 value5 " +
                     "value6 value7 value8 value9 value10 label"
  
  val schema =
  StructType(schemaString.split(" ").map(fieldName =&gt; StructField(fieldName, DoubleType, true)))
  
  val df = sqlContext.createDataFrame(myrdd, schema).cache()
  
      val splitsH = df.randomSplit(Array(0.8, 0.1))
  val trainsetH = splitsH(0).cache()
  val testsetH = splitsH(1).cache()
  
  println("now saving training and test samples into files")
  
  trainsetH.write.save("TrainingSample.parquet")
  testsetH.write.save("TestSample.parquet")
  </code></pre>