title: spark parquet write gets slow as partitions grow
body: <p>I have a spark streaming application that writes parquet data from stream.</p>
  
  <pre><code>sqlContext.sql(
        """
          |select
          |to_date(from_utc_timestamp(from_unixtime(at), 'US/Pacific')) as event_date,
          |hour(from_utc_timestamp(from_unixtime(at), 'US/Pacific')) as event_hour,
          |*
          |from events
          | where at &gt;= 1473667200
        """.stripMargin).coalesce(1).write.mode(SaveMode.Append).partitionBy("event_date", "event_hour","verb").parquet(Config.eventsS3Path)
  </code></pre>
  
  <p>this piece of code runs every hour but over time the writing to parquet has slowed down. When we started it took 15 mins to write data, now it takes 40 mins. It is taking time propotional to data existing in that path. I tried running the same application to a new location and that runs fast.</p>
  
  <p>I have disabled schemaMerge and summary metadata:</p>
  
  <pre><code>sparkConf.set("spark.sql.hive.convertMetastoreParquet.mergeSchema","false")
  sparkConf.set("parquet.enable.summary-metadata","false")
  </code></pre>
  
  <p>using spark 2.0</p>
  
  <p>batch execution:
  empty directory
  <a href="https://i.stack.imgur.com/EJ1pl.png" rel="noreferrer"><img src="https://i.stack.imgur.com/EJ1pl.png" alt="enter image description here"></a>
  <a href="https://i.stack.imgur.com/gdhIS.png" rel="noreferrer"><img src="https://i.stack.imgur.com/gdhIS.png" alt="enter image description here"></a>
  <a href="https://i.stack.imgur.com/jpViT.png" rel="noreferrer"><img src="https://i.stack.imgur.com/jpViT.png" alt="enter image description here"></a>
  directory with 350 folders
  <a href="https://i.stack.imgur.com/YijeN.png" rel="noreferrer"><img src="https://i.stack.imgur.com/YijeN.png" alt="enter image description here"></a>
  <a href="https://i.stack.imgur.com/GDsLV.png" rel="noreferrer"><img src="https://i.stack.imgur.com/GDsLV.png" alt="enter image description here"></a>
  <a href="https://i.stack.imgur.com/pd6Oq.png" rel="noreferrer"><img src="https://i.stack.imgur.com/pd6Oq.png" alt="enter image description here"></a></p>