title: ConcurrentModificationException in Spark
body: <p>Below is my code:</p>
  
  <pre><code>scala&gt; spark.sql(s" select * from $da_table_nm a where metric_date=date_add(date_sub('$order_dt','$start'),3)").repartition(num_partitions).createOrReplaceTempView("temp")
  
  scala&gt; spark.catalog.cacheTable("temp")
  
  scala&gt; spark.sql(s"insert into table $cortex_table_nm partition(year,month,day) select a.*, year(metric_date)as year, month(metric_date)as month,day(metric_date)as day from temp a")
  </code></pre>
  
  <p>I am getting this error in Spark 2.0:</p>
  
  <pre><code>ERROR LiveListenerBus: Listener EventLoggingListener threw an exception
  java.util.ConcurrentModificationException
  </code></pre>
  
  <p>I am calling these three lines in a loop to process data from different dates (metric_date) in the above code.  The job progresses in spite of this error but after processing a few days, it hangs and fails.  Any idea on how I can avoid this?</p>