title: generating DataFrames in for loop in Scala Spark cause out of memory
body: <p>I'm generating small dataFrames in for loop. At each round of for loop, I pass the generated dataFrame to a function which returns double. This simple process (which I thought could be easily taken care of by garbage collector) blow up my memory. When I look at Spark UI at each round of for loop it adds a new  "SQL{1-500}" (my loop runs 500 times). My question is how to drop this sql object before generating a new one?</p>
  
  <p>my code is something like this:</p>
  
  <pre><code>Seq.fill(500){
    val data = (1 to 1000).map(_=&gt;Random.nextInt(1000))
    val dataframe = createDataFrame(data)
    myFunction(dataframe)
    dataframe.unpersist()
  }
  
  def myFunction(df: DataFrame)={
    df.count()
  }
  </code></pre>
  
  <p>I tried to solve this problem by dataframe.unpersist() and sqlContext.clearCache() but neither of them worked.</p>