title: How does foldLeft in Scala work on DataFrame?
body: <p>I had a requirement to ingest an RDBMS table into Hive and I had to clean the data in its String columns, before inserting it into a Hive table using a regex_replace pattern. After failing to understand how to apply it on my dataFrame, I finally I came across a method in Scala which is <code>foldLeft</code> which helped in fulfilling the requirement.</p>
  
  <p>I understand how foldLeft works on a collection, for example:</p>
  
  <pre><code>List(1,3,9).foldLeft(100)((x,y) =&gt; x+y)
  </code></pre>
  
  <p>foldLeft takes arguments: initialValue and a function. It adds the result of the function to the accumulator. In the above case, the result is: 113.</p>
  
  <p>But when it comes to dataframe, I am unable to understand how it works.</p>
  
  <pre><code>val stringColumns = yearDF.schema.fields.filter(_.dataType == StringType).map(_.name)
  val finalDF = stringColumns.foldLeft(yearDF){ (tempdf, colName) =&gt; tempdf.withColumn(colName, regexp_replace(col(colName), "\n", "")) }
  </code></pre>
  
  <p>In the above code, I got the String columns from the dataFrame: <code>yearDF</code> which is kept in the accumulator of the <code>foldLeft</code>. I have the following doubts regarding the function used in <code>foldLeft</code>:</p>
  
  <ol>
  <li>What value does tempDF hold ? If it is the same as yearDF, how is it mapped to yearDF ?</li>
  <li>If <code>withColumns</code> is used in the function and the result is added to
  yearDF, how come it is not creating duplicating columns when</li>
  </ol>
  
  <p>Could anyone explain it, so that I can have a better understanding about foldLeft.</p>