title: How to avoid using withColumn iteratively in Spark Scala?
body: <p>Currently, we have a code that has iterative usage of withColumn. When-Otherwise Conditional check &amp; we do arithmetic computations on top of that.</p>
  
  <p>Sample Code:</p>
  
  <pre><code>df.withColumn("col4", when(col("col1")&gt;10, col("col2").+col("col3")).otherwise(col("col2")))
  </code></pre>
  
  <p>Other arithmetic computations happens iteratively for another 40 Columns.
  Total Record Count - 2M.</p>
  
  <p><strong>Code that is reiterated:</strong></p>
  
  <pre><code>  df.withColumn(colName1, when((col(amountToSubtract).&lt;("0")) &amp;&amp; (col(colName1).===("0")) &amp;&amp; (col(amountToSubtract).&lt;=(col(colName2))) &amp;&amp; (col(colName2).!==("0")), col(colName2)).otherwise(col(colName1))).
        withColumn(amountToSubtract, when(col(amountToSubtract).!==("0"), col(amountToSubtract).-(col(colName2))).otherwise(col(amountToSubtract))).
        withColumn(colName1, when((col(amountToSubtract).&gt;("0")) &amp;&amp; (col(colName1).===("0")), col(colName2).+(col(amountToSubtract))).otherwise(col(colName1))).
        withColumn(amountToSubtract, when((col(amountToSubtract).&gt;("0")) &amp;&amp; (col(colName1).===("0")), "0").otherwise(col(amountToSubtract)))
  </code></pre>
  
  <p>This is followed for 7 or 8 other sets of computation.</p>
  
  <p>And the job hangs at this point for a longer time. Sometimes it throws GC overhead error.
  Being aware that iterative usage of .withColumn does not benefit performance, I could not find an alternative to implement these above mentioned conditional checks. Kindly assist.</p>