title: improvement of a snippet of scala/spark code to improve performance
body: <p>I am trying to calculate the mean of a certain column and save it as a new column, following is my code snippet of achieving this:</p>
  <pre><code>df = df.withColumn(&quot;avg_colname&quot;, lit(df.select(avg(&quot;colname&quot;).as(&quot;temp&quot;)).first().getAs(&quot;temp&quot;)))
  
  </code></pre>
  <p>In total, there are 8 columns to be calculated. On a small 3-node cluster using the &quot;spark-submit&quot; command, the code execution takes much more time than on a single machine using the &quot;spark-shell&quot; command(several minutes vs. a few seconds).</p>
  <p>Why does the code execute on a cluster slower than on a single machine, and how can the code snippet above be improved?</p>