title: Replace missing values with mean - Spark Dataframe
body: <p>I have a Spark Dataframe with some missing values. I would like to perform a simple imputation by replacing the missing values with the mean for that column. I am very new to Spark, so I have been struggling to implement this logic. This is what I have managed to do so far:</p>
  
  <p>a) To do this for a single column (let's say Col A), this line of code seems to work:</p>
  
  <pre><code>df.withColumn("new_Col", when($"ColA".isNull, df.select(mean("ColA"))
    .first()(0).asInstanceOf[Double])
    .otherwise($"ColA"))
  </code></pre>
  
  <p>b) However, I have not been able to figure out, how to do this for all the columns in my dataframe. I was trying out the Map function, but I believe it loops through each row of a dataframe </p>
  
  <p>c) There is a similar question on SO - <a href="https://stackoverflow.com/questions/38356476/spark-replace-null-values-in-dataframe-with-mean-of-column">here</a>. And while I liked the solution (using Aggregated tables and coalesce), I was very keen to know if there is a way to do this by looping through each column (I come from R, so looping through each column using a higher order functional like lapply seems more natural to me).</p>
  
  <p>Thanks!</p>