title: Remove columns with only NA values with SparkR
body: <p>I am using SparkR in databricks to manipulate large dataframes. I have a spark DataFrame <code>df</code> from which I want to remove all the columns containing only <code>NA</code> values. In typical R I would use either the tidyverse approach</p>
  
  <pre><code>df %&gt;% select_if(~any(!is.na(.)))
  </code></pre>
  
  <p>or in base R</p>
  
  <pre><code>df[!sapply(df, function(x) all(is.na(x)))]
  </code></pre>
  
  <p>However, none of those two methods seem to work in SparkR. I also want to use as much as possible the capabilities of Spark to distribute the work. I have found the following solution</p>
  
  <pre><code>nacols&lt;- sapply(columns(df), function(c){
    nbna &lt;- SparkR::select(df, c) %&gt;% SparkR::na.omit() %&gt;% SparkR::nrow()
    return(nbna != 0)
  })
  
  df_nona&lt;- df %&gt;% SparkR::select(columns(df)[c(T, nacols)])
  </code></pre>
  
  <p>but I think it is inefficient considering SparkR does not distribute the work of <code>sapply</code>.</p>
  
  <p>Please let me know if you can think of a better way to do so. I am open to doing this part of my script in python as well using pyspark. However, as I am using a high concurrency cluster, I cannot use scala code.</p>
  
  <p>Thank you very much in advance.</p>