title: Filter columns having count equal to the input file rdd Spark
body: <p>I'm filtering Integer columns from the input parquet file with below logic and been trying to modify this logic to add additional validation to see if any one of the input columns have count equals to the input parquet file rdd count. I would want to filter out such column. </p>
  
  <p><strong>Update</strong></p>
  
  <p>The number of columns and names in the input file will not be static, it will change every time we get the file.
  The objective is to also filter out column for which the count is equal to the input file rdd count. Filtering integer columns is already achieved with below logic.</p>
  
  <pre><code>e.g input parquet file count = 100
      count of values in column A in the input file  = 100 
  </code></pre>
  
  <p>Filter out any such column.  </p>
  
  <p><strong>Current Logic</strong> </p>
  
  <pre><code> //Get array of structfields
  
  val columns = df.schema.fields.filter(x =&gt; 
                  x.dataType.typeName.contains("integer"))
  
    //Get the column names
    val z = df.select(columns.map(x =&gt; col(x.name)): _*)
  
    //Get array of string 
    val m = z.columns
  </code></pre>
  
  <p><strong>New Logic be like</strong></p>
  
  <pre><code>  val cnt = spark.read.parquet("inputfile").count()
  
    val d = z.column.where column count is not equals cnt 
  </code></pre>
  
  <p>I do not want to pass the column name explicitly to the new condition, since the column having count equal to input file will change ( val d = .. above) 
  How do we write logic for this ?</p>