title: How to filter after group by and aggregate in Spark dataframe?
body: <p>I have a spark dataframe <i>df</i> with schema as such: </p>
  
  <pre><code>[id:string, label:string, tags:string]
  
  id | label | tag
  ---|-------|-----
   1 | h     | null
   1 | w     | x
   1 | v     | null
   1 | v     | x
   2 | h     | x
   3 | h     | x
   3 | w     | x
   3 | v     | null
   3 | v     | null
   4 | h     | null
   4 | w     | x
   5 | w     | x
  </code></pre>
  
  <p>(h,w,v are labels. x can be any non-empty values)</p>
  
  <p>For each id, there is at most one label "h" or "w", but there might be multiple "v". I would like to select all the ids that satisfies following conditions:</p>
  
  <p>Each id has:
   1. one label "h" and its tag = null,
   2. one label "w" and its tag != null,
   3. at least one label "v" for each id.</p>
  
  <p>I am thinking that I need to create three columns checking each above conditions. And then I need to do a group by "id".</p>
  
  <pre><code>val hCheck = (label: String, tag: String) =&gt; {if (label=="h" &amp;&amp; tag==null) 1 else 0}
  val udfHCheck = udf(hCheck)
  val wCheck = (label: String, tag: String) =&gt; {if (label=="w" &amp;&amp; tag!=null) 1 else 0}
  val udfWCheck = udf(wCheck)
  val vCheck = (label: String) =&gt; {if (label==null) 1 else 0}
  val udfVCheck = udf(vCheck)
  
  dfx = df.withColumn("hCheck", udfHCheck(col("label"), col("tag")))
          .withColumn("wCheck", udfWCheck(col("label"), col("tag")))
          .withColumn("vCheck", udfVCheck(col("label")))
          .select("id","hCheck","wCheck","vCheck")
          .groupBy("id")
  </code></pre>
  
  <p>Somehow I need to group three columns {"hCheck","wCheck","vCheck"} into vector of list [x,0,0],[0,x,0],[0,0,x]. And check if these vector contain all three {[1,0,0],[0,1,0],[0,0,1]}</p>
  
  <p>I have not been able to solve this problem yet. And there might be a better approach than this one. Hope someone can give me suggestions. Thanks</p>