title: Pyspark- Subquery in a case statement
body: <p>I am trying to run a subquery inside a case statement in Pyspark and it is throwing an exception. I am trying to create a new flag if id in one table is present in a different table.</p>
  
  <p>Can anyone please let me know if this is even possible in pyspark?</p>
  
  <pre><code>temp_df=spark.sql("select *, case when key in (select distinct key from Ids) then 1 else 0 end as flag from main_table")
  </code></pre>
  
  <p>Here is the error:</p>
  
  <pre><code>AnalysisException: 'Predicate sub-queries can only be used in a Filter
  </code></pre>