title: Conditional explode in pyspark
body: <p>I have data as below</p>
  <pre><code>+----------+-----------------------------------+---------------------------------------------------------------------+
  |athl_id   |Interest                           |branch                                                               |
  +----------+-----------------------------------+---------------------------------------------------------------------+
  |123       |Running                            |Running,Outdoor                                                      |
  |856       |Running                            |Running                                                              |
  |902       |Training,Fitness                   |Fitness,Training                                                     |
  |9567      |Swimming,Training,Fitness          |Swimming,Training,Fitness                                            |
  |477       |All                                |Running,All,Training,Soccer,Swimming,Fitness,Outdoor,Indoor          |
  |490       |Running,Indoor                     |Running,Indoor                                                       |
  +----------+-----------------------------------+---------------------------------------------------------------------+
  </code></pre>
  <p>Now I want to explode two fields Interest and branch with below conditions.</p>
  <ol>
  <li>For every athl_id, explode Interest field completely</li>
  <li>If any of the comma separated values of branch equals to any of the comma separated values of Interest then ignore that value alone completely from branch and explode rest.</li>
  <li>If no comma separated values of branch equals to any of the comma separated values of Interest then explode the field branch.</li>
  </ol>
  <p>Ex - In the above table, athl_id - 902 has Interest in Training,Fitness  and since branch values are also same (Fitness,Training) they expected result has nulls for branch and Interest exploded to two rows.
  Similarly, athl_id - 477 has Interest in All and branch has value  &quot;Running,All,Training,Soccer,Swimming,Fitness,Outdoor,Indoor&quot; but, since &quot;All&quot; is part of Interest, in result the filed branch which is exploded does not contain &quot;All&quot; but the rest &quot;Running,Training,Soccer,Swimming,Fitness,Outdoor,Indoor&quot;</p>
  <p><strong>Expected result:</strong></p>
  <pre><code>
  +----------+-----------------------------------+---------------------------------------------------------------------+
  |athl_id   |Interest                           |branch                                                               |
  +----------+-----------------------------------+---------------------------------------------------------------------+
  |123       |Running                            |Outdoor                                                              |
  |856       |Running                            |                                                                     |
  |902       |Training                           |                                                                     |
  |902       |Fitness                            |                                                                     |
  |9567      |Swimming                           |                                                                     |
  |9567      |Training                           |                                                                     |
  |9567      |Fitness                            |                                                                     |
  |477       |All                                |Running                                                              |
  |477       |All                                |Training                                                             |
  |477       |All                                |Soccer                                                               |
  |477       |All                                |Swimming                                                             |
  |477       |All                                |Fitness                                                              |
  |477       |All                                |Outdoor                                                              |
  |477       |All                                |Indoor                                                               |
  |490       |Running                            |                                                                     |
  |490       |Indoor                             |                                                                     |
  +----------+-----------------------------------+---------------------------------------------------------------------+
  </code></pre>
  <p>Now, I have tried something as below but, struck with an error. And also, I think &quot;array_contains&quot; isn't matching the exact values..</p>
  <pre><code>spark.sql(&quot;&quot;&quot;  
  select athl_id, Interest,
  case when array_contains(split(branch,','),Interest) then null
  else explode(split(branch,',')) end as branch
  from (
  select athl_id, explode(split(Interest,',')) as   Interest ,branch from athl_details)a
  &quot;&quot;&quot;).show(100,False )
  
  
  Traceback (most recent call last):
    File &quot;&lt;stdin&gt;&quot;, line 7, in &lt;module&gt;
    File &quot;/usr/lib/spark/python/pyspark/sql/session.py&quot;, line 767, in sql
      return DataFrame(self._jsparkSession.sql(sqlQuery), self._wrapped)
    File &quot;/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py&quot;, line 1257, in __call__
    File &quot;/usr/lib/spark/python/pyspark/sql/utils.py&quot;, line 69, in deco
      raise AnalysisException(s.split(': ', 1)[1], stackTrace)
  pyspark.sql.utils.AnalysisException: u&quot;Generators are not supported when it's nested in expressions
  </code></pre>
  <p>Can someone suggest me the right approach.</p>
  <p>Thanks!! </p>