title: PySpark - compare single list of integers to column of lists
body: <p>I'm trying to check which entries in a spark dataframe (column with lists) contain the largest quantity of values from a given list.</p>
  
  <p>The best approach I've came up with is iterating over a dataframe with <code>rdd.foreach()</code> and comparing a given list to every entry using python's <code>set1.intersection(set2)</code>.</p>
  
  <p>My question is does spark have any built-in functionality for this so iterating with <code>.foreach</code> could be avoided?</p>
  
  <p>Thanks for any help!</p>
  
  <p>P.S. my dataframe looks like this:</p>
  
  <pre><code>+-------------+---------------------+                                           
  |   cardnumber|collect_list(article)|
  +-------------+---------------------+
  |2310000000855| [12480, 49627, 80...|
  |2310000008455| [35531, 22564, 15...|
  |2310000011462| [117112, 156087, ...|
  +-------------+---------------------+
  </code></pre>
  
  <p>And I'm trying to find entries with the most intersections in the second column with a given list of articles, e.g <code>[151574, 87239, 117908, 162475, 48599]</code></p>