title: Pyspark join and operation on values within a list in column
body: <p>I have two dataframes, namely</p>
  
  <pre><code>df1 = sc.parallelize([
    ['u1', 'type1', ['a', 'b']],
    ['u2', 'type1', ['a', 'c', 'd']],
    ['u1', 'type2', ['d']]
  ]).toDF(('person', 'type', 'keywords'))
  
  df2 = sc.parallelize([
    ['a', 2],
    ['b', 1],
    ['c', 0],
    ['d', 1],
    ['e', 3],
  ]).toDF(('keyword', 'score'))
  </code></pre>
  
  <p>I need to calculate, for each <code>person</code> and per <code>type</code>, the average <code>score</code> of its <code>keywords</code>. So, this average would be 1.5 for <code>person</code> 'u1' on <code>type</code> 'type1' as it has keywords 'a' and 'b' which contribute with 2+1/2=1.5</p>
  
  <p>I have tried an approach encompassing a join:</p>
  
  <pre><code>df = df1.join(df2) \
    .select('person', 'type', 'keywords', 'keyword', 'score') \
    .groupBy('person', 'type') \
    .agg(avg('score'))
  </code></pre>
  
  <p>but the problem is, it is computing the average on each possible keyword, not solely on those which said user and type have, so that I obtain 1.4 everywhere, which is the sum of all scores for all keywords divided by their number.
  I need to sum up only scores for those keywords in the list <code>keywords</code> per user and type.</p>