title: Token frequency of tokens in a list inside DataFrame
body: <p>I wasn't quite sure how to phrase the titel.
  I have a Dataframe with one column where each row consist of a list of tokens. I need to get the frequency of the words and then sort them in order to get the most frequent words. Here is a image of the DataFrame schema: <a href="https://i.stack.imgur.com/elkZz.png" rel="nofollow noreferrer">https://i.stack.imgur.com/elkZz.png</a></p>
  
  <p>A tokenizer was used to get the array of the tokens .</p>
  
  <p><a href="https://i.stack.imgur.com/OfMoX.png" rel="nofollow noreferrer">What the array inside the Dataframe row looks like</a></p>
  
  <p>Now I've done this before when the "row" consisted of just one string with plenty of words instead of an list with words at each index. </p>
  
  <p>With :</p>
  
  <p><code>frequency = dataframe.flatMap(lambda line: line.split(" ")).map(lambda word: (word, 1)).reduceByKey(lambda a, b: a+b)</code></p>
  
  <p>However I can't for the life of me manage to figure out how to access the elements. The lambda expression above doesn't work since it tries to do it on the list. 'unhashable type: 'list''</p>
  
  <p>So the question in tldr: How do I properly access the elements within the array to count the frequency of the words within all rows of the Dataframe? Furthermore how would I go about to get the results into an array?</p>