title: Spark/Python, reduceByKey() then find top 10 most frequent words and frequencies
body: <p>I have a VirtualMachine setup with Hadoop + Spark and I'm reading a text file &quot;words.txt&quot; from my HDFS and then calling map(), flatmap(), then reduceByKey() and <strong>attempting to get the Top 10 most frequent words and their occurrences</strong>. I already have the majority of the code complete, and then the tuple list are being aggregated but <strong>I just need a way to find the top 10</strong>. I know I need to simply iterate through the value in the tuple (key is the actual str word, but the value is the integer of how many times the word appeared in the words.txt file) and just have a counter that counts the top 10. The (K,V) value pair is Key = word from words.txt, and Value = integer aggregated value for how many times it appeared in the file. This screenshot below is after reduceByKey() had already been called, you can see 'the' appears 40 times (and the end of the screen shot to the right)</p>
  <p>Here's the output:
  <a href="https://i.stack.imgur.com/hHfwQ.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/hHfwQ.png" alt="enter image description here" /></a></p>
  <p>Here's my code so far:</p>
  <pre><code>from pyspark import SparkcConf, SparkContext
  
  # Spark set-up
  conf = SparkConf()
  conf.setAppName(&quot;Word count App&quot;)
  sc = SparkContext(conf=conf)
  
  # read from text file words.txt on HDFS
  rdd = sc.textFile(&quot;/user/spark/words.txt&quot;)
  
  # flatMap() to output multiple elements for each input value, split on space and make each word lowercase
  rdd = rdd.flatMap(lamda x: x.lower().split(' '))
  
  # Map a tuple and append int 1 for each word in words.txt
  rdd = rdd.map(lamda x: (x,1))
  
  # Perform aggregation (sum) all the int values for each unique key)
  rdd = rdd.reduceByKey(lamda x, y: x+y)
  
  # This is where I need a function or lambda to sort by descending order so I can grab the top 10 outputs, then print them out below with for loop
  
  # for item in out:
  print(item[0], '\t:\t', str(item[1]))
  </code></pre>
  <p>I know I would normally just make a variable called &quot;max&quot; and only update it if the max had be found in the list or tuple, but whats confusing me is I 'm dealing with Spark and RDD so I have been getting errors because I was a bit confused what the RDDs are returning when they do map, flatmap, reduceByKey etc...</p>
  <p>Any help much appreciated</p>