title: Spark - how to get top N of rdd as a new rdd (without collecting at the driver)
body: <p>I am wondering how to filter an RDD that has one of the top N values.  Usually I would sort the RDD and take the <code>top</code> N items as an array in the driver to find the Nth value that can be broadcasted to filter the rdd like so: </p>
  
  <pre><code>val topNvalues = sc.broadcast(rdd.map(_.fieldToThreshold).distict.sorted.take(N))
  val threshold = topNvalues.last
  val rddWithTopNValues = rdd.filter(_.fieldToThreshold &gt;= threshold)
  </code></pre>
  
  <p>but in this case my N is too large, so how can I do this purely with RDDs like so?:</p>
  
  <pre><code>def getExpensiveItems(itemPrices: RDD[(Int, Float)], count: Int): RDD[(Int, Float)] = {
       val sortedPrices = itemPrices.sortBy(-_._2).map(_._1).distinct
  
       // How to do this without collecting results to driver??
       val highPrices = itemPrices.getTopNValuesWithoutCollect(count)
  
       itemPrices.join(highPrices.keyBy(x =&gt; x)).map(_._2._1)
  }
  </code></pre>