title: How to find median and quantiles using Spark
body: <p>How can I find median of an <code>RDD</code> of integers using a distributed method, IPython, and Spark? The <code>RDD</code> is approximately 700,000 elements and therefore too large to collect and find the median.</p>
  
  <p>This question is similar to this question. However, the answer to the question is using Scala, which I do not know.</p>
  
  <p><a href="https://stackoverflow.com/questions/28158729/how-can-i-calculate-exact-median-with-apache-spark">How can I calculate exact median with Apache Spark?</a></p>
  
  <p>Using the thinking for the Scala answer, I am trying to write a similar answer in Python.</p>
  
  <p>I know I first want to sort the <code>RDD</code>. I do not know how. I see the <code>sortBy</code> (Sorts this RDD by the given <code>keyfunc</code>) and <code>sortByKey</code> (Sorts this <code>RDD</code>, which is assumed to consist of (key, value) pairs.) methods. I think both use key value and my <code>RDD</code> only has integer elements.</p>
  
  <ol>
  <li>First, I was thinking of doing <code>myrdd.sortBy(lambda x: x)</code>? </li>
  <li>Next I will find the length of the rdd (<code>rdd.count()</code>).</li>
  <li>Finally, I want to find the element or 2 elements at the center of the rdd. I need help with this method too.</li>
  </ol>
  
  <p>EDIT:</p>
  
  <p>I had an idea. Maybe I can index my <code>RDD</code> and then key = index and value = element. And then I can try to sort by value? I don't know if this is possible because there is only a <code>sortByKey</code> method.</p>