title: Spark - How to calculate percentiles in Spark?
body: <p>I was trying to to get the 0.8 percentile of a single column dataframe. I tried in this way:</p>
  
  <pre><code>val limit80 = 0.8
  val dfSize = df.count()
  val perfentileIndex = dfSize*limit80 
  
  dfSorted = df.sort()
  val percentile80 = dfSorted .take(perfentileIndex).last()
  </code></pre>
  
  <p>But I think this will fail for big dataframes, since they may be distributed across different nodes.</p>
  
  <p>Is there any better way to calculate the percentile? or how could I have all the rows of the dataframe in the same machine (even if that is very anti-pattern) so the <code>df.take(index)</code> will really take into account the whole dataset and not just a partition in a node.</p>