title: Calculate percentile with groupBy on PySpark dataframe
body: <p>I am trying to <code>groupBy</code> and then calculate percentile on PySpark dataframe. I've tested the following piece of code according to <a href="https://stackoverflow.com/questions/40006395/applying-udfs-on-groupeddata-in-pyspark-with-functioning-python-example">this Stack Overflow post</a>:</p>
  <pre><code>from pyspark.sql.types import FloatType
  import pyspark.sql.functions as func
  import numpy as np
  
  qt_udf = func.udf(lambda x,qt: float(np.percentile(x,qt)), FloatType())
  df_out = df_in.groupBy('Id').agg(func.collect_list('value').alias('data'))\
  .withColumn('median', qt_udf(func.col('data'),func.lit(0.5)).cast(&quot;string&quot;))  
  
  df_out.show()
  </code></pre>
  <p>But get the following error:</p>
  <blockquote>
  <p>Traceback (most recent call last):  &gt; df_out.show() ....&gt; return lambda *a: f(*a)  AttributeError: 'module' object has no attribute 'percentile'</p>
  </blockquote>
  <p>This is because of numpy version (1.4.1), the percentile function was added from version 1.5. It is not possible to update numpy version in the short term.</p>