title: Is there any efficient function for extracting the probability vector after running pyspark ML algorithm other than rdd.extract in Pyspark?
body: <p>I want to extract the probability from the vector and make it into rows. For that i have used <code>rdd.map(extract)</code></p>
  
  <pre><code>def extract(row):
      return(row.prediction,)+tuple(row.probability.toArray() .tolist())
  </code></pre>
  
  <p>I have 96 probabilities within that vector. After extracting them into rows i sorted and selected top 10 probabilities. This works good for small dataset like 1000 records (ie. 96*1000 =96000 rows). But for 100k records the function is taking more time. So is there any other way to extract those probabilities and make them as rows?</p>