title: How do I speed up sortBy or sortBykey method in spark?
body: <p>I've been trying to speed up <code>sortBy()</code> or <code>sortByKey()</code> method in spark.</p>
  
  <p>When I run spark app with these methods, It consumes the most time when sorting in total program time.</p>
  
  <p>Here is example code. Both <code>sortBy()</code> and <code>sortByKey()</code> are slow. </p>
  
  <pre><code>val textFile = sc.textFile("hdfs://...")
  val counts = textFile.flatMap(line =&gt; line.split(" "))
                   .map(word =&gt; (word, 1))
                   .reduceByKey(_ + _).sortByKey(true, 1)
  </code></pre>
  
  <p>If I have a big data file, Is there an effective way to sort for the big data file?</p>