title: Apache Spark slow on reduceByKey step
body: <p>I have a 2MB plain text file in /usr/local/share/data/.
  And then I run against the following code in Apache Spark.</p>
  
  <pre><code>conf = SparkConf().setMaster("local[*]").setAppName("test").set("spark.executor.memory", "2g") 
  sc = SparkContext(conf=conf)
  doc_rdd = sc.textFile("/usr/local/share/data/")
  unigrams = doc_rdd.flatMap(word_tokenize)
  step1 = unigrams.flatMap(word_pos_tagging)
  step2 = step1.filter(lambda x: filter_punctuation(x[0]))
  step3 = step2.map(lambda x: (x, 1))
  freq_unigrams = step3.reduceByKey(lambda x, y: x + y)
  </code></pre>
  
  <p><strong>Expected result</strong></p>
  
  <pre><code>[((u'showing', 'VBG'), 24), ((u'Ave', 'NNP'), 1), ((u'Scrilla364', 'NNP'), 1), ((u'internally', 'RB'), 4), ...]
  </code></pre>
  
  <p>But it takes very very long time (6 minutes) to return the expected word count.
  It stucks at reduceByKey steps.
  <em>How to resolve this performance issue?</em></p>
  
  <p>-- Reference --</p>
  
  <p><strong>Hardware Specification</strong></p>
  
  <p>Model Name: MacBook Air Model
  Identifier: MacBookAir4,2
  Processor Name: Intel Core i7 Processor
  Speed: 1.8 GHz
  Number of Processors: 1
  Total Number of Cores: 2
  L2 Cache (per Core): 256 KB
  L3 Cache: 4 MB
  Memory: 4 GB </p>
  
  <p><strong>Log</strong></p>
  
  <pre><code>15/10/02 16:05:12 INFO HadoopRDD: Input split: file:/usr/local/share/data/enronsent01:0+873602
  15/10/02 16:05:12 INFO HadoopRDD: Input split: file:/usr/local/share/data/enronsent01:873602+873602
  15/10/02 16:09:11 INFO BlockManagerInfo: Removed broadcast_2_piece0 on localhost:53478 in memory (size: 4.1 KB, free: 530.0 MB)
  15/10/02 16:09:11 INFO BlockManagerInfo: Removed broadcast_3_piece0 on localhost:53478 in memory (size: 4.6 KB, free: 530.0 MB)
  15/10/02 16:09:11 INFO ContextCleaner: Cleaned accumulator 4
  15/10/02 16:09:11 INFO ContextCleaner: Cleaned accumulator 3
  15/10/02 16:09:11 INFO BlockManagerInfo: Removed broadcast_1_piece0 on localhost:53478 in memory (size: 3.9 KB, free: 530.0 MB)
  15/10/02 16:09:11 INFO ContextCleaner: Cleaned accumulator 2
  15/10/02 16:10:05 INFO PythonRDD: Times: total = 292892, boot = 8, init = 275, finish = 292609
  15/10/02 16:10:05 INFO Executor: Finished task 1.0 in stage 3.0 (TID 4). 2373 bytes result sent to driver
  15/10/02 16:10:05 INFO TaskSetManager: Finished task 1.0 in stage 3.0 (TID 4) in 292956 ms on localhost (1/2)
  15/10/02 16:10:35 INFO PythonRDD: Times: total = 322562, boot = 5, init = 276, finish = 322281
  15/10/02 16:10:35 INFO Executor: Finished task 0.0 in stage 3.0 (TID 3). 2373 bytes result sent to driver
  15/10/02 16:10:35 INFO TaskSetManager: Finished task 0.0 in stage 3.0 (TID 3) in 322591 ms on localhost (2/2)
  </code></pre>