Pyspark VS native python speed
Question: I am running Spark on a Ubuntu VM (4GB, 2 cores). I am doing this simple test
of a word count. I am comparing it to a simple Python `dict()` counter. I find
that Pyspark is 5x slower (takes more time).

Is this because of the initialisation or do I need to tune a parameter?

    
    
    import sys, os
    
    sys.path.append('/home/dirk/spark-1.4.1-bin-hadoop2.6/python')
    os.environ['SPARK_HOME']='/home/dirk/spark-1.4.1-bin-hadoop2.6/'
    os.environ['SPARK_WORKER_CORES']='2'
    os.environ['SPARK_WORKER_MEMORY']='2g'
    
    import time
    import py4j
    from pyspark import SparkContext, SparkConf
    from operator import add
    
    conf=(SparkConf().setMaster('local').setAppName('app'))
    sc=SparkContext(conf=conf)
    
    f='big3.txt'
    
    s=time.time()
    lines=sc.textFile(f)
    counts=lines.flatMap(lambda x:x.split(' ')).map(lambda x: (x,1)).reduceByKey(add)
    output=counts.collect()
    print len(output)
    for (word, count) in output:
        if count>100: print word, count
    sc.stop()
    print 'elapsed',time.time()-s
    
    s=time.time()
    f1=open(f,'r')
    freq={}
    for line in f1:
        words=line.split(' ')
        for w in words:
            if w not in freq:
                freq[w]=0
            freq[w]+=1
    f1.close()
    print len(freq)
    for (w,c) in freq.iteritems():
        if c>100: print w,c
    print 'elapsed',time.time()-s
    


Answer: As mentioned in comments, Spark isn't ideal for such short-lived jobs. You'd
better run it on hundreds of files, each being at least a few megabytes in
size. Even then it might saturate the HDD reading speed before the CPU gets
100% utilized, ideally you'd have at least a few computers with SSDs to get
real advantage from Spark :)

