title: Improve speed of spark app
body: <p>This is part of my python-spark code which parts of it run too slow for my needs.
  Especially this part of the code, which I would really like to improve it's speed but don't know how to. It currently takes around 1 minute for 60 Million data rows and I would like to improve it to under 10 seconds.</p>
  
  <pre><code>sqlContext.read.format("org.apache.spark.sql.cassandra").options(table="axes", keyspace=source).load() 
  </code></pre>
  
  <p>More context of my spark app:</p>
  
  <pre><code>article_ids = sqlContext.read.format("org.apache.spark.sql.cassandra").options(table="article_by_created_at", keyspace=source).load().where(range_expr).select('article','created_at').repartition(64*2)
  
  axes = sqlContext.read.format("org.apache.spark.sql.cassandra").options(table="axes", keyspace=source).load()
  speed_df = article_ids.join(axes,article_ids.article==axes.article).select(axes.article,axes.at,axes.comments,axes.likes,axes.reads,axes.shares) \
       .map(lambda x:(x.article,[x])).reduceByKey(lambda x,y:x+y) \
       .map(lambda x:(x[0],sorted(x[1],key=lambda y:y.at,reverse = False))) \
       .filter(lambda x:len(x[1])&gt;=2) \
       .map(lambda x:x[1][-1]) \
       .map(lambda x:(x.article,(x,(x.comments if x.comments else 0)+(x.likes if x.likes else 0)+(x.reads if x.reads else 0)+(x.shares if x.shares else 0))))    
  </code></pre>
  
  <p>Thanks a lot for your suggestions.</p>
  
  <p>EDIT:</p>
  
  <p>Count takes up most of the time (50s) not join</p>
  
  <p>I also tried increasing parallelism with but it didn't have any obvious effect:</p>
  
  <pre><code>sqlContext.read.format("org.apache.spark.sql.cassandra").options(table="axes", keyspace=source).load().repartition(number) 
  </code></pre>
  
  <p>and</p>
  
  <pre><code>sqlContext.read.format("org.apache.spark.sql.cassandra").options(table="axes", keyspace=source,numPartitions=number).load()
  </code></pre>
  
  <p><a href="https://i.stack.imgur.com/zTXhg.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/zTXhg.png" alt="This is the picture from spark showing how long each operation takes"></a></p>