title: Baseline for measuring Apache Spark jobs execution times
body: <p>I am fairly new to Apache Spark. I have been using it for several months, but this is my first project that uses it.</p>
  
  <p>I use Spark to compute dynamic reports from data, stored in a NoSQL database (Cassandra). So far I have created several reports and they are computed correctly. Inside them I use <code>DataFrame</code> <code>.unionAll()</code>, <code>.join()</code>, <code>.count()</code>, <code>.map()</code>, etc.</p>
  
  <p>I am running a 1.4.1 Spark cluster on my local machine with the following setup:</p>
  
  <pre><code>export SPARK_WORKER_INSTANCES=6
  export SPARK_WORKER_CORES=8
  export SPARK_WORKER_MEMORY=1g
  </code></pre>
  
  <p>I have also populated the database with test data which is around 10-12k records per table.</p>
  
  <p>By using the driver's web UI (<a href="http://localhost:4040/" rel="nofollow">http://localhost:4040/</a>), I have noticed that the jobs are taking 40s-50s to execute, so lately I have been researching ways to tune Apache Spark and the jobs.</p>
  
  <p>I have configured Spark to use the <code>KryoSerializer</code>, I have set the <code>spark.io.compression.codec</code> to <code>lzf</code>, I have optimized the jobs as much as I can and as much as my knowledge allows me to.</p>
  
  <p>This led to the jobs taking 20s-30s to compute (which I think is a good improvement). The problem is that because this is my first Spark project, I have no baseline to compare the jobs times, so I have no idea if the execution is slow or fast and whether there is some problem in the code or with the Spark config.</p>
  
  <p>What is the best way to proceed? Is there a graph or benchmark that shows how much time an action with N data should take?</p>