title: Actual run time of Spark job from context
body: <p>I'm trying to get the time an application spends in the state <code>RUNNING</code>. This can be easily done if a <a href="https://spark.apache.org/docs/latest/api/java/org/apache/spark/launcher/SparkLauncher.html" rel="nofollow noreferrer">SparkLauncher</a> is used by adding <a href="https://spark.apache.org/docs/latest/api/java/org/apache/spark/launcher/SparkAppHandle.Listener.html" rel="nofollow noreferrer">SparkAppHandle.Listener</a> callbacks and listen to the <code>stateChanged</code> event.</p>
  <p>Long story short, I don't have the option to use <a href="https://spark.apache.org/docs/latest/api/java/org/apache/spark/launcher/SparkLauncher.html" rel="nofollow noreferrer">SparkLauncher</a> but I need to know the actual runtime of an application (without the time it spends in the <code>WAITING</code> state).</p>
  <p>Can this information be gleaned from the <code>SparkContext</code>? <a href="https://spark.apache.org/docs/latest/api/java/org/apache/spark/scheduler/SparkListener.html" rel="nofollow noreferrer">SparkListener</a> doesn't seem to provide a callback for application state change. The actual duration cannot be worked out from <code>onApplicationStart</code> and <code>onApplicationEnd</code> because <code>onApplicationStart</code> fires when the application starts waiting.</p>
  <p>I'm also aware of <a href="https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/util/QueryExecutionListener.html" rel="nofollow noreferrer">QueryExecutionListener</a> but the individual jobs also include the <code>WAITING</code> period (<code>onSuccess</code> callback duration param).</p>
  <p>Edit: Upon further inspection, <code>SparkAppHandle.Listener</code> doesn't seem to be able to do this either.</p>