title: Spark execution memory monitoring
body: <p>What I want is to be able to monitor Spark <strong>execution memory</strong> as opposed to <strong>storage memory</strong> available in SparkUI. I mean, <em>execution</em> memory <strong>NOT executor memory</strong>.</p>
  
  <p>By execution memory I mean:</p>
  
  <blockquote>
    <p>This region is used for buffering intermediate data when performing shuffles, joins, sorts and aggregations. The size of this region is configured through spark.shuffle.memoryFraction (default0.2).
    According to: <a href="https://issues.apache.org/jira/browse/SPARK-10000" rel="noreferrer">Unified Memory Management in Spark 1.6</a> </p>
  </blockquote>
  
  <p>After intense search for answers I found nothing but unanswered StackOverflow questions, answers that relate only to storage memory or ones with vague answers of the type <em>use Ganglia</em>, <em>use Cloudera console</em> etc... </p>
  
  <p>There seems to be a demand for this information on Stack Overflow, and yet not a single satisfactory answer is available. Here are some top posts of StackOverflow when searching <em>monitoring spark memory</em></p>
  
  <p><a href="https://stackoverflow.com/questions/48670247/monitor-spark-execution-and-storage-memory-utilization">Monitor Spark execution and storage memory utilisation
  </a></p>
  
  <p><a href="https://stackoverflow.com/questions/39615702/monitoring-the-memory-usage-of-spark-jobs">Monitoring the Memory Usage of Spark Jobs
  </a></p>
  
  <p><a href="https://stackoverflow.com/questions/40022599/spark-how-to-monitor-the-memory-consumption-on-spark-cluster">SPARK: How to monitor the memory consumption on Spark cluster?
  </a></p>
  
  <p><a href="https://stackoverflow.com/questions/45428903/spark-monitor-actual-used-executor-memory">Spark - monitor actual used executor memory
  </a></p>
  
  <p><a href="https://stackoverflow.com/questions/46884555/how-can-i-monitor-memory-and-cpu-usage-by-spark-application">How can I monitor memory and CPU usage by spark application?
  </a></p>
  
  <p><a href="https://stackoverflow.com/questions/48168127/how-to-get-memory-and-cpu-usage-by-a-spark-application">How to get memory and cpu usage by a Spark application?
  </a></p>
  
  <p><strong>Questions</strong></p>
  
  <p>Spark version > 2.0</p>
  
  <ol>
  <li><p>Is it possible to monitor <strong>Execution memory</strong> of Spark job? By monitoring I mean at minimum see used/available just like for storage memory per executor in Executor tab of SparkUI. Yes or No?</p></li>
  <li><p>Could I do it with SparkListeners (@JacekLaskowski ?) How about history-server? Or the only way is through the external tools? Graphana, Ganglia, others? If external tools, could you please point to a tutorial or provide some more detailed guidelines?</p></li>
  <li><p>I saw this <a href="https://issues.apache.org/jira/browse/SPARK-9103?attachmentOrder=desc" rel="noreferrer">SPARK-9103 Tracking spark's memory usage</a> seems like it is not yet possible to monitor execution memory. Also this seems relevant <a href="https://issues.apache.org/jira/browse/SPARK-23206" rel="noreferrer">SPARK-23206 Additional Memory Tuning Metrics</a>.</p></li>
  <li><p>Does <code>Peak Execution memory</code> is reliable estimate of usage/occupation of execution memory in a task? If for example it a Stage UI says that a task uses 1 Gb at peak, and I have 5 cpu per executor, does it mean I need at least 5 Gb execution memory available on each executor to finish a stage?</p></li>
  <li><p>Are there some other proxies we could use to get a glimpse of execution memory?</p></li>
  <li><p>Is there a way to know when the execution memory starts to eat into storage memory? When my cached table disappears from Storage tab in SparkUI or only part of it remains, does it mean it was evicted by the execution memory?</p></li>
  </ol>