title: Spark 2.3.1 on YARN : how to monitor stages progress programatically?
body: <p>I have a setup with Spark running on YARN, and my goal is to programmatically get updates of the progress of a Spark job by its application id.</p>
  
  <p>My first idea was to parse HTML output of the YARN GUI. However, the problem of such GUI, is that the progress bar associated to a spark job don't get updated regularly and even don't change most of time : when the job start, the percent is something like 10%, and it stuck to this value until the job finish. So such YARN progress bar is just irrelevant for Spark Jobs.</p>
  
  <p>When I click to the Application Master link corresponding to a Spark Job, I'm redirected to the Spark GUI that is temporarily binded during the job run. The stages page is very relevant about progress of the Spark job. However it is plain HTML, so it is a pain to parse. On the Spark documentation, they talk about a JSON API, however it seems that I can't access to it as I'm under YARN and I'm accessing Spark GUI trough YARN proxy pages.</p>
  
  <p>May be a solution, in order to have access to more things, could be to access to the real Spark GUI ip:port, and not the YARN proxied one, but I don't know if I can get such source URL easily...</p>
  
  <p>All of that sound complicated to just get Spark job progress... As of 2018, could you please tell us what are the preferred methods to get relevant <em>stages progress</em> of a Spark Job running on YARN ?</p>