title: Running spark application doesn't show up on spark history server
body: <p>I am creating a long running spark application. After spark session has been created and application starts to run, I am not able to see it after click on the "show incomplete applications" on the spark history server. However, If I force my application to close, I can see it under the "completed applications" page. </p>
  
  <p>I have spark parameters configured correctly on both client and server, as follow: 
  spark.eventLog.enabled=true</p>
  
  <p>spark.eventLog.dir=hdfs://10.18.51.117:8020/history/ (a hdfs path on my spark history server)</p>
  
  <p>I also configured the same on server side. So configuration shouldn't be a concern (since completed applications can also show up after I force my application to stop).</p>
  
  <p>Do you guys have any thoughts on this behavior?? </p>
  
  <p>I look at the hdfs files on spark history server, I see a very small size .inprogress file associated with my running application (close to empty, see the picture below). It seems that the results get flushed to the file only when the application stops, which is not ideal for my long running application...Is there any way or parameters we can tweak to force flushing the log?  </p>
  
  <p><a href="https://i.stack.imgur.com/ogksa.jpg" rel="nofollow noreferrer">Very small size .inprogress file shown on hdfs during application is running</a></p>