title: Is there a way to catch executor killed exception in Spark?
body: <p>During execution of my Spark program, sometimes (The reason for it is still a mystery to me) yarn kills containers (executors) giving the message that the memory limit was exceeded. My program does recover though with Spark re-executing the task by spawning a new container. However, in my program, a task also creates some intermediate files on the disk. When a container is killed, those files are left behind. Is there a way I can catch the executor-killed as an exception so that I can delete the intermediate files left behind. Obviously, the exception handling code also needs to run on the same node the executor was running on, so that I can delete the files from there.</p>