While trying to optimise a Spark job, I am having trouble understanding a delay of 3-4s in the launch of the second and of 6-7s third and fourth executors.

This is what I'm working with:

* Spark 2.2
* Two worker nodes having 8 cpu cores each. (master node separate)
* Executors are configured to use 3 cores each.

Following is the screenshot of the jobs tab in Spark UI.

The job is divided into three stages. As seen, second, third and fourth executors are added only during the second stage.
[![Job description](https://i.stack.imgur.com/3R3pq.png)](https://i.stack.imgur.com/3R3pq.png)

Following is the snap of the Stage 0.
[![Stage 0 description](https://i.stack.imgur.com/MvR40.png)](https://i.stack.imgur.com/MvR40.png)

And following the snap of the Stage 1.
[![Stage 1 description](https://i.stack.imgur.com/eQZU6.png)](https://i.stack.imgur.com/eQZU6.png)

As seen in the image above, executor 2 (on the same worker as the first) takes around 3s to launch. Executors 3 and 4 (on the second worker) taken even longer, approximately 6s.

I tried playing around with the `spark.locality.wait` variable : values of 0s, 1s, 1ms. But there does not seem to be any change in the launch times of the executors.

Is there some other reason for this delay? Where else can I look to understand this better?