Title: Spark not adding executors
Tags: apache-spark;pyspark
Question: My dataproc configurations are as follows
nodes - 3
master - 64 GB, 16 cores, 32 GB disk space
executors - 64 GB, 16 cores, 32 GB disk space
I submit the job with the following configuration:
--properties=spark.executor.memory=19g,
spark.executor.instances=5,
spark.executor.cores=5,
spark.driver.memory=40g,
spark.default.parallelism=50,
spark.driver.cores=5,
spark.driver.memoryOverhead=1843m,
spark.executor.memoryOverhead=1843m,
spark.dynamicAllocation.enabled=false
My calculations are as follows:
2 workers totalling 128 GB RAM and 32 cores
1 core and 1 GB for hadoop and system daemons - 126 GB and 30 cores left
6 executors with 5 cores each (totaling 30 cores)
1 executor reserved for application master
19 GB per executor with less than 1.9 GB memory overhead (20.9 per executor). Overall memory allocated 20.9*6 = 125.4 GB (less than 126 GB)
But, my spark UI shows that only 4 executors are getting added
Why is my spark job not starting 5 executors? I see the same 4 executors only even if I pass --spark.executor.instances 6



{'source': 'StackExchangeDataset', 'subset': 'stackoverflow.com', 'metadata': {'AnswerCount': 0, 'CommentCount': 0, 'ContentLicense': 'CC BY-SA 4.0', 'CreationDate': '2022:00:02/19/22', 'FavoriteCount': None, 'PostTypeId': 1, 'Tags': '<apache-spark><pyspark>', 'Title': 'Spark not adding executors', 'ViewCount': 118, 'id': 71187565, 'user_AboutMe': '<p>"Be careful, lest in casting out your demon you exorcise the best thing in you." - Fredrick Nietzsche</p>\n', 'user_AccountId': 4548208.0, 'user_DisplayName': 'MiloMinderbinder', 'user_Reputation': 3460.0, 'user_Views': 344.0, 'user_id': 8089695.0}}