title: How do you programmatically get the number of available slave nodes and number of executors per node at runtime in Spark/Scala?
body: <p>I want to optimize the number of partitions that I use to parallelize a collection into an RDD. Ideally I would do this with the following lines of code:</p>
  
  <pre><code>val numPartitions = numSlaveNodes * numExecutorsPerNode * numCoresPerExecutor
  val myRDD = sparkContext.parallelize(myCollection, numPartitions)
  </code></pre>
  
  <p>I have only been able to figure out how to get the numCoresPerExecutor so far, which I do with:</p>
  
  <pre><code>sparkSession.conf.get("spark.executor.cores").toInt
  </code></pre>
  
  <p>Is it possible to programmatically get the other two values (numSlaveNodes and numExecutorsPerNode)? I already know how to get the number of cores per executor.</p>