API to find out how many executors are running my Spark jobs? I'm writing code that captures statistics from Spark runs and stores the results in XML files. I'd like to capture, ideally with the REST API, information about the number of executors and the number and size of the Data Frames that were created. Is there an easy way to do this? What are the API calls?
Keep in mind that the number of executors is independent of the number of partitions of your dataframe. You set the number of executors when creating SparkConf() object.

https://spark.apache.org/docs/latest/submitting-applications.html

To retrieve the number of executors from code, it depends of the API you use. For python, this is not currently implemented. For Scala, you can. Here is a good stack post : 

https://stackoverflow.com/questions/38660907/how-to-get-the-number-of-workersexecutors-in-pyspark

Number of partitions refers to the number of parts your dataframe is composed of. With pyspark API, you can get it by transform your dataframe to a rdd and then use method .getNumPartitions() :

num_partitions = df.rdd.getNumPartitions()