Is Microsoft Azure running jobs in parallel automatically?
The Spark job you're describing does not have the same meaning with sklearn's job (which defining the parallelism via n_jobs parameter).
Spark does run your classifier in parallel (in the background). The "Job 1" and "Job 2" etc is more about running some sequential steps, one after another, and each of them still running with multiple executors behind the scene.
I am running a classification model in Microsoft Azure using the pyspark.ml.classification library with RandomForestClassifier.
My question:
I know in sklearn.ensemble.RandomForestClassifier you can specify the n_jobs parameter to configure number of jobs to run in parallel.
When using pyspark.ml.classification.RandomForestClassifier in Azure, I find that each job is run separately. It first runs, Job 1, when done it runs Job 2 etc.
Is there a way to specify the number of jobs to run in parallel in the pyspark.ml.classification.RandomForestClassifier function?