title: Parallelism in spark.mllib
body: <p>Let's say I have an object <code>data</code> of type <code>Array[RDD]</code>.
  I want to learn <strong>independent</strong> Machine learning models on each <code>RDD</code> in this object. For example, with random forests:</p>
  
  <pre><code>data.map{ d =&gt; RandomForest.trainRegressor(d,2,Map[Int,Int](),2,"auto","gini",2,10) }
  </code></pre>
  
  <p>When I launch this job with <code>spark-submit --master yarn-client ...</code>, the independent learning tasks does not seem to be parallelized on multiple nodes. Almost all the job is done by only one node (namely node 10 here), as it can be seen on the screenshot from the application UI:</p>
  
  <p><a href="https://i.stack.imgur.com/geUjo.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/geUjo.png" alt="enter image description here"></a></p>
  
  <p><em>Addendum</em></p>
  
  <p>For completeness, the whole code is below:</p>
  
  <pre><code>import org.apache.spark.SparkContext
  import org.apache.spark.SparkConf
  import org.apache.spark.mllib.linalg.Vectors
  import org.apache.spark.mllib.regression.LabeledPoint
  import org.apache.spark.mllib.tree.RandomForest
  
  
  object test {
    def main(args: Array[String]) {
  
      val conf = new SparkConf().setMaster("local").setAppName("test")
      conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
      val sc = new SparkContext(conf)
  
      // Load data
      val rawData = sc.textFile("data/mllib/sample_tree_data.csv")
      val data = rawData.map { line =&gt;
        val parts = line.split(',').map(_.toDouble)
        LabeledPoint(parts(0), Vectors.dense(parts.tail))
      }
  
      val CV_data = (1 to 100).toArray.map(_ =&gt; {val splits = data.randomSplit(Array(0.7, 0.3)) ;  splits(0)})
  
      CV_data.map(d =&gt; RandomForest.trainClassifier(d, 2, Map[Int, Int](), 2, "sqrt", "gini", 2, 100))
  
      sc.stop()
      System.exit(0)
    }
  }
  </code></pre>