title: Is using parallel collections encouraged in Spark
body: <p>Does it make sense to use parallel collections on Spark?</p>
  
  <p>All the Spark examples I saw so far always used RDD of a very simple data types (Single classes and Tuples). But in fact collections and specifically parallel collections may be used as residents of the RDD. </p>
  
  <p>The worker may have several cores available for execution and if a regular collection is used as RDD resident those extra cores will stay idle. </p>
  
  <p>Test I ran with local manager.</p>
  
  <pre><code>val conf: SparkConf = new SparkConf().setAppName("myApp").setMaster("local[2]")
  val sc = new SparkContext(conf)
  
  val l = List(1,2,3,4,5,6,7,8)
  val l1 = l.map(item =&gt; (item, 1 to item toList))
  val l2 = l1.map(item =&gt; (item._1, item._2.toParArray))
  val l3 = sc.parallelize(l2)
  l3.sortBy(_._1).foreach(t =&gt; t._2.map(x =&gt; {println(t._1 + " " +Thread.currentThread.getName); x / 2}))
  </code></pre>
  
  <p>In this case when I use parArray I see 16 threads working and when I used simple Array only 2 threads worked. This may be seen as 2 workers having avaialble 8 threads.</p>
  
  <p>On the other hand every logic of the parallel collection may be changed to RDD transformations of simple types.</p>
  
  <p>Is using those parallel collections encouraged and considered good practice? </p>