title: spark machine learning -- better design
body: <p>I am trying to implement a machine learning algorithm in Spark. The basic idea is to split the data into N partitions and learn N individual models on each individual dataset partition separately. During prediction we poll these N models and get their o/p and combine them.</p>
  
  <p>a) to partition data into N portions we use <code>repartition(N)</code> 
  b) to learn models on each partition we use <code>mapPartitionsWithIndex</code></p>
  
  <p>The problem is that multiple partitions map to the same physical machine -- which launches multiple model builds on the same physical node (invoked by <code>mapPartitionsWithIndex</code>) -- causing the node to run out of memory (we have maxed the executor memory).</p>
  
  <p>Is there a better way to design this ?     </p>