title: How can I parallelize a for loop in spark with scala?
body: <p>For example, we have a parquet file with 2000 stock symbols' closing price in the past 3 years, and we want to calculate the 5-day moving average for each symbol.</p>
  
  <p>So I create a spark SQLContext and then</p>
  
  <pre><code>val marketData = sqlcontext.sql("select DATE, SYMBOL, PRICE from stockdata order by DATE").cache()
  </code></pre>
  
  <p>To get the symbol list,</p>
  
  <pre><code>val symbols = marketData.select("SYMBOL").distinct().collect()
  </code></pre>
  
  <p>and here is the for loop:</p>
  
  <pre><code>for (symbol &lt;- symbols) {
    marketData.filter(symbol).rdd.sliding(5).map(...calculating the avg...).save()
  }
  </code></pre>
  
  <p>Obviously, doing the for loop on spark is slow, and <code>save()</code> for each small result also slows down the process (I have tried define a <code>var result</code> outside the for loop and union all the output to make the IO operation together, but I got a stackoverflow exception), so how can I parallelize the for loop and optimize the IO operation?</p>