title: Best approach for looping through a csv file in spark
body: <p>I am new to spark. I am following some of the basic example in the documentation.</p>
  
  <p>I have a csv file like this: (a simplified version, the real one has nearly 40,000 lines)</p>
  
  <pre><code>date,category
  19900108,apples
  19900108,apples
  19900308,peaches
  19900408,peaches
  19900508,pears
  19910108,pears
  19910108,peaches
  19910308,apples
  19910408,apples
  19910508,apples
  19920108,pears
  19920108,peaches
  19920308,apples
  19920408,peaches
  19920508,pears
  </code></pre>
  
  <p>This bit of scala code works fine for counting category totals</p>
  
  <pre><code>val textFile = sc.textFile("sample.csv")
  textFile.filter(line =&gt; line.contains("1990")).filter(line =&gt;line.contains("peaches")).count()
  textFile.filter(line =&gt; line.contains("1990")).filter(line =&gt; line.contains("apples")).count()
  textFile.filter(line =&gt; line.contains("1990")).filter(line =&gt; line.contains("pears")).count()
  </code></pre>
  
  <p>What is the best approach for looping through each line, adding category totals by year so that I end up writing a csv file like this:</p>
  
  <pre><code>date,apples,peaches,pears
  1990,2,2,1
  1991,3,1,1
  1992,1,2,2
  </code></pre>
  
  <p>Any help would be appreciated.</p>