title: Explanation of fold method of spark RDD
body: <p>I am running Spark-1.4.0 pre-built for Hadoop-2.4 (in local mode) to calculate the sum of squares of a DoubleRDD. My Scala code looks like </p>
  
  <pre><code>sc.parallelize(Array(2., 3.)).fold(0.0)((p, v) =&gt; p+v*v)
  </code></pre>
  
  <p>And it gave a surprising result <code>97.0</code>.</p>
  
  <p>This is quite counter-intuitive compared to the Scala version of <code>fold</code></p>
  
  <pre><code>Array(2., 3.).fold(0.0)((p, v) =&gt; p+v*v)
  </code></pre>
  
  <p>which gives the expected answer <code>13.0</code>.</p>
  
  <p>It seems quite likely that I have made some tricky mistakes in the code due to a lack of understanding. I have read about how the function used in <code>RDD.fold()</code> should be communicative otherwise the result may depend on partitions and etc. So example, if I change the number of partitions to 1, </p>
  
  <pre><code>sc.parallelize(Array(2., 3.), 1).fold(0.0)((p, v) =&gt; p+v*v)
  </code></pre>
  
  <p>the code will give me <code>169.0</code> on my machine!</p>
  
  <p>Can someone explain what exactly is happening here?</p>