title: How is Spark's default partitions calculated for HadoopPartitions?
body: <p>I'm reading <a href="https://jaceklaskowski.gitbooks.io/mastering-apache-spark/spark-rdd-partitions.html" rel="nofollow noreferrer">Jacek Laskowski's online book about Apache Spark</a>, and regarding partitioning, he states that</p>
  
  <blockquote>
    <p>By default, a partition is created for each HDFS partition, which by
    default is 64MB</p>
  </blockquote>
  
  <p>I'm not extremely familiar with HDFS, but I ran into some questions replicating this statement. I have a file called <code>Reviews.csv</code> which is about 330MB text file of Amazon food reviews. Given the default 64MB blocks, I'd expect <code>ceiling(330 / 64) = 6</code> partitions. However, when I load the files into my Spark Shell, I get 9 partitions:</p>
  
  <pre><code>scala&gt; val tokenized_logs = sc.textFile("Reviews.csv")
  tokenized_logs: org.apache.spark.rdd.RDD[String] = Reviews.csv MapPartitionsRDD[1] at textFile at &lt;console&gt;:24
  
  scala&gt; tokenized_logs
  res0: org.apache.spark.rdd.RDD[String] = Reviews.csv MapPartitionsRDD[1] at textFile at &lt;console&gt;:24
  
  scala&gt; tokenized_logs.partitions
  res1: Array[org.apache.spark.Partition] = Array(org.apache.spark.rdd.HadoopPartition@3c1, org.apache.spark.rdd.HadoopPartition@3c2, org.apache.spark.rdd.HadoopPartition@3c3, org.apache.spark.rdd.HadoopPartition@3c4, org.apache.spark.rdd.HadoopPartition@3c5, org.apache.spark.rdd.HadoopPartition@3c6, org.apache.spark.rdd.HadoopPartition@3c7, org.apache.spark.rdd.HadoopPartition@3c8, org.apache.spark.rdd.HadoopPartition@3c9)
  
  scala&gt; tokenized_logs.partitions.size
  res2: Int = 9
  </code></pre>
  
  <p>I do notice that if I create another smaller version of <code>Reviews.csv</code> called <code>Reviews_Smaller.csv</code> that is only 135MB, I have a significantly reduced partition size:</p>
  
  <pre><code>scala&gt; val raw_reviews = sc.textFile("Reviews_Smaller.csv")
  raw_reviews: org.apache.spark.rdd.RDD[String] = Reviews_Smaller.csv MapPartitionsRDD[11] at textFile at &lt;console&gt;:24
  
  scala&gt; raw_reviews.partitions.size
  res7: Int = 4
  </code></pre>
  
  <p>However, by my math, there should be <code>ceiling(135 / 4) = 3</code> partitions, not 4.</p>
  
  <p>I'm running everything locally, on my MacBook Pro. Can anyone help explain how the number of default partitions is calculated for HDFS?</p>