title: How to set the number of partitions for newAPIHadoopFile?
body: <p>The "old" <code>SparkContext.hadoopFile</code> takes a <code>minPartitions</code> argument, which is a hint for the number of partitions:</p>
  
  <pre><code>def hadoopFile[K, V](
    path: String,
    inputFormatClass: Class[_ &lt;: InputFormat[K, V]],
    keyClass: Class[K],
    valueClass: Class[V],
    minPartitions: Int = defaultMinPartitions
    ): RDD[(K, V)]
  </code></pre>
  
  <p>But there is no such argument on <code>SparkContext.newAPIHadoopFile</code>:</p>
  
  <pre><code>def newAPIHadoopFile[K, V, F &lt;: NewInputFormat[K, V]](
    path: String,
    fClass: Class[F],
    kClass: Class[K],
    vClass: Class[V],
    conf: Configuration = hadoopConfiguration): RDD[(K, V)]
  </code></pre>
  
  <p>In fact <code>mapred.InputFormat.getSplits</code> takes a hint argument, but <code>mapreduce.InputFormat.getSplits</code> takes a <code>JobContext</code>. What is the way to influence the number of splits through the new API?</p>
  
  <p>I have tried setting <code>mapreduce.input.fileinputformat.split.maxsize</code> and <code>fs.s3n.block.size</code> on the <code>Configuration</code> object, but they had no effect. I am trying to load a 4.5 GB file from <code>s3n</code>, and it gets loaded in a single task.</p>
  
  <p><a href="https://issues.apache.org/jira/browse/HADOOP-5861">https://issues.apache.org/jira/browse/HADOOP-5861</a> is relevant, but it suggests that I should already see more than one split, since the default block size is 64 MB.</p>