title: How to creating a MapFile with Spark and access it?
body: <p>I am trying to create a MapFile from a Spark RDD, but can't find enough information. Here are my steps so far:</p>
  
  <p>I started with,</p>
  
  <p><code>rdd.saveAsNewAPIHadoopFile(....MapFileOutputFormat.class)</code> </p>
  
  <p>which threw an Exception as the <code>MapFiles</code> must be sorted.
  So I modified to: </p>
  
  <p><code>rdd.sortByKey().saveAsNewAPIHadoopFile(....MapFileOutputFormat.class)</code> </p>
  
  <p>which worked fine and my MapFile was created. So the next step was  accessing the file. Using the directory name where <code>parts</code> were created failed saying that it cannot find the <code>data</code> file. Back to Google, I found that in order to access the <code>MapFile</code> parts I needed to use:</p>
  
  <pre><code>Object ret = new Object();//My actual WritableComparable impl
  Reader[] readers = MapFileOutputFormat.getReaders(new Path(file), new Configuration());
  Partitioner&lt;K,V&gt; p = new HashPartitioner&lt;&gt;();
  Writable e = MapFileOutputFormat.getEntry(readers, p key, ret);
  </code></pre>
  
  <p>Naively, I ignored the <code>HashPartioner</code> bit and expected that this would find my entry, but no luck. So my next step was to loop over the readers and do a <code>get(..)</code>. This solution did work, but it was extremely slow as the files were created by 128 tasks resulting in 128 <code>part</code> files.</p>
  
  <p>So I investigated the importance of <code>HashPartitioner</code> and found that internally it uses it to identify which reader to use, but it seems that Spark is not using the same partitioning logic. So I modified to:</p>
  
  <p><code>rdd.partitionBy(new org.apache.spark.HashPartitioner(128)).sortByKey().saveAsNewAPIHadoopFile(....MapFileOutputFormat.class)</code></p>
  
  <p>But again the 2 <code>HashPartioner</code> did not match. So the questions part...</p>
  
  <ul>
  <li>Is there a way to combine the <code>MapFiles</code> efficiently (as this would ignore the paritioning logic)?</li>
  <li><code>MapFileOutputFormat.getReaders(new Path(file), new
  Configuration());</code> is very slow. Can I identify the reader more
  efficiently?</li>
  <li>I am using MapR-FS as the underlying DFS. Will this be using the same <code>HashParitioner</code> implementation?</li>
  <li>Is there a way to avoid repartitioning, or should the data be sorted over the whole file? (In contrast to being sorted within the partition)</li>
  <li>I am also getting an exception <code>_SUCCESS/data does not exist</code>. Do I need to manually delete this file?</li>
  </ul>
  
  <p>Any links about this would be greatly appreciated.</p>
  
  <p>PS. If entries are sorted, then how is it possible to use the  <code>HashPartitioner</code> to locate the correct <code>Reader</code>? This would imply that data <code>parts</code> are <code>Hash Partitioned</code> and then Sorted by key. So I also tried <code>rdd.repartiotionAndSortWithinPartitions(new HashPartitioner(280))</code>, but again without any luck.</p>