title: Why is Spark saveAsTable with bucketBy creating thousands of files?
body: <p><strong>Context</strong></p>
  
  <p>Spark 2.0.1, spark-submit in cluster mode. I am reading a parquet file from hdfs:</p>
  
  <pre><code>val spark = SparkSession.builder
        .appName("myApp")
        .config("hive.metastore.uris", "thrift://XXX.XXX.net:9083")
        .config("spark.sql.sources.bucketing.enabled", true)
        .enableHiveSupport()
        .getOrCreate()
  
  val df = spark.read
                .format("parquet")
                .load("hdfs://XXX.XX.X.XX/myParquetFile")
  </code></pre>
  
  <p>I am saving the <code>df</code> to a hive table with 50 buckets grouped by <code>userid</code>:</p>
  
  <pre><code>df0.write
     .bucketBy(50, "userid")
     .saveAsTable("myHiveTable")
  </code></pre>
  
  <p>Now, when I look into the hive warehouse at my hdfs <code>/user/hive/warehouse</code> there is a folder named <code>myHiveTable</code>. Inside it are a bunch of <code>part-*.parquet</code> files. I would expect there to be 50 files. But no, there are <strong>3201</strong> files!!!! There are 64 files per partition, why? There are different number of files per partitions for different files I saved as hive table. All the files are very small, just tens of Kb each!</p>
  
  <p>Let me add, that number of different <code>userid</code> is about <code>1 000 000</code> in <code>myParquetFile</code>.</p>
  
  <p><strong>Question</strong></p>
  
  <p>Why are there 3201 files in the folder instead of 50! What are they? </p>
  
  <p>When I read this table back into DataFrame and print number of partitions:</p>
  
  <pre><code>val df2 = spark.sql("SELECT * FROM myHiveTable") 
  println(df2.rdd.getNumPartitions)
  </code></pre>
  
  <p>The number of partitions isIt is correctly 50 and I confirmed that the data is correctly partitioned by <code>userid</code>.</p>
  
  <p>For one of my large datasets 3Tb I create a table with 1000 partitions which created literally ~million of files! Which exceeds a directory item limit of 1048576 and gives <code>org.apache.hadoop.hdfs.protocol.FSLimitException$MaxDirectoryItemsExceededException</code> </p>
  
  <p><strong>Question</strong></p>
  
  <p>What does the number of files created depend on?</p>
  
  <p><strong>Question</strong></p>
  
  <p>Is there a way to limit number of files created?</p>
  
  <p><strong>Question</strong></p>
  
  <p>Should I worry about these files? Does it hurt performance on <code>df2</code> by having all these files? It is always said we should not create too many partitions because it is problematic. </p>
  
  <p><strong>Question</strong></p>
  
  <p>I found this info <a href="https://qubole.zendesk.com/hc/en-us/articles/214885263-HIVE-Dynamic-Partitioning-tips" rel="noreferrer">HIVE Dynamic Partitioning tips</a> that the number of files might be related to number of mappers. It is suggested to use <code>distribute by</code> while inserting to hive table. How could I do it in Spark?</p>
  
  <p><strong>Question</strong></p>
  
  <p>If the problem is indeed as in the link above, here <a href="http://www.openkb.info/2014/12/how-to-control-file-numbers-of-hive.html" rel="noreferrer">How to control the file numbers of hive table after inserting data on MapR-FS</a> they suggest using options such as <code>hive.merge.mapfiles</code> or <code>hive.merge.mapredfiles</code> to merge all the small files after map reduce job. Are there options for this in Spark?</p>