title: Hive Merge Small ORC Files
body: <p>My input consists of large number of small ORC files which I would like to merge every end of the day and I would like to split the data into 100MB blocks.</p>
  
  <p>My Input and Output Both Are S3 and Environment using is EMR,</p>
  
  <p>Hive Parameters which am setting,</p>
  
  <pre><code>set hive.msck.path.validation=ignore;
  set hive.exec.reducers.bytes.per.reducer=256000000;
  SET hive.exec.dynamic.partition = true;
  SET hive.exec.dynamic.partition.mode = nonstrict;
  SET hive.mapred.mode = nonstrict;
  
  set hive.merge.mapredfiles=true;
  set hive.merge.mapfile=true ;
  
  set hive.exec.parallel = true;
  set hive.exec.parallel.thread.number = 8;
  
  SET hive.exec.stagingdir=/tmp/hive/  ;
  SET hive.exec.scratchdir=/tmp/hive/ ;
  
  set mapred.max.split.size=68157440;
  set mapred.min.split.size=68157440;
  set hive.merge.smallfiles.avgsize=104857600;
  set hive.merge.size.per.task=104857600;
  set mapred.reduce.tasks=10;
  </code></pre>
  
  <p>My Insert Statement:</p>
  
  <pre><code>insert into table dev.orc_convert_zzz_18 partition(event_type) select * from dev.events_part_input_18 where event_type = 'ScreenLoad' distribute by event_type;
  </code></pre>
  
  <p>Now the problem is , I have around 80 input files which are of 500MB size in total and after this insert statement, I was expecting 4 files in S3, but all these files are getting merged into a single file which is not desired output.</p>
  
  <p>Can someone please let me know, what's going wrong ,</p>