title: Hive Vertex failed, vertexName=Map 2 while running on EMR for on big files
body: <p>I am running my hive query on EMR cluster that which is 25 nodes cluster and i have used r4.4xlarge in stances to run this .</p>
  
  <p>When i run my query i get below error .</p>
  
  <pre><code>Job Commit failed with exception 'org.apache.hadoop.hive.ql.metadata.HiveException(java.io.IOException: com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: FEAF40B78D086BEE; S3 Extended Request ID: yteHc4bRl1MrmVhqmnzm06rdzQNN8VcRwd4zqOa+rUY8m2HC2QTt9GoGR/Qu1wuJPILx4mchHRU=), S3 Extended Request ID: yteHc4bRl1MrmVhqmnzm06rdzQNN8VcRwd4zqOa+rUY8m2HC2QTt9GoGR/Qu1wuJPILx4mchHRU=)'
  FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.tez.TezTask
  /mnt/var/lib/hadoop/steps/s-10YQZ5Z5PRUVJ/./hive-script:617:in `&lt;main&gt;': Error executing cmd: /usr/share/aws/emr/scripts/hive-script "--base-path" "s3://us-east-1.elasticmapreduce/libs/hive/" "--hive-versions" "latest" "--run-hive-script" "--args" "-f" "s3://205067-pcfp-app-stepfun-s3appbucket-qa/2019-02-22_App/d77a6a82-26f4-4f06-a1ea-e83677256a55/01/DeltaOutPut/processing/Scripts/script.sql" (RuntimeError)
  Command exiting with ret '1'
  </code></pre>
  
  <p>I have tried settings all king of HIVE parameter combinations like below </p>
  
  <pre><code> emrfs-site fs.s3.consistent.retryPolicyType    exponential
  emrfs-site  fs.s3.consistent.metadata.tableName EmrFSMetadataAlt
  emrfs-site  fs.s3.consistent.metadata.write.capacity    300
  emrfs-site  fs.s3.consistent.metadata.read.capacity 600
  emrfs-site  fs.s3.consistent    true
  hive-site   hive.exec.stagingdir    .hive-staging
  hive-site   hive.tez.java.opts  -Xmx47364m
  hive-site   hive.stats.fetch.column.stats   true
  hive-site   hive.stats.fetch.partition.stats    true
  hive-site   hive.vectorized.execution.enabled   false
  hive-site   hive.vectorized.execution.reduce.enabled    false
  hive-site   tez.am.resource.memory.mb   15000
  hive-site   hive.auto.convert.join  false
  hive-site   hive.compute.query.using.stats  true
  hive-site   hive.cbo.enable true
  hive-site   tez.task.resource.memory.mb 16000
  </code></pre>
  
  <p>But every time it failed .
  I tried increasing the no of nodes/bigger instances  in the EMR cluster but result is still same .</p>
  
  <p>I also tried with and without Tez but still did not worked for me .</p>
  
  <p>Here is my sample query .I am just copying the part of my query </p>
  
  <pre><code>insert into filediffPcfp.TableDelta
  Select rgt.FILLER1,rgt.DUNSNUMBER,rgt.BUSINESSNAME,rgt.TRADESTYLENAME,rgt.REGISTEREDADDRESSINDICATOR
  </code></pre>
  
  <p>Please help me identify the issue .</p>
  
  <p>Adding full yarn logs </p>
  
  <pre><code>2019-02-26 06:28:54,318 [INFO] [TezChild] |exec.FileSinkOperator|: Final Path: FS s3://205067-pcfp-app-stepfun-s3appbucket-qa/2019-02-26_App/d996dfaa-1a62-4062-9350-d0c2bd62e867/01/DeltaOutPut/processing/Delta/.hive-staging_hive_2019-02-26_06-15-00_804_541842212852799084-1/_tmp.-ext-10000/000000_1
  2019-02-26 06:28:54,319 [INFO] [TezChild] |exec.FileSinkOperator|: Writing to temp file: FS s3://205067-pcfp-app-stepfun-s3appbucket-qa/2019-02-26_App/d996dfaa-1a62-4062-9350-d0c2bd62e867/01/DeltaOutPut/processing/Delta/.hive-staging_hive_2019-02-26_06-15-00_804_541842212852799084-1/_task_tmp.-ext-10000/_tmp.000000_1
  2019-02-26 06:28:54,319 [INFO] [TezChild] |exec.FileSinkOperator|: New Final Path: FS s3://205067-pcfp-app-stepfun-s3appbucket-qa/2019-02-26_App/d996dfaa-1a62-4062-9350-d0c2bd62e867/01/DeltaOutPut/processing/Delta/.hive-staging_hive_2019-02-26_06-15-00_804_541842212852799084-1/_tmp.-ext-10000/000000_1
  2019-02-26 06:28:54,681 [INFO] [TezChild] |exec.FileSinkOperator|: FS[11]: records written - 1
  2019-02-26 06:28:54,877 [INFO] [TezChild] |exec.MapOperator|: MAP[0]: records read - 1000
  2019-02-26 06:28:56,632 [INFO] [TezChild] |exec.MapOperator|: MAP[0]: records read - 10000
  2019-02-26 06:29:13,301 [INFO] [TezChild] |exec.MapOperator|: MAP[0]: records read - 100000
  2019-02-26 06:31:59,207 [INFO] [TezChild] |exec.MapOperator|: MAP[0]: records read - 1000000
  2019-02-26 06:34:42,686 [INFO] [TaskHeartbeatThread] |task.TaskReporter|: Received should die response from AM
  2019-02-26 06:34:42,686 [INFO] [TaskHeartbeatThread] |task.TaskReporter|: Asked to die via task heartbeat
  2019-02-26 06:34:42,687 [INFO] [TaskHeartbeatThread] |task.TezTaskRunner2|: Attempting to abort attempt_1551161362408_0001_7_01_000000_1 due to an invocation of shutdownRequested
  2019-02-26 06:34:42,687 [INFO] [TaskHeartbeatThread] |tez.TezProcessor|: Received abort
  2019-02-26 06:34:42,687 [INFO] [TaskHeartbeatThread] |tez.TezProcessor|: Forwarding abort to RecordProcessor
  2019-02-26 06:34:42,687 [INFO] [TaskHeartbeatThread] |tez.MapRecordProcessor|: Forwarding abort to mapOp: {} MAP
  2019-02-26 06:34:42,687 [INFO] [TaskHeartbeatThread] |exec.MapOperator|: Received abort in operator: MAP
  2019-02-26 06:34:42,705 [INFO] [TezChild] |s3.S3FSInputStream|: Encountered exception while reading '2019-02-26_App/d996dfaa-1a62-4062-9350-d0c2bd62e867/01/IncrFile/WB.ACTIVE.OCT17_01_OF_10.gz', will retry by attempting to reopen stream.
  com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.AbortedException: 
      at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.internal.SdkFilterInputStream.abortIfNeeded(SdkFilterInputStream.java:53)
      at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:81)
      at com.amazon.ws.emr.hadoop.fs.s3n.InputStreamWithInfo.read(InputStreamWithInfo.java:173)
      at com.amazon.ws.emr.hadoop.fs.s3.S3FSInputStream.read(S3FSInputStream.java:136)
      at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
      at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
      at java.io.DataInputStream.read(DataInputStream.java:149)
      at org.apache.hadoop.io.compress.DecompressorStream.getCompressedData(DecompressorStream.java:179)
      at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:163)
      at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:105)
      at java.io.InputStream.read(InputStream.java:101)
      at org.apache.hadoop.util.LineReader.fillBuffer(LineReader.java:182)
      at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:218)
      at org.apache.hadoop.util.LineReader.readLine(LineReader.java:176)
      at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:255)
      at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:48)
      at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:360)
      at org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:79)
      at org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:33)
      at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:116)
      at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(TezGroupedSplitsInputFormat.java:151)
      at org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:116)
      at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:62)
  </code></pre>