title: /var/mapr/cluster/yarn/rm/staging/username/.staging/job_id_1234_4321/libjars/myudfs.jar (Invalid argument)
body: <p>We are using Mapr with PAM Authentication to execute select query on hive. Select query uses a myudfs.jar where we have defined our UDFs. </p>
  
  <p>I have tried many links but couldn't figure out why this is happening. From the stacktrace it seems like Hadoop is not able to copy the jar into libjars directory inside /.staging. But UDF is there in the classpath.</p>
  
  <p>Any help would be really appreciated.</p>
  
  <pre><code>java.sql.SQLException: org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. 2063.142.526190 /var/mapr/cluster/yarn/rm/staging/username/.staging/job_112233333_0002/libjars/myudfs.jar (Invalid argument)
  
  at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:380)
  
  at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:257)
  
  at org.apache.hive.service.cli.operation.SQLOperation.access$800(SQLOperation.java:91)
  
  at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:348)
  
  at java.security.AccessController.doPrivileged(Native Method)
  
  at javax.security.auth.Subject.doAs(Subject.java:422)
  
  at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1669)
  
  at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:362)
  
  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  
  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  
  at java.lang.Thread.run(Thread.java:748)
  
  Caused by: java.io.IOException: 2063.142.526190 /var/mapr/cluster/yarn/rm/staging/username/.staging/job_112233333_0002/libjars/myudfs.jar (Invalid argument)
  
  at com.mapr.fs.Inode.throwIfFailed(Inode.java:390)
  
  at com.mapr.fs.Inode.flushPages(Inode.java:505)
  
  at com.mapr.fs.Inode.releaseDirty(Inode.java:583)
  
  at com.mapr.fs.MapRFsOutStream.dropCurrentPage(MapRFsOutStream.java:73)
  
  at com.mapr.fs.MapRFsOutStream.write(MapRFsOutStream.java:85)
  
  at com.mapr.fs.MapRFsDataOutputStream.write(MapRFsDataOutputStream.java:39)
  
  at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:87)
  
  at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:59)
  
  at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:119)
  
  at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:376)
  
  at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:346)
  
  at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:297)
  
  at org.apache.hadoop.mapreduce.JobResourceUploader.copyRemoteFiles(JobResourceUploader.java:203)
  
  at org.apache.hadoop.mapreduce.JobResourceUploader.uploadFiles(JobResourceUploader.java:128)
  
  at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:98)
  
  at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:193)
  
  at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
  
  at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)
  
  at java.security.AccessController.doPrivileged(Native Method)
  
  at javax.security.auth.Subject.doAs(Subject.java:422)
  
  at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1669)
  
  at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
  
  at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:575)
  
  at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:570)
  
  at java.security.AccessController.doPrivileged(Native Method)
  
  at javax.security.auth.Subject.doAs(Subject.java:422)
  
  at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1669)
  
  at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:570)
  
  at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:561)
  
  at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:414)
  
  at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:151)
  
  at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:201)
  
  at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
  
  at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:79)
  
  
  
  at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:283)
  </code></pre>
  
  <p>Following is the process how we execute the query and what we do before executing it.</p>
  
  <p>First we create a function as given below
  CREATE FUNCTION func_name AS 'com.test.ClassContainingMapper'</p>
  
  <p>This ClassContainingMapper class is packed in a udf jar named myudf.jar. We have this jar added in the classpath by using property hive.aux.jars.path.</p>
  
  <p>Now, our query is as follows:
  Select func_name(col1, col2) from dbname.tablename;</p>
  
  <p>When this query is executed Hadoop tries to upload the jars mentioned in the classpath to it's libjars folder under staging directory. That's when it fails. </p>
  
  <p>Interesting thing is that on one similar cluster this query executes successfully but on other cluster it is being failed with the exception mentioned in the stacktrace. </p>
  
  <p>UPDATE:</p>
  
  <p>Actually there is another query executed before the Select query. It's</p>
  
  <p>ADD JAR '/path/to/the/jar/file/myudf.jar';</p>
  
  <p>This makes more sense that when this query is executed then the jar is uploaded to the cluster. And during that action of uploading query fails. </p>