title: HDFS Data Transfer from One cluster to Another is not working with distcp
body: <p>I need to transfer HDFS data from one cluster to another. I see "distcp" command to be helpful for this case. But it was not. Both cluster Namenode is privately interconnected with other datanodes. So I have two proxy machines to connect publically with the namenode. Say, I made namenode's 8070 port to run under 20000 in haproxy. Now I can ping both clusters namenode. So, I went for distcp option. There the mapreduce job starts executing for the data transfer but it is not completing.</p>
  
  <pre><code>[hdfs@ip-20-0-42-252 ~]$ hadoop distcp  hdfs://YY.YY.YY.YY:20000/user/ce_prasith/filter.txt  hdfs://xx.xx.xx.xx:20000/user/gl_qauser
  18/10/09 10:12:15 INFO tools.DistCp: Input Options: DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false, ignoreFailures=false, overwrite=false, append=false, useDiff=false, useRdiff=false, fromSnapshot=null, toSnapshot=null, skipCRC=false, blocking=true, numListstatusThreads=0, maxMaps=20, mapBandwidth=100, sslConfigurationFile='null', copyStrategy='uniformsize', preserveStatus=[], preserveRawXattrs=false, atomicWorkPath=null, logPath=null, sourceFileListing=null, sourcePaths=[hdfs:/user/ce_prasith/filter.txt], targetPath=hdfs://xx.xx.xx.xx:20000/user/gl_qauser, targetPathExists=true, filtersFile='null'}
  18/10/09 10:12:16 INFO tools.SimpleCopyListing: Paths (files+dirs) cnt = 1; dirCnt = 0
  18/10/09 10:12:16 INFO tools.SimpleCopyListing: Build file listing completed.
  18/10/09 10:12:16 INFO Configuration.deprecation: io.sort.mb is deprecated. Instead, use mapreduce.task.io.sort.mb
  18/10/09 10:12:16 INFO Configuration.deprecation: io.sort.factor is deprecated. Instead, use mapreduce.task.io.sort.factor
  18/10/09 10:12:16 INFO tools.DistCp: Number of paths in the copy list: 1
  18/10/09 10:12:16 INFO tools.DistCp: Number of paths in the copy list: 1
  18/10/09 10:12:16 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm97
  18/10/09 10:12:16 INFO mapreduce.JobSubmitter: number of splits:1
  18/10/09 10:12:16 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1539063069030_0003
  18/10/09 10:12:16 INFO impl.YarnClientImpl: Submitted application application_1539063069030_0003
  18/10/09 10:12:17 INFO mapreduce.Job: The url to track the job: http://ip-20-0-21-94.ec2.internal:8088/proxy/application_1539063069030_0003/
  18/10/09 10:12:17 INFO tools.DistCp: DistCp job-id: job_1539063069030_0003
  18/10/09 10:12:17 INFO mapreduce.Job: Running job: job_1539063069030_0003
  18/10/09 10:12:22 INFO mapreduce.Job: Job job_1539063069030_0003 running in uber mode : true
  18/10/09 10:12:22 INFO mapreduce.Job:  map 0% reduce 0%
  18/10/09 10:13:22 INFO mapreduce.Job:  map 100% reduce 0%
  </code></pre>
  
  <p>For your information I have taken few logs of the job</p>
  
  <pre><code>2018-10-09 12:01:42,715 WARN [CommitterEvent Processor #2] org.apache.hadoop.tools.mapred.CopyCommitter: Unable to cleanup temp files
  org.apache.hadoop.net.ConnectTimeoutException: Call From ip-YY.YY.YY.YY.ec2.internal/YY.YY.YY.YY to ec2-xx.xx.xx.xx.ap-south-1.compute.amazonaws.com:20000 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 20000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=ec2-xx.xx.xx.xx.ap-south-1.compute.amazonaws.com/xx.xx.xx.xx:20000]; For more details see:  http://wiki.apache.org/hadoop/SocketTimeout
      at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
      at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
      at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:791)
      at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:750)
      at org.apache.hadoop.ipc.Client.call(Client.java:1508)
      at org.apache.hadoop.ipc.Client.call(Client.java:1441)
      at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
      at com.sun.proxy.$Proxy10.getListing(Unknown Source)
      at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getListing(ClientNamenodeProtocolTranslatorPB.java:573)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256)
      at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
      at com.sun.proxy.$Proxy11.getListing(Unknown Source)
      at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:2101)
      at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:2084)
      at org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:731)
      at org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:110)
      at org.apache.hadoop.hdfs.DistributedFileSystem$16.doCall(DistributedFileSystem.java:796)
      at org.apache.hadoop.hdfs.DistributedFileSystem$16.doCall(DistributedFileSystem.java:792)
      at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
      at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:792)
      at org.apache.hadoop.fs.Globber.listStatus(Globber.java:76)
      at org.apache.hadoop.fs.Globber.doGlob(Globber.java:237)
      at org.apache.hadoop.fs.Globber.glob(Globber.java:151)
      at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1714)
      at org.apache.hadoop.tools.mapred.CopyCommitter.deleteAttemptTempFiles(CopyCommitter.java:145)
      at org.apache.hadoop.tools.mapred.CopyCommitter.cleanupTempFiles(CopyCommitter.java:131)
      at org.apache.hadoop.tools.mapred.CopyCommitter.abortJob(CopyCommitter.java:118)
      at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobAbort(CommitterEventHandler.java:298)
      at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:240)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)
  Caused by: org.apache.hadoop.net.ConnectTimeoutException: 20000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=ec2-xx.xx.xx.xx.ap-south-1.compute.amazonaws.com/xx.xx.xx.xx:20000]
      at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:533)
      at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:494)
      at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:648)
      at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:744)
      at org.apache.hadoop.ipc.Client$Connection.access$3000(Client.java:396)
      at org.apache.hadoop.ipc.Client.getConnection(Client.java:1557)
      at org.apache.hadoop.ipc.Client.call(Client.java:1480)
      ... 31 more
  2018-10-09 12:01:42,716 INFO [CommitterEvent Processor #2] org.apache.hadoop.tools.mapred.CopyCommitter: Cleaning up temporary work folder: /user/hdfs/.staging/_distcp1087004350
  </code></pre>
  
  <p>I get stuck over here. Does somebody have any idea to overcome this? </p>