title: Namenode HA (UnknownHostException: nameservice1)
body: <p>We enable Namenode High Availability through Cloudera Manager, using  </p>
  
  <p>Cloudera Manager >> HDFS >> Action > Enable High Availability >> Selected Stand By Namenode &amp; Journal Nodes
   Then nameservice1 </p>
  
  <p>Once the whole process completed then Deployed Client Configuration.</p>
  
  <p>Tested from Client Machine by listing HDFS directories (hadoop fs -ls /) then manually failover to standby namenode &amp; again listing HDFS directories (hadoop fs -ls /). This test worked perfectly.</p>
  
  <p>But When I ran hadoop sleep job using following command it failed </p>
  
  <pre><code>$ hadoop jar /opt/cloudera/parcels/CDH-4.6.0-1.cdh4.6.0.p0.26/lib/hadoop-0.20-mapreduce/hadoop-examples.jar sleep -m 1 -r 0
  java.lang.IllegalArgumentException: java.net.UnknownHostException: nameservice1
  at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:414)
  at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:164)
  at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:129)
  at org.apache.hadoop.hdfs.DFSClient.&lt;init&gt;(DFSClient.java:448)
  at org.apache.hadoop.hdfs.DFSClient.&lt;init&gt;(DFSClient.java:410)
  at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:128)
  at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2308)
  at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:87)
  at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2342)
  at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2324)
  at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:351)
  at org.apache.hadoop.fs.Path.getFileSystem(Path.java:194)
  at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:103)
  at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:980)
  at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:974)
  at java.security.AccessController.doPrivileged(Native Method)
  at javax.security.auth.Subject.doAs(Subject.java:416)
  at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1438)
  at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:974)
  at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:948)
  at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1410)
  at org.apache.hadoop.examples.SleepJob.run(SleepJob.java:174)
  at org.apache.hadoop.examples.SleepJob.run(SleepJob.java:237)
  at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
  at org.apache.hadoop.examples.SleepJob.main(SleepJob.java:165)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:622)
  at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
  at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
  at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:622)
  at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
  Caused by: java.net.UnknownHostException: nameservice1
  ... 37 more
  </code></pre>
  
  <p>I dont know why its not able to resolved nameservice1 even after deploying client configuration.</p>
  
  <p>When I google this issue I found only one solution to this issue </p>
  
  <p>Add the below entry in configuration entry for fix the issue
  dfs.nameservices=nameservice1
  dfs.ha.namenodes.nameservice1=namenode1,namenode2
  dfs.namenode.rpc-address.nameservice1.namenode1=ip-10-118-137-215.ec2.internal:8020
  dfs.namenode.rpc-address.nameservice1.namenode2=ip-10-12-122-210.ec2.internal:8020
  dfs.client.failover.proxy.provider.nameservice1=org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</p>
  
  <p>My impression was Cloudera Manager take cares of it. I checked client for this configuration &amp; configuration was there (/var/run/cloudera-scm-agent/process/1998-deploy-client-config/hadoop-conf/hdfs-site.xml).</p>
  
  <p>Also some more details of config files :</p>
  
  <pre><code>[11:22:37 root@datasci01.dev:~]# ls -l /etc/hadoop/conf.cloudera.*
  /etc/hadoop/conf.cloudera.hdfs:
  total 16
  -rw-r--r-- 1 root root  943 Jul 31 09:33 core-site.xml
  -rw-r--r-- 1 root root 2546 Jul 31 09:33 hadoop-env.sh
  -rw-r--r-- 1 root root 1577 Jul 31 09:33 hdfs-site.xml
  -rw-r--r-- 1 root root  314 Jul 31 09:33 log4j.properties
  
  /etc/hadoop/conf.cloudera.hdfs1:
  total 20
  -rwxr-xr-x 1 root root  233 Sep  5  2013 container-executor.cfg
  -rw-r--r-- 1 root root 1890 May 21 15:48 core-site.xml
  -rw-r--r-- 1 root root 2546 May 21 15:48 hadoop-env.sh
  -rw-r--r-- 1 root root 1577 May 21 15:48 hdfs-site.xml
  -rw-r--r-- 1 root root  314 May 21 15:48 log4j.properties
  
  /etc/hadoop/conf.cloudera.mapreduce:
  total 20
  -rw-r--r-- 1 root root 1032 Jul 31 09:33 core-site.xml
  -rw-r--r-- 1 root root 2775 Jul 31 09:33 hadoop-env.sh
  -rw-r--r-- 1 root root 1450 Jul 31 09:33 hdfs-site.xml
  -rw-r--r-- 1 root root  314 Jul 31 09:33 log4j.properties
  -rw-r--r-- 1 root root 2446 Jul 31 09:33 mapred-site.xml
  
   /etc/hadoop/conf.cloudera.mapreduce1:
  total 24
  -rwxr-xr-x 1 root root  233 Sep  5  2013 container-executor.cfg
  -rw-r--r-- 1 root root 1979 May 16 12:20 core-site.xml
  -rw-r--r-- 1 root root 2775 May 16 12:20 hadoop-env.sh
  -rw-r--r-- 1 root root 1450 May 16 12:20 hdfs-site.xml
  -rw-r--r-- 1 root root  314 May 16 12:20 log4j.properties
  -rw-r--r-- 1 root root 2446 May 16 12:20 mapred-site.xml
  [11:23:12 root@datasci01.dev:~]# 
  </code></pre>
  
  <p>I doubt its issue with old configuration in /etc/hadoop/conf.cloudera.hdfs1 &amp; /etc/hadoop/conf.cloudera.mapreduce1 , but not sure.</p>
  
  <p>looks like /etc/hadoop/conf/* never got updated</p>
  
  <pre><code># ls -l /etc/hadoop/conf/
  total 24
  -rwxr-xr-x 1 root root  233 Sep  5  2013 container-executor.cfg
  -rw-r--r-- 1 root root 1979 May 16 12:20 core-site.xml
  -rw-r--r-- 1 root root 2775 May 16 12:20 hadoop-env.sh
  -rw-r--r-- 1 root root 1450 May 16 12:20 hdfs-site.xml
  -rw-r--r-- 1 root root  314 May 16 12:20 log4j.properties
  -rw-r--r-- 1 root root 2446 May 16 12:20 mapred-site.xml
  </code></pre>
  
  <p>Anyone has any idea about this issue?</p>