title: Not able to get rid of famous DataStreamer Exception: org.apache.hadoop.ipc.RemoteException: error
body: <p>I am running hadoop cluster with Ubuntu host as master-slave and virtual machine running on it as another slave(2 node cluster).</p>
  
  <p>It seems the solution to the problem which is supposed to be resolved at <a href="https://stackoverflow.com/questions/10097246/no-data-nodes-are-started">No data nodes are started</a> is not working for me. I tried both the solutions explained there.</p>
  
  <p>It seems that when i manually equate the namespace ids of the affected datanodes to name node
  and start the cluster(solution 2 in the linked post) i still get the same error( DataStreamer Exception).
  Next the logs of one of the datanode shows the same Incompatible namespaceIDs error, but the namespace id of data node which is shown in the log is different from that present my tmp/dfs/data/current/version file(which is not changed and is same as that of tmp/dfs/name/current/version)</p>
  
  <p>After many hours of debugging i am still clueless :(.</p>
  
  <p>PS: </p>
  
  <ul>
  <li><p>There is no connection problem from my host to slave</p></li>
  <li><p>When i start the cluster using start-dfs.sh then datanodes on both the nodes are started which is normall just to clarify.</p></li>
  <li><p>This error i am facing when i copy file from local to hdfs.</p></li>
  </ul>
  
  <p>I performed a simple test after all this</p>
  
  <ul>
  <li>Deleted the tmp/dfs/data and tmp/dfs/name folder on master </li>
  <li>Deleted    tmp/dfs/data on slave </li>
  <li>format the namenode using hadoop    namenode    -format </li>
  <li>started the cluster using start-dfs. all nodes    </li>
  <li>it started normally    on master and datanode is also up on slave </li>
  <li>now    ran    the       copyfromLocal command    and it gave me the
  same    error as below</li>
  <li>But this time there is no namespace mismatch error in any of  datanode logs master or slave</li>
  </ul>
  
  <blockquote>
    <p>14/05/04 04:12:54 WARN hdfs.DFSClient: DataStreamer Exception:
    org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
    /user/dsingh/mysample could only be replicated to 0 nodes, instead of
    1     at
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1920)
      at
    org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:783)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)     at
    org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)   at
    org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1432)  at
    org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1428)  at
    java.security.AccessController.doPrivileged(Native Method)    at
    javax.security.auth.Subject.doAs(Subject.java:415)    at
    org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1426)</p>
    
    <p>at org.apache.hadoop.ipc.Client.call(Client.java:1113)  at
    org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)    at
    com.sun.proxy.$Proxy1.addBlock(Unknown Source)    at
    sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)   at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)     at
    org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
      at
    org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
      at com.sun.proxy.$Proxy1.addBlock(Unknown Source)   at
    org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:3720)
      at
    org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3580)
      at
    org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2600(DFSClient.java:2783)
      at
    org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:3023)</p>
    
    <p>14/05/04 04:12:54 WARN hdfs.DFSClient: Error Recovery for null bad
    datanode[0] nodes == null 14/05/04 04:12:54 WARN hdfs.DFSClient: Could
    not get block locations. Source file "/user/dsingh/mysample" -
    Aborting... put: java.io.IOException: File /user/dsingh/mysample could
    only be replicated to 0 nodes, instead of 1 14/05/04 04:12:54 ERROR
    hdfs.DFSClient: Failed to close file /user/dsingh/mysample
    org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
    /user/dsingh/mysample could only be replicated to 0 nodes, instead of
    1     at
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1920)
      at
    org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:783)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)     at
    org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)   at
    org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1432)  at
    org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1428)  at
    java.security.AccessController.doPrivileged(Native Method)    at
    javax.security.auth.Subject.doAs(Subject.java:415)    at
    org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1426)</p>
    
    <p>at org.apache.hadoop.ipc.Client.call(Client.java:1113)  at
    org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)    at
    com.sun.proxy.$Proxy1.addBlock(Unknown Source)    at
    sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)   at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)     at
    org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
      at
    org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
      at com.sun.proxy.$Proxy1.addBlock(Unknown Source)   at
    org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:3720)
      at
    org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3580)
      at
    org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2600(DFSClient.java:2783)
      at
    org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:3023)</p>
  </blockquote>
  
  <p>Any clue will help me .</p>