title: Writing to HDFS from Java, getting "could only be replicated to 0 nodes instead of minReplication"
body: <p>I’ve downloaded and started up Cloudera's Hadoop Demo VM for CDH4 (running Hadoop 2.0.0).  I’m trying to write a Java program that will run from my windows 7 machine (The same machine/OS that the VM is running in).  I have a sample program like:</p>
  
  <pre><code>public static void main(String[] args) {
      try{
          Configuration conf = new Configuration();
          conf.addResource("config.xml");
          FileSystem fs = FileSystem.get(conf);
          FSDataOutputStream fdos=fs.create(new Path("/testing/file01.txt"), true);
          fdos.writeBytes("Test text for the txt file");
          fdos.flush();
          fdos.close();
          fs.close();
      }catch(Exception e){
          e.printStackTrace();
      }
  
  }
  </code></pre>
  
  <p>My config.xml file only has on property defined: fs.default.name=hdfs://CDH4_IP:8020.</p>
  
  <p>When I run it I’m getting the following exception:</p>
  
  <pre><code>org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /testing/file01.txt could only be replicated to 0 nodes instead of minReplication (=1).  There are 1 datanode(s) running and 1 node(s) are excluded in this operation.
      at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1322)
      at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2170)
      at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:471)
      at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:297)
      at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:44080)
      at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
      at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
      at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:396)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
      at org.apache.hadoop.ipc.Client.call(Client.java:1160)
      at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
      at $Proxy9.addBlock(Unknown Source)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
      at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
      at $Proxy9.addBlock(Unknown Source)
      at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:290)
      at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1150)
      at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1003)
      at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:463)
  </code></pre>
  
  <p>I’ve looked around the internet and it seem that this happens when disk space is low but that’s not the case for me when I run "hdfs dfsadmin -report" I get the following:</p>
  
  <pre><code>Configured Capacity: 25197727744 (23.47 GB)
  Present Capacity: 21771988992 (20.28 GB)
  DFS Remaining: 21770715136 (20.28 GB)
  DFS Used: 1273856 (1.21 MB)
  DFS Used%: 0.01%
  Under replicated blocks: 0
  Blocks with corrupt replicas: 0
  Missing blocks: 0
  
  -------------------------------------------------
  Datanodes available: 1 (1 total, 0 dead)
  
  Live datanodes:
  Name: 127.0.0.1:50010 (localhost.localdomain)
  Hostname: localhost.localdomain
  Decommission Status : Normal
  Configured Capacity: 25197727744 (23.47 GB)
  DFS Used: 1273856 (1.21 MB)
  Non DFS Used: 3425738752 (3.19 GB)
  DFS Remaining: 21770715136 (20.28 GB)
  DFS Used%: 0.01%
  DFS Remaining%: 86.4%
  Last contact: Fri Jan 11 17:30:56 EST 201323 EST 2013
  </code></pre>
  
  <p>I can also run this code just fine from with in the VM. I’m not sure what the problem is or how to fix it.  This is my first time using hadoop so I’m probably missing something basic.  Any ideas?</p>
  
  <h1>Update</h1>
  
  <p>The only thing I see in the logs is an exception similar to the one on get on the client:</p>
  
  <pre><code>java.io.IOException: File /testing/file01.txt could only be replicated to 0 nodes instead of minReplication (=1).  There are 1 datanode(s) running and 1 node(s) are excluded in this operation.
      at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1322)
      at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2170)
      at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:471)
      at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:297)
      at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:44080)
      at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
      at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
      at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:396)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
  </code></pre>
  
  <p>I tried changing the permissions on the data directory (/var/lib/hadoop-hdfs/cache/hdfs/dfs/data) and that didn't fix it (I went so far as giving full access to everyone).</p>
  
  <p>I notice that when I'm browsing the HDFS via the HUE web app I see that the folder structure was created and that the file does exist but it is empty.  I tried putting the file under the default user directory by using</p>
  
  <pre><code>FSDataOutputStream fdos=fs.create(new Path("testing/file04.txt"), true); 
  </code></pre>
  
  <p>instead of</p>
  
  <pre><code>FSDataOutputStream fdos=fs.create(new Path("/testing/file04.txt"), true);
  </code></pre>
  
  <p>Which makes the file path become "/user/dharris/testing/file04.txt" ('dharris' is my windows user). But that gave me the same kind of error.</p>