title: Reading remote HDFS file with Java
body: <p>I’m having a bit of trouble with a simple Hadoop install.  I’ve downloaded hadoop 2.4.0 and installed on a single CentOS Linux node (Virtual Machine).  I’ve configured hadoop for a single node with pseudo distribution as described on the apache site (<a href="http://hadoop.apache.org/docs/r2.4.0/hadoop-project-dist/hadoop-common/SingleCluster.html" rel="noreferrer">http://hadoop.apache.org/docs/r2.4.0/hadoop-project-dist/hadoop-common/SingleCluster.html</a>).  It starts with no issues in the logs and I can read + write files using the “hadoop fs” commands from the command line.</p>
  
  <p>I’m attempting to read a file from the HDFS on a remote machine with the Java API.  The machine can connect and list directory contents.  It can also determine if a file exists with the code:</p>
  
  <pre><code>Path p=new Path("hdfs://test.server:9000/usr/test/test_file.txt");
  FileSystem fs = FileSystem.get(new Configuration());
  System.out.println(p.getName() + " exists: " + fs.exists(p));
  </code></pre>
  
  <p>The system prints “true” indicating it exists.  However, when I attempt to read the file with:</p>
  
  <pre><code>BufferedReader br = null;
  try {
      Path p=new Path("hdfs://test.server:9000/usr/test/test_file.txt");
      FileSystem fs = FileSystem.get(CONFIG);
      System.out.println(p.getName() + " exists: " + fs.exists(p));
  
      br=new BufferedReader(new InputStreamReader(fs.open(p)));
      String line = br.readLine();
  
      while (line != null) {
          System.out.println(line);
          line=br.readLine();
      }
  }
  finally {
      if(br != null) br.close();
  }
  </code></pre>
  
  <p>this code throws the exception:</p>
  
  <p><strong><em>Exception in thread "main" org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-13917963-127.0.0.1-1398476189167:blk_1073741831_1007 file=/usr/test/test_file.txt</em></strong></p>
  
  <p>Googling gave some possible tips but all checked out. The data node is connected, active, and has enough space.  The admin report from hdfs dfsadmin –report shows:</p>
  
  <blockquote>
    <p>Configured Capacity: 52844687360 (49.22 GB)<br/>
    Present Capacity: 48507940864 (45.18 GB)<br/>
    DFS Remaining: 48507887616 (45.18 GB)<br/>
    DFS Used: 53248 (52 KB)<br/>
    DFS Used%: 0.00%<br/>
    Under replicated blocks: 0<br/>
    Blocks with corrupt replicas: 0<br/>
    Missing blocks: 0<br/>
    <br/>
    Datanodes available: 1 (1 total, 0 dead)<br/>
    <br/>
    Live datanodes:<br/>
    Name: 127.0.0.1:50010 (test.server)<br/>
    Hostname: test.server<br/>
    Decommission Status : Normal <br/>
    Configured Capacity: 52844687360 (49.22 GB)<br/> 
    DFS Used: 53248 (52 KB) <br/>
    Non DFS Used: 4336746496 (4.04 GB) <br/>
    DFS Remaining: 48507887616 (45.18 GB) <br/>
    DFS Used%: 0.00% <br/>
    DFS Remaining%: 91.79%<br/>
    Configured Cache Capacity: 0 (0 B) <br/>
    Cache Used: 0 (0 B) <br/>
    Cache Remaining: 0 (0 B) <br/>
    Cache Used%: 100.00% <br/>
    Cache Remaining%: 0.00% <br/>
    Last contact: Fri Apr 25 22:16:56 PDT 2014</p>
  </blockquote>
  
  <p>The client jars were copied directly from the hadoop install so no version mismatch there.  I can browse the file system with my Java class and read file attributes.  I just can’t read the file contents without getting the exception.  If I try to write a file with the code:</p>
  
  <pre><code>FileSystem fs = null;
  BufferedWriter br = null;
  
  System.setProperty("HADOOP_USER_NAME", "root");
  
  try {
      fs = FileSystem.get(new Configuraion());
  
      //Path p = new Path(dir, file);
      Path p = new Path("hdfs://test.server:9000/usr/test/test.txt");
      br = new BufferedWriter(new OutputStreamWriter(fs.create(p,true)));
      br.write("Hello World");
  }
  finally {
      if(br != null) br.close();
      if(fs != null) fs.close();
  }
  </code></pre>
  
  <p>this creates the file but doesn’t write any bytes and throws the exception:</p>
  
  <p><strong><em>Exception in thread "main" org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /usr/test/test.txt could only be replicated to 0 nodes instead of minReplication (=1).  There are 1 datanode(s) running and 1 node(s) are excluded in this operation.</em></strong></p>
  
  <p>Googling for this indicated a possible space issue but from the dfsadmin report, it seems there is plenty of space.  This is a plain vanilla install and I can’t get past this issue.  </p>
  
  <p>The environment summary is:</p>
  
  <p>SERVER:</p>
  
  <p>Hadoop 2.4.0 with pseudo-distribution (<a href="http://hadoop.apache.org/docs/r2.4.0/hadoop-project-dist/hadoop-common/SingleCluster.html" rel="noreferrer">http://hadoop.apache.org/docs/r2.4.0/hadoop-project-dist/hadoop-common/SingleCluster.html</a>)</p>
  
  <p>CentOS 6.5 Virtual Machine 64 bit server
  Java 1.7.0_55</p>
  
  <p>CLIENT:</p>
  
  <p>Windows 8 (Virtual Machine)
  Java 1.7.0_51</p>
  
  <p>Any help is greatly appreciated.</p>