title: Failed to connect to server: localhost/127.0.0.1:9000: try once and fail. java.net.ConnectException: Connection refused
body: <p>I'm trying to put a file into my local hdfs by running this: <code>hadoop fs -put part-00000 /hbase/</code>, it gave me this:</p>
  
  <pre><code>17/05/30 16:11:52 WARN ipc.Client: Failed to connect to server: localhost/127.0.0.1:9000: try once and fail.
  java.net.ConnectException: Connection refused
      at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
      at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
      at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
      at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
      at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:681)
      at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:777)
      at org.apache.hadoop.ipc.Client$Connection.access$3500(Client.java:409)
      at org.apache.hadoop.ipc.Client.getConnection(Client.java:1542)
      at org.apache.hadoop.ipc.Client.call(Client.java:1373)
      at org.apache.hadoop.ipc.Client.call(Client.java:1337)
      at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
      at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
      at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source)
      at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:787)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:398)
      at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163)
      at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155)
      at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
      at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:335)
      at com.sun.proxy.$Proxy11.getFileInfo(Unknown Source)
      at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1700)
      at org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1436)
      at org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1433)
      at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
      at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1433)
      at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:64)
      at org.apache.hadoop.fs.Globber.doGlob(Globber.java:269)
      at org.apache.hadoop.fs.Globber.glob(Globber.java:148)
      at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1685)
      at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:326)
      at org.apache.hadoop.fs.shell.CommandWithDestination.getRemoteDestination(CommandWithDestination.java:195)
      at org.apache.hadoop.fs.shell.CopyCommands$Put.processOptions(CopyCommands.java:256)
      at org.apache.hadoop.fs.shell.Command.run(Command.java:164)
      at org.apache.hadoop.fs.FsShell.run(FsShell.java:315)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
      at org.apache.hadoop.fs.FsShell.main(FsShell.java:378)
  put: Call From steves-macbook-pro.local/172.29.16.117 to localhost:9000 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
  </code></pre>
  
  <p>before that I did: <code>$hadoop fs -mkdir /hbase</code> which ran successfully.</p>
  
  <p>I checked my logs for datanode, here's it:</p>
  
  <pre><code>2017-05-30 16:21:48,137 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server: localhost/127.0.0.1:9000
  2017-05-30 16:21:54,147 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:9000. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
  2017-05-30 16:21:55,150 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:9000. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
  2017-05-30 16:21:56,154 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:9000. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
  2017-05-30 16:21:57,158 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:9000. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
  2017-05-30 16:21:58,162 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:9000. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
  2017-05-30 16:21:59,165 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:9000. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
  2017-05-30 16:22:00,168 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:9000. Already tried 6 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
  2017-05-30 16:22:01,174 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:9000. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
  2017-05-30 16:22:02,179 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:9000. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
  2017-05-30 16:22:03,183 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:9000. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
  2017-05-30 16:22:03,183 WARN org.apache.hadoop.ipc.Client: Failed to connect to server: localhost/127.0.0.1:9000: retries get failed due to exceeded maximum allowed retries number: 10
  java.net.ConnectException: Connection refused
          at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
          at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
          at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
          at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
          at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
          at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:681)
          at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:777)
          at org.apache.hadoop.ipc.Client$Connection.access$3500(Client.java:409)
          at org.apache.hadoop.ipc.Client.getConnection(Client.java:1542)
          at org.apache.hadoop.ipc.Client.call(Client.java:1373)
          at org.apache.hadoop.ipc.Client.call(Client.java:1337)
          at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
          at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
          at com.sun.proxy.$Proxy15.versionRequest(Unknown Source)
          at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.versionRequest(DatanodeProtocolClientSideTranslatorPB.java:274)
          at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.retrieveNamespaceInfo(BPServiceActor.java:215)
          at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:261)
          at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:746)
          at java.lang.Thread.run(Thread.java:745)
  </code></pre>
  
  <p>I found a couple very similar questions on StackOverflow, in sum, here's what I've tried:</p>
  
  <pre><code>/usr/local/Cellar/hadoop/2.8.0/sbin/stop-all.sh
  
  /usr/local/Cellar/hadoop/2.8.0/bin/hdfs namenode -format
  
  /usr/local/Cellar/hadoop/2.8.0/sbin/start-all.sh
  
  /usr/local/Cellar/hadoop/2.8.0/sbin/start-dfs.sh
  </code></pre>
  
  <p>then I do a $jps, this is what I have:</p>
  
  <pre><code>13568 Main
  23154 NodeManager
  13477 HMaster
  21927 DataNode
  12696 Launcher
  13674 GradleDaemon
  22042 SecondaryNameNode
  23052 ResourceManager
  23502 Jps
  </code></pre>
  
  <p>Also, I've checked my <code>/usr/local/Cellar/hadoop/2.8.0/libexec/etc/hadoop/core-site.xml</code>, it's pointing to localhost:9000</p>
  
  <pre><code>&lt;configuration&gt;
  &lt;property&gt;
  &lt;name&gt;hadoop.tmp.dir&lt;/name&gt;
  &lt;value&gt;/usr/local/Cellar/hadoop/hdfs/tmp&lt;/value&gt;
  &lt;description&gt;A base for other temporary directories.&lt;/description&gt;
  &lt;/property&gt;
  &lt;property&gt;
  &lt;name&gt;fs.default.name&lt;/name&gt;
  &lt;value&gt;hdfs://localhost:9000&lt;/value&gt;
  &lt;/property&gt;
  &lt;/configuration&gt;
  </code></pre>
  
  <p>So, somehow my hadoop service is NOT up? any pointers where I should go next please?</p>
  
  <p>Thanks a lot! Really appreciate it!</p>
  
  <p>Edit:</p>
  
  <p>I found something else really interesting/weird: (I'm not sure why this is happening and if it's related)</p>
  
  <ol>
  <li><p>when I don't have a <code>datanode</code> running, I'm able to access this web UI: <a href="http://localhost:50070/" rel="nofollow noreferrer">http://localhost:50070/</a> to see how my local hadoop is working.</p></li>
  <li><p>when I do</p></li>
  </ol>
  
  <p><code>/usr/local/Cellar/hadoop/2.8.0/bin/hdfs namenode -format</code></p>
  
  <p><code>/usr/local/Cellar/hadoop/2.8.0/sbin/stop-all.sh</code></p>
  
  <p><code>/usr/local/Cellar/hadoop/2.8.0/sbin/start-all.sh</code></p>
  
  <p>and then I do <code>jps</code>, I got a running <code>datanode</code>, but I cannot access the web UI anymore: <a href="http://localhost:50070/" rel="nofollow noreferrer">http://localhost:50070/</a></p>