title: Cassandra UNREACHABLE node
body: <p>I have Cassandra v3.9, 3 node cluster. Replication factor 2</p>
  
  <p>10.0.0.11,10.0.0.12,10.0.0.13</p>
  
  <p>10.0.0.11,10.0.0.12 are seed nodes</p>
  
  <p><strong>What could be the possible reasons</strong> for following error in /etc/cassandra/conf/debug.log</p>
  
  <p>The error is</p>
  
  <p>DEBUG [RMI TCP Connection(174)-127.0.0.1] 2017-07-12 04:47:49,002 StorageProxy.java:2254 - Hosts not in agreement. Didn't get a response from everybody: 10.0.0.13</p>
  
  <p><strong>UPDATE1:</strong> </p>
  
  <p>At the time of the error here are some statistics from all the servers</p>
  
  <pre><code>[user1@ip-10-0-0-11 ~]$ nodetool status
  Datacenter: datacenter1
  =======================
  Status=Up/Down
  |/ State=Normal/Leaving/Joining/Moving
  --  Address     Load       Tokens       Owns (effective)  Host ID                               Rack
  UN  10.0.0.12  2.55 MiB   256          67.2%             83a68750-2238-4a6e-87be-03a3d7246824  rack1
  UN  10.0.0.11  1.78 GiB   256          70.6%             052fda9d-0474-4dfb-b2f8-0c5cbec15266  rack1
  UN  10.0.0.13  1.78 GiB   256          62.2%             86438dc9-77e0-43b2-a672-5b2e7cf216bf  rack1
  
  [user1@ip-10-0-0-11 ~]$ nodetool describecluster
  Cluster Information:
      Name: PiedmontCluster
      Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
      Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
      Schema versions:
          3c8d9e82-c688-3d16-a3e9-b84894168283: [10.0.0.12, 10.0.0.11]
  
          UNREACHABLE: [10.0.0.13]
  
  
  [pnm@ip-10-0-0-13 ~]$ nodetool status
      Datacenter: datacenter1
      =======================
      Status=Up/Down
      |/ State=Normal/Leaving/Joining/Moving
      --  Address     Load       Tokens       Owns (effective)  Host ID                               Rack
      UN  10.0.0.12  2.55 MiB   256          67.2%             83a68750-2238-4a6e-87be-03a3d7246824  rack1
      UN  10.0.0.11  1.78 GiB   256          70.6%             052fda9d-0474-4dfb-b2f8-0c5cbec15266  rack1
      UN  10.0.0.13  1.78 GiB   256          62.2%             86438dc9-77e0-43b2-a672-5b2e7cf216bf  rack1
  
      [pnm@ip-10-0-0-13 ~]$ nodetool describecluster
          Cluster Information:
          Name: PiedmontCluster
          Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
          Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
          Schema versions:
              3c8d9e82-c688-3d16-a3e9-b84894168283: [10.0.0.12, 10.0.0.13]
  
          UNREACHABLE: [10.0.0.11]
  
  
  [user1@ip-10-0-0-12 ~]$ nodetool status
      Datacenter: datacenter1
      =======================
      Status=Up/Down
      |/ State=Normal/Leaving/Joining/Moving
      --  Address     Load       Tokens       Owns (effective)  Host ID                               Rack
      UN  10.0.0.12  2.55 MiB   256          67.2%             83a68750-2238-4a6e-87be-03a3d7246824  rack1
      UN  10.0.0.11  1.78 GiB   256          70.6%             052fda9d-0474-4dfb-b2f8-0c5cbec15266  rack1
      UN  10.0.0.13  1.78 GiB   256          62.2%             86438dc9-77e0-43b2-a672-5b2e7cf216bf  rack1
  
      [user1@ip-10-0-0-12 ~]$ nodetool describecluster
          Cluster Information:
          Name: PiedmontCluster
          Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
              Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
              Schema versions:
                  3c8d9e82-c688-3d16-a3e9-b84894168283: [10.0.0.12, 10.0.0.11, 10.0.0.13]
  </code></pre>
  
  <p>The above mentioned error is in the /var/log/cassandra/debug.log on 10.0.0.11</p>
  
  <p>Error in /var/log/cassandra/debug.php on 10.0.0.13 is</p>
  
  <pre><code>    DEBUG [RMI TCP Connection(4)-127.0.0.1] 2017-07-13 02:31:23,846       StorageProxy.java:2254 - Hosts not in agreement. Didn't get a response from everybody: 10.0.0.11
      ERROR [MessagingService-Incoming-/10.0.0.11] 2017-07-13 02:35:04,982 CassandraDaemon.java:226 - Exception in thread Thread[MessagingService-Incoming-/10.0.0.11,5,main]
      java.lang.ArrayIndexOutOfBoundsException: 4
          at  org.apache.cassandra.db.filter.AbstractClusteringIndexFilter$FilterSerializer.deserialize(AbstractClusteringIndexFilter.java:74) ~[apache-cassandra-3.9.0.jar:3.9.0]
          at org.apache.cassandra.db.SinglePartitionReadCommand$Deserializer.deserialize(SinglePartitionReadCommand.java:1041) ~[apache-cassandra-3.9.0.jar:3.9.0]
          at org.apache.cassandra.db.ReadCommand$Serializer.deserialize(ReadCommand.java:696) ~[apache-cassandra-3.9.0.jar:3.9.0]
          at org.apache.cassandra.db.ReadCommand$Serializer.deserialize(ReadCommand.java:626) ~[apache-cassandra-3.9.0.jar:3.9.0]
          at org.apache.cassandra.io.ForwardingVersionedSerializer.deserialize(ForwardingVersionedSerializer.java:50) ~[apache-cassandra-3.9.0.jar:3.9.0]
          at org.apache.cassandra.net.MessageIn.read(MessageIn.java:114) ~[apache-cassandra-3.9.0.jar:3.9.0]
          at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:190) ~[apache-cassandra-3.9.0.jar:3.9.0]
          at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:178) ~[apache-cassandra-3.9.0.jar:3.9.0]
          at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:92) ~[apache-cassandra-3.9.0.jar:3.9.0]
  </code></pre>
  
  <p>No error in /var/log/cassandra/debug.php on 10.0.0.12</p>
  
  <p>Remember 10.0.0.11 &amp; 10.0.0.12 are seed nodes</p>
  
  <p>Thanks</p>