title: Cassandra Kubernetes Statefulset NoHostAvailableException
body: <p>I have an application deployed in kubernetes, it consists of cassandra, a go client, and a java client (and other things, but they are not relevant for this discussion).
  We have used helm to do our deployment.
  We are using a stateful set and a headless service for cassandra.
  We have configured the clients to use the headless service dns as a contact point for cluster creation.</p>
  <p>Everything works great.
  Until all of the nodes go down, or some other nefarious combination of nodes going down, I am simulating it by deleting all pods using kubectl delete in succession on all of the cassandra nodes.</p>
  <p>When I do this the clients throw NoHostAvailableException
  in java its</p>
  <pre><code>    &quot;java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.200.23.151:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (1 required but only 0 alive)), /10.200.152.130:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency ONE (1 required but only 0 alive)))&quot;
      which eventually becomes
      &quot;java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (no host was tried)&quot;
  in go its
     &quot;gocql: no hosts available in the pool&quot;
  </code></pre>
  <p>I can query cassandra using cqlsh, the node seems fine using nodetool status, all of the new ips are there
  the image I am using doesnt have netstat so I have not yet confirmed its listening on the expected port.</p>
  <p>Via executing bash on the two client pods I can see the dns makes sense using nslookup, but...
  netstat does not show any established connections to cassandra (they are present before I take the nodes down)</p>
  <p>If I restart my clients everything works fine.</p>
  <p>I have googled a lot (I mean a lot), most of what I have found is related to never having a working connection, the most relevant things seem very old (like 2014, 2016).</p>
  <p>So a node going down is very basic and I would expect everything to work, the cassandra cluster manages itself, it discovers new nodes as they come online, it balances the load, etc. etc.</p>
  <p>If I take my all of my cassandra nodes down slowly, one at a time, everything works fine (I have not confirmed that the load is distributed appropriately and to the correct node, but at least it works)</p>
  <p>So, is there a point where this behaviour is expected? ie I have taken everything down, nothing was up and running before the last from the first cluster was taken down.. is this behaviour expected?</p>
  <p>To me it seems like it should be an easy issue to resolve, not sure whats missing / incorrect, I am surprised that both clients show the same symptoms, makes me think something is not happening with our statefulset and service</p>