title: Cassandra keyspace does not propagate to newly added node (after previous successful additions and removals)
body: <p>I was testing rotating through a 4 node cluster, adding and removing nodes in a cyclic manner so the members of the cluster adhered to the following repeating sequence</p>
  
  <pre><code>1 2 3
    2 3
    2 3 4
      3 4
  1   3 4
  1     4
  1 2   4
  1 2
  1 2 3
    2 3
    2 3 4
      3 4
  1   3 4
  1     4
  ...
  </code></pre>
  
  <p>Node addition was performed by stopping cassandra, wiping <code>/var/lib/cassandra/*</code>, and restarting cassandra (with the same <code>cassandra.yaml</code> file, which listed nodes 1 and 2 as seeds). Node removal was performed by stopping cassandra and then issueing <code>nodetool removenode $nodeId</code> from another node. In all cases, the next operation was not started until the previous one was completed.</p>
  
  <p>The above sequence of node members repeated several times until after about 4 iterations I was performing an "add node" operation to transtion from a cluster of nodes {1, 2} to a cluster of nodes {1, 2, 3}. On this iteration, my custom keyspace failed to propagate to node 3. Nodetool status looked fine:</p>
  
  <pre><code>$ nodetool status
  Datacenter: datacenter1
  =======================
  Status=Up/Down
  |/ State=Normal/Leaving/Joining/Moving
  --  Address         Load       Tokens  Owns (effective)  Host ID                               Rack
  UN  192.168.12.206  164.88 KB  256     66.2%             7018ef8a-af08-40e9-b3d3-065f4ba6eb0d  rack1
  UN  192.168.12.207  60.85 KB   256     63.2%             ff18b636-6287-4c70-bf23-0a1a1814b864  rack1
  UN  192.168.12.205  217.19 KB  256     70.6%             2bc38fa8-42a1-457f-84d7-35b3b46e1daa  rack1
  </code></pre>
  
  <p>But <code>cqlsh</code> on node 3 didn't know about my keyspace. I tried to run <code>nodetool repair</code>, which seemed to loop infinitely, while spewing the following couple of stacks in the log:</p>
  
  <pre><code>WARN [Thread-9781] 2014-09-16 19:34:30,081 IncomingTcpConnection.java (line 83) UnknownColumnFamilyException reading from socket; closing
  org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find cfId=08768b1d-97a1-3528-8191-9acee7b08ef4
          at org.apache.cassandra.db.ColumnFamilySerializer.deserializeCfId(ColumnFamilySerializer.java:178)
          at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:103)
          at org.apache.cassandra.service.paxos.Commit$CommitSerializer.deserialize(Commit.java:145)
          at org.apache.cassandra.service.paxos.Commit$CommitSerializer.deserialize(Commit.java:134)
          at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99)
          at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:153)
          at org.apache.cassandra.net.IncomingTcpConnection.handleModernVersion(IncomingTcpConnection.java:130)
          at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:74)
  ERROR [Thread-9782] 2014-09-16 19:34:31,484 CassandraDaemon.java (line 199) Exception in thread Thread[Thread-9782,5,main]
  java.lang.NullPointerException
          at org.apache.cassandra.db.RangeSliceCommandSerializer.deserialize(RangeSliceCommand.java:247)
          at org.apache.cassandra.db.RangeSliceCommandSerializer.deserialize(RangeSliceCommand.java:156)
          at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99)
          at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:153)
          at org.apache.cassandra.net.IncomingTcpConnection.handleModernVersion(IncomingTcpConnection.java:130)
          at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:74)
  </code></pre>
  
  <p>Any ideas what's going on and how to fix this (ideally, a reliable working repair and a way to avoid entering this state in the first place)?</p>