I have run into a horrible situation on one of my prod clusters. The version the cluster is on is 3.0.5. I am running a 2 DC setup with close 30 nodes, 18 in one DC and the rest in the other. I did everything possible with my knowledge, but still looking for answers.

Of late we were having a few issues with respect to GC pauses, a few turnings were done on the jvm(MAX\_HEAP\_SIZE was changed) on all nodes and the cluster was ready for the rolling restart to take effect.

The 1st node went through well with the rolling restart, but the 2nd node just did not comeback up after the shut down. And the error below.

```
INFO  07:45:34 Initializing system_schema.keyspaces
INFO  07:45:34 Initializing system_schema.tables
INFO  07:45:34 Initializing system_schema.columns
INFO  07:45:34 Initializing system_schema.triggers
INFO  07:45:34 Initializing system_schema.dropped_columns
INFO  07:45:34 Initializing system_schema.views
INFO  07:45:34 Initializing system_schema.types
INFO  07:45:34 Initializing system_schema.functions
INFO  07:45:34 Initializing system_schema.aggregates
INFO  07:45:34 Initializing system_schema.indexes
Exception (java.lang.IllegalStateException) encountered during startup: One row required, 2 found
java.lang.IllegalStateException: One row required, 2 found
    at org.apache.cassandra.cql3.UntypedResultSet$FromResultSet.one(UntypedResultSet.java:84)
    at org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:948)
    at org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:938)
    at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:901)
    at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:878)
    at org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:866)
    at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:134)
    at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:124)
    at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:229)
    at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:551)
    at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:679)

```

after running repairs on the cluster, and specifically on the system keyspaces, the error still persisted. When the node did not comeup eventually, i had it removed from the cluster, using the nodetool removenode command from a healthy node.

Again, another node was taken up for restart in the same cluster and datacenter, again it did not come back up, with the same error.

I was also unable to login to the cqlsh shell from a healthy node, with the below error

```
Connection error: ('Unable to connect to any servers', {'<<VM hostname>>': UnicodeDecodeError('utf8', '\x7f\x00\x00\x80C\x02', 3, 4, 'invalid start byte')})

```

This error also was seen on a few other nodes

```
Connection error: ('Unable to connect to any servers', {'<<VM Hostname>>': ConnectionShutdown("'utf8' codec can't decode byte 0x80 in position 3: invalid start byte",)})

```

Essentially, the cluster has nothing working except the nodetool commands.

When i ran a nodetool describe cluster, i saw 5 different schema versions for various nodes and also saw some 9 nodes as unreachable, below is the output

```
./nodetool describecluster
Cluster Information:
    Name: Dummy cluster
    Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
    Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
    Schema versions:
        1590ea6a-8c19-342a-8269-204c64a12176: [9 nodes here]
        668d9efd-13c1-3fb3-9b89-7fc07d9ddf0b: [1 node here]
        d20dc0de-dd34-3183-b459-31e3feb8f118: [3 nodes here]
        3ec9610c-d241-3215-84f2-2413b8cad7d2: [7 nodes here]
        59adb24e-f3cd-3e02-97f0-5b395827453f: [1 node here]
        UNREACHABLE: [9 nodes unreachable]

```

Can someone pls help in understanding what the issue could be and also a way to bring the nodes back up? I also tried the ignore schema mismatch flag in cassandra-env.sh/jvm.options to bring the node up, but that did not help as well.