Complete Failover Cluster Failure SQL Server 2012 We have experienced a full cluster failure on our 3 node cluster running SQL Server 2008 R2 with 7 SQL Server instances. It started when we rebooted one of the nodes and it returned an error:


  Node '%1' failed to establish a communication session while joining the cluster. The was due to an authorization failure. Please verify that the nodes are running compatible versions of the cluster service software.


The other two nodes were still online. Running a cluster validation test passed. After consulting with Microsoft we decided to take down all 3 nodes, bring one node online with force quorum option (so the cluster would come up on a single node), then bring the other nodes online one at a time.  Unfortunately, after doing this, we were left with a single node forced online, because neither of the other nodes would join. The same error above was being reported on both the other nodes.

No resolution has been found for this after 2 weeks of investigation so we were presented with two options:


Rebuild the cluster
Evict the nodes and try to get them rejoined


My question is, this cluster has 7 SQL Server instances on it, and one File Server service. If the cluster is completely rebuilt, or a node is evicted, what are the steps to get SQL Server back working? Do we have to completely reinstall SQL Server or dDo we just have to add the instances back into the new cluster using Failover Cluster Manager without lengthy software installs?
If you evict nodes you'll end up formatting the nodes and rebuilding them.  Once a node has been evicted from the cluster you won't be able to uninstall the SQL Server instance from the node.  You can try manually removing all the registry keys and files, but honestly it's probably faster to reinstall Windows and try getting it back into the cluster.

Has Microsoft not been able to get you towards any resolution on this?

Either way you're looking at a cluster rebuild basically from scratch here with 21 SQL installs, which I'm sure as you know takes forever.  There are some ways to make this take less take (building a new two node cluster, installing everything there, then doing a controlled migration between clusters, use DNS to redirect connections to the new names so applications don't need to change, etc.

If this was my environment I'd burn it down and bring it back fresh, obviously with as little downtime as possible.

--sales pitch--
It might be a good idea to bring someone in that has a lot of clustering experience (me, someone else, either way) to help with the rebuild to make sure that everything is setup exactly as it should be and to help take the pressure off your internal team (who probably hasn't been getting a lot of sleep the last two weeks).  If you've got a lot of clustering experience in house then this isn't needed of course.  I'm just worried because something went horribly wrong doing something which souldn't have been that big of a deal to do.
--end sales pitch--

Disclaimer: I'm a consultant.