Using Galera for simple master-master replication I was reading the documentation and it specifically mentions that a Galera DB cluster must have at least three nodes to achieve quorum, although our current setup is working with two.

I am tasked with rebuilding the cluster.  My use case for Galera is a simple master-master cluster.  It is in actually a master-slave, with the secondary master sitting there in case the first server fails (then we can point the PHP configs at it).  Galera was chosen due to the fact that it is synchronous.

My questions are:


Is a two node configuration safe/supported?
In a 3 node setup, if the third node dies it now becomes a two node setup.  Is this now unsafe meaning I should really have a 4 node setup?
What exactly is a Galera "arbiter"?
1) Is a two node configuration safe\supported?


Galera will still run in a two node setup. However, there is always the threat of a split-brain scenario. For example, suppose you have DB1 and DB2 form a two node Galera Cluster. If DB1 goes down, you need to failover to DB2. While DB1 is having maintenance done to bring it back up, DB2 gets all your changes. By the time DB1 is back up, doing an IST is virtually impossible (or at least too late at this point). You would have to perform a full SST from DB2. Easiest way to do this is to delete the Galera cache files before starting up DB1. During the SST, DB2 is in a read-only state (as a donor) and cannot process any inserts, updates, or deletes. When there is a third node in the Cluster, at least one server can collect inserts, updates, and deletes.

You will also find 
Percona XtraDB Cluster: Failure Scenarios with only 2 nodes more informative.


  2) In a 3 node setup, if the third node dies it now becomes a two node setup. Is this now unsafe meaning I should really have a 4 node setup?


A two node Cluster is still operational. When introducing the third node for the first time or bringing back the third node from being down, one of the two operating nodes enters a read-only state (as a donor) to help the third node play catchup. Only one

As I said in the first question: When there is a third node in the Cluster, at least one server can collect inserts, updates, and deletes.


  3) What exactly is a Galera "arbiter"?


I think you mean Arbitrator (See the subheading Using an arbitrator). It is a mechanism that helps decide who plays the role of donor and who play the role of the cluster when introducing node back into a Cluster. It helps with mitigating split-brain scenarios More can be found in the Galera Documentation.