> the storage between the 2.
Do you want the storage to be redundant between the two like RAID 1 or just all of it accessible to any guest on either machine like a JBOD? For either, you can use a suitable mix of drbd, iSCSI, GNBD, LVM, heartbeat, keepalived, and a clustered filesystem. If you do the latter, remember that this squares your risk. If you want a simple A/B failover setup where the second physical host is idle until the first physical host fails, you can just use drbd and heartbeat or RHCS.
> They all use xfs.
> I want to keep the guests the same on each server.
> this FS and block device.
> re-attach it back to each guest.
> server and then switch them and do the same to the mainserver.
That sounds about right. You need to make your backups (!), destroy the data on of your servers by creating the new clustered filesystem, copy the data from the other server, and then share it with your reconfigured guests. So, basically what you said.
> I have one more question which is another whole thread.
drbd, but rsync does a pretty good job in a lot of use cases. I wouldn't say that it fits this one, though.