Circuit for choosing one power supply from several of them
Your idea works in principal, but has some drawbacks.

With a 1.8 V supply, your PMOS load switch will only see a Vgs on of -1.8V, that is fairly low. You likely won't achieve the FET's rated RDSon. Secondly, running a FET below its ZTC point can result in hot-spotting on die due to the negative tempCo of the threshold voltage.

Personally, I would move to a NMOS design where you use a charge pump to a achieve a gate drive signal above the input supply rail. You can roll your own design, or buy a load switch driver IC.

As you likely know, you will want to design some enable logic to ensure you break before make any of your load switches to avoid back-feeding onto the lower supply rail.