Select power source with reverse current
You shouldn’t need an ideal diode on the unit-to-unit link, just one on each of the USB inputs and have the +5V bus shared between units. There might be a concern with connecting the grounds to each other though.