Select capacitor to stop voltage drop
The load regulator error is simply a resistance ratio.
This also applies to the transient brownout.
The TLV75533PDBVR in the DBV package has Load regulation spec = 0.060 V/A which means it has an output impedance of 60 mohms  The 1 uF ceramic cap could be 10 mohm more or less.
Analysis
The output time constant of the source caps would be 60 mohm*1uf = 60 us which isn’t 63% of the voltage swing (58%) but close enough.   This means your load has a similar or slightly better ESR and a slightly smaller capacitance.
So your load if it were the same ESR, the voltage dips  50%. if your source is 10x lower ESR, or 6 mohms , you might have a 5A supply or 10 caps in parallel or add a series resistance 10x bigger to the switch if you don’t need such load regulation as the 10 mOhm 1 uF cap.
recommendations

It depends on your voltage error tolerance, but to reduce 50% to 10% you need 5x 1uF caps in parallel.  (?) Will a bigger 5uF cap work or a 10uF cap.? ...
only if the datasheet says it has a proportionally lower ESR. But  unlikely with the same size, so then it has to be 5x the physical size at least and spec the lower ESR
a better solution is to use an LDO in the auxiliary PCB from 5V and send an enable signal.