What is wrong with this XOR gate built with four transistors?
One problem is that the upper of the two inputs is being used sometimes to drive the output, so that load on the output will appear as a load on the input.  You could fix that problem by using a second inverter cascaded after the first, to produce a copy of the input without loading it.

More seriously, the output in the top two pictures is being driven low through a p-type transistor or high through an n-type transistor.  In these circumstances, the gate-to-channel voltage of the transistor will not be high enough to turn the transistor fully on, and the drive to the output will be weak, and (depending on the load) might not be sufficient to drive the output near enough to ground or Vdd for any following logic to work properly.