Why would plugging in a logic probe change values in a digital circuit?
Put a scope on the power supply.  It is probably oscillating.  This could be due to a faulty component, a bad solder joint (output cap not properly connected, for example), or a marginal design that happens to work in the other units.  Put a 1 &micro;F ceramic cap accross the power supply and see if you get the same effect.  If so, put such a cap directly accross the power supply output as close to the regulator chip as you can.

When you say the power supply is "wonky", is it just that unit or all these units.  If all of them, then it sounds like a bad design.  If just this unit, then it is even more evidence that something is wrong around the power supply of that unit.  Inspect the parts and solder joints of the power supply of the wonky unit carefully.