Old radiant heat ceiling not cooperating, looking for ways to fix it myself
Because they DO heat, I think that the problem is not in the heaters themselves, but in their thermostat or thermostat's probe.

What you need to do is to disconnect the heater element itself (the one on the ceiling) and connect a lightbulb (a real, incandescent bulb) in it's place. This way you can begin analyzing on when the panels are on and what to do to make them off, starting with what breaker they're on. 

Your second step should be to map the entire system. Trace the wires, eg by switching off all breakers except the one controlling heater and tracing them with contactless voltage detector. The system must have 2 elements at least (heater and thermostat controlling it), but it can be much more sophisticated; eg thermostat can have remote probe or remote knob and the heating element can be controlled indirectly via a contactor. As you've said you've disconnected the thermostat, a stuck contactor is very likely cause.  Replacing it is quite straightforward operation, IF you can locate and access it.

Another but radical solution is to ditch the old control system altogether and keep only the heaters themselves (of course AFTER confirming they work on mains voltage directly and old control system isn't doing some tricky power conversion). Then you get a brand new thermostat and a contactor(s) and set them up using new wiring at best. Probably best way do to new wiring is to keep the contactor in or near the breaker box and run two separate sets of wiring - one set (live and neutral for single phase / L1-N-L2 in case of 2phase / neutral + 3 live in case of 3phase) of power wires to the heaters controlled by the contactor and another set of control wires (live and neutral to power the thermostat + one live control wire back from the thermostat to drive the contactor - "control" but still operating on mains) to the thermostat. Or some fancy digital wireless system that would save you the wiring for the thermostat.