How to troubleshoot heat that's always on?
If you had a DMM or VOM and know how to use it, that would make diagnosis easier.

However, the approach to take is to divide and conquer.  Identify what component(s) are causing the signal for heat.  Is it the boiler controller, the thermostat, or a defect in the wiring?

I would begin by inspecting the areas where anything has changed since it last worked okay.  Look for anything amiss:  stray wires, debris, bird nests, junk.

Next, I would disconnect the thermostat from the wiring to the boiler.  This should be just a couple screws to loosen and then pull the wires out from under the terminal.  If they are not naturally stable, wrap each individually with tape&mdash;masking, electrical, scotch, duct:  whatever you have handy.  If the system continues to heat, the defect is probably1 not the thermostat.

At the end where the wires leading to the thermostat connect to the boiler controller, disconnect them, being sure to record which goes where&mdash;use a camera, labeling the wires, or writing on paper.  If the system continues to call for heat, the fault must be in the boiler controller.  Otherwise the wire itself has a flaw.  If there are unused wires in the bundle, you could try swapping unused wires for the needed one.

If you get here, it is likely the new controller is either defective, or more likely, not correctly connected after its replacement.  Double, triple, quadruple check the connections and everything else which could go wrong.  Reread the installation instructions to see if maybe there is a configuration option which might relate to the problem.

1 There could be several problems at once, so repeating tests of all items might be needed after the first round of repairs.