Light never turns off
Just a hunch, but worth investigating.
I suspect you have this light fixture somehow linked into two circuits. That should never happen, but if it does, you can end up with a lot of strange things.
Find the Breakers
You need to figure out which breaker controls the switch and which breaker controls the light separate from the switch. Doing that may get a bit tricky, as I would not connect the switch back to the light again until this is straightened out.
I would get a non-contact tester and/or a multimeter to test the switch wires. You should see one of the wires hot (lit up with the non-contact tester) and 120V (assuming US for the moment) between the wires. When you turn off the relevant breaker you should no longer see a hot wire and the voltage between the wires will either be zero or some relatively low phantom voltage.
When you have the switch breaker off, does the light go out? If it does, then the next step is to check each other device (light, switch, receptacle, built-in appliance) on the same circuit (i.e., everything that goes off when you turn off that breaker), looking for loose wires, things wired incorrectly, etc.
If the light stays on when the switch breaker is off then you have to keep flipping breakers until you find the one that turns off the light. Then you know that you have a cross-connection between the two circuits which is NOT a good thing. That could take take several forms, and will likely require checking out the wiring on both circuits. But I would actually start in that case with the wires going into the problem light fixture. One possibility is that wires for the 2nd circuit pass through the ceiling junction box and that when the light fixture was replaced, the wires got messed up. But there are other possibilities as well.