Transformer powering an alarm system heating up
I was able to figure out why the transformer was heating. I was mislead by the yellow color of one of the output wires from the transformer, thinking it was ground. I connected it to the ground pin on the board and to the ground wire from the metal casing. Even with all consumers disconnected, the transformer was still heating and this is what made me thinking that maybe there was a problem with the power wires. 

As soon as I disconnected the yellow output from the ground and connected it only to the second AC pin from board everything was OK and the transformer didn't heat over night.