Can anybody explain the heat produced in the circuit during operating and non-operating state
If would be good if you added a schematic so we know what you are talking about.

Generally though the transformer will create a little heat when under no load due to losses in the transformer, but not so much heat that you will be able to feel much. Similarly, the rectifier has a little leakage current that makes some minimal heat.

If it is still getting hot, then either the transformer is not sized right or there is something else going on.

When the thing is driving the load of course there will be more current passing through the transformer and regulator and heat will be produced due to the losses in the transformer and the voltage drops across the diodes in the rectifier. With a large load, this can produce significant heating.

UPDATE: Thanks for the schematic. Despite the fact the diode is in backwards, this answer remains basically unchanged. The voltage regulator will, like the rectifier, consume minimal power ( ~24mW ) because of the eight or so milliamps it needs to function when the relay is open.