Why does the voltage regulator not regulate when ac is around 200V AC?
The transformer you've linked is rated for 9VAC at 230VAC input so you're already running under spec if you're trying to get 12V.  The transformer ratio is around 25:1 so 203V would give you around 8VAC.

The 12V variant of the L78S series of regulators has a minimum output voltage of 11.5V and a dropout voltage at 25&deg;C between 1.5-2.5V depending on current draw which means your minimum input voltage would have to be at least 13.5VDC.  This doesn't take into account whatever rectification/smoothing you're doing from the transformer.