What determines the output voltage of this SMPS?
The zener DOES control the output voltage - at about 19V across the output terminals, the zener begins to conduct and starts to turn on the the opto-isolator. This in turn signals the TNY267 that the correct level of output voltage has been met and it's time to start backing off dumping too much energy into the transformer.

If loading increases and the output voltage starts to drop the zener reduces its conduction and starts to turn off the opto isolator.

The transil "snubs" out back-emfs from the transformer - it needs to be rated in volts below the level at which the MOSFET in the TNY267 might suffer damage. Given that there could be a DC voltage of 240 x 1.414 volts (339 volts) the MOSFET will be protected to no more than 339V + 180V = 519 volts. Note also that the BA159 prevents it conducting in normal diode mode hence the voltage rating of 180V is the reverse breakdown voltage.

I suspect the BA159 needs to be fast recovery type and the 1N400x series is certainly not that! BA159 is specified as having 500 ns reverse recovery time whereas the 1N4007 is about 30 us. When a diode goes from forward conducting to reverse blocking there is this time that even though the diode is reverse biased it acts as a forward conducting device. Clearly if you are switching at (say) 100 kHz, 500 ns represents 20% of the time that it could be reverse conducting. 30us would be a disaster!