Selecting correct zener diode
The way you've drawn it the zener diode clamps the power supply reducing the 34V to 12V. What's even worse is that there's no current limiting and the diode will be destroyed.  

The zener diode needs a series resistor to take care of the voltage difference between the 34V and the 12V.  



To calculate the resistor value you need to know how much current the zener diode needs and how much current should go into the load (the opamp in your drawing). Zener diodes are often specified at the rather high current of 50mA, though more modern types will work properly at current less than 1mA. (This one only needs 50\$\mu\$A.) Let's suppose it needs 10mA and that the opamp will also consume 10mA, that's a total of 20mA. That's the current through the resistor. Since we already know the voltage across the resistor Ohm's Law will give us the value:  


  \$ R = \dfrac{\Delta V}{I} = \dfrac{34V - 12V}{20mA} = 1100\Omega \$  


If you check the power dissipated in the resistor (something you always should do) you'll see that this is 0.44W, so use a 0.5W power resistor.

edit
My original answer talked about 50mA zener current. That's very high, and I also realized that at 50mA a 12V/400mA zener would operate out of spec, as also jippie pointed out. It appears that the high current is only used for lower voltages (76mA for the 3.3V 1N4728A), and at 12V the datasheet talks about a safer value of 20mA. It's OK to use a lower current, though load and line regulation will be worse.