Is LNK304 not suitable to drive a 12V relay?
The LNK304 should do the required task if it is operated correctly.
WARNING - Note, as jippie says, the device is NOT isolated from mains. The output and all connected parts are potentially at mains potential. If things go wrong mains could be connected to your controller - or to you.

You need to provide more information.

Presumably 230 VAC feed to LNK304?

Please show the whole LNK304 circuit that you are using.
If you operate it into a 30 mA load at 12V (~= 400 Ohms) does it provide 12V to the load reliably for long periods?

Below is a typical LNK304 circuit diagram.
How does uyours compare?
 Which of these parts are you using or not using?



Circuit from datasheet.



ADDED:

You say relay coil requires 30 mA. So Rrelay ~= V/I = 12/0.03 = 400 Ohm.
If you uses a 400 Ohm resistor instead of a relay, does the system survive properly? 

Does the system need a minimum load - some systems do - datasheet will tell you.
You MAY need to place a resistor across output at all times.
But, maybe not. 

The diode  shown across relay coil MUST be present and should be mounted as close to coil terminals as possible.
Using a fast diode is advised.



Using a 1N4005 for D2 is OK. BUT - is the rest of the circuit identical to what I have shown - if not, what else is different. eg what did you use for L1 and L2? What are their current ratings at the operating frequency?