LT3663-5 with 5V and 3.3V output, can RTN be connected to common ground on circuit board?
Yes, it's safe.  The 3.3V output is isolated, so you can connect its ground wherever you want.  However, it's not clear why you would want an isolated topology if you are just going to connect the grounds together.  Depending on the current required you could derive the 3.3V from the 5V using an LDO or a small switcher and avoid the need for a transformer.