Transfer Function of Non-Ideal Transformer
I have never seen any formula as the non-linear behavior as a function of freq. depends on stray and winding capacitance, ferrite non-linear properties and many other factors.

For 60Hz here's a simple model http://claymore.engineer.gvsu.edu/~johnsodw/egr325mine/paper2/paper2.html

Which are very different to RF tansformers
http://www.minicircuits.com/app/AN20-002.pdf

which are very different to pulse SMPS transformers !

which are very different to Flyback transformers !
http://www.wisegeek.com/what-is-a-flyback-transformer.htm