Is there a method for proving non-regularity of string transformations?
Your question is not entirely well defined: how are you given the transformation you start with? 
For example, if you assume the transformation is given by e.g. a Turing machine, then clearly there is no algorithmic way of deciding whether it's a regular transduction.

However, it seems that what you're asking is whether there is some "machine-independent" characterization of string transductions (e.g., Myhill-Nerode).

While I don't know of such a characterization in general (I'm pretty sure no such characterization is known), there is such a characterization for string transducers with origin information, developed by Bojnaczyk. 

You can start here.