Is there an analogue of coherence between data sets related by a non-linear transformation?
Are you interested in coherence's ability to discriminate correlations at different frequencies, or are you interested in any measure of non-linear correlation? Coherence is basically cross-correlation in the frequency domain.

Mutual information can measure non-linear correlations, but says nothing about the form of the correlation. It's also difficult to estimate accurately without large amounts of data (kernel density estimation is a popular tool to help with that right now, though I think cupolas have begun to gain traction).

I don't know that anybody has done so, but you could alternatively use one signal to predict another using an artificial neural network with one hidden layer. Your correlation metric could then be something like an RMSE between predicted and target signals. It feels like a hack, though. With some work you could try to infer something about the relationship type based on the trained weights of the network. I'm not an expert, but I don't know of a formal framework to actually do that. It's also not an established metric, so you have to also do the work to convince people that it's meaningful (much more difficult if you can't cite sources that did it before you).