calculate SNR of two signals with power spectral density
Since you only say your channel is nonlinear, but not exactly which math it follows, the only answer possible to give here is

1.-5.: By filling in the signals and noise in your device equation, and transforming the math until you get an expression for what you're looking for.

"nonlinear" doesn't really say anything about what's going on (aside from the transfer function not being linear; especially, questions like determination, time-variance, boundedness, memory... are left unanswered).

If you think about it, it can be shown that with the lack specification given, the statement above is the strictest statement possible.

Even incorporating the fact that you claim the power spectral density exists doesn't do us much good – it only tells us the output of the device is weak-sense stationary, and that might as well mean the thing is just generating random tones.