how to wire a mono audio signal to a 3.5 TRS stereo jack
Try 'shorting' right and left TRS jack channels with resistors in line.  It would mostly get rid off impedance mismatch.  Some phones will work with your design but some will not like it.  Try using 1k to 10k resistors.  They shouldn't affect sound.

Here is a very simple and very informative photo of how to do it: