Does non-matching impedance of amplifier and speaker distort the sound?
You can simply forget about impedance matching for home Audio.

Impedance matching is needed only where the wavelength of the signal comes close to the length of the cable transporting that signal. Electrical signals travel with almost the speed of light through cables, for the highest audio frequency (giving the shortest wavelength) the wavelength is about 15 km. I'm guessing that your cables aren't that long.

Impedance matching is needed to prevent signals reflecting and distorting them. This is usually only relevant for high frequency signals, not audio (exception:  analog telephone lines).

In my opinion the "impedance matching" for audio amplifiers is really better understood as: "Can this amplifier drive this speaker?"

Example: some amplifiers are only suitable for 4 and 8 ohm speakers. Using it with 2 ohm speakers (or two 4 ohm speakers in parallel) can give issues.

For headphones this is almost never an issue unless the impedance of the headphone is very low (less than 10 ohms) or very high (600 ohms). And even then, if there is a "mismatch" the maximum volume might be reduced.

Usually home audio amplifiers drive the headphone output from the speaker output via series resistors to give a bit of protection against overloading the headphones as they need a lot less power than the speakers. Because of this almost any headphone can be driver from a home audio amplifier.

Mobile devices running on batteries cannot deliver so much power and voltage so overloading is less of an issue. Since the output voltage on these devices is limited I recommend using low impedance headphones, 30 or 50 ohms would be a good choice.

In either case, you do not have to worry about impedance matching, it is really a non-issue for headphones.

Sidenote:

For speakers the output impedance of the amplifier is relevant. The usual recommendation is that the amplifier needs a low output impedance. The lower the better as that will give it better "control" over the speaker. This is not impedance matching, it is actually a "best mismatch" situation as amplifier output impedance ( &lt; 0.1 ohms) and speaker impedance (> 4 ohms) are not the same.