IR Audio Receiver and Transmitter
Interesting design - literally the simplest thing that could possibly work.

The risk of damage to the audio devices from this setup is pretty minimal, as there's no external power supply involved. The only question is the current tolerance of the output of the cellphone headphone driver. Make sure all your exposed wires are insulated from one another (tape, heatshrink etc).

Adding an amplifier to the system is an idea in the right direction, although now you're at risk of destroying the LED driven by it - extremely likely with an audio amp of any power.

I would suggest building a standard transistor LED driver (e.g. Driving LEDs from audio signal ). Use a visible LED first to check it's working then swap it out for the IR one.

It's also worth applying volume normalisation to the WAV file you recorded and playing it back at maximum non-distorting volume.