Distortion due to transmission line effects of low pass filtering?
I've just realized that you are driving a twisted pair from a single ended driver - this would be incorrect. If you are going to do this use coax. The big problem is that twisted pair only sensibly works when it is differentially driven. It appears you are not doing that. You may well compound the problem by having a single ended (not differential) receiver too.