CMOS Gate and Coupling Noise from Loose Wires
Your horrible noise could be grossly underdamped LC ringing .You cant up the C because you have to keep fast rise time .If you place Resistance in your circuit you should be able to damp out the oscillations .More resistance lowers the Q so why not start with 100 ohm.Keep increasing to see your waveform improve.Twisting all the loose wires will help too .The reason that the coax worked well is because the cable has very very low inductance .In fact the effective output resistance of the logic chip was probably enough to damp the wave. The now old HCMOS had an output resistance that would ballpark into 50 ohm quite well . Newer logic with lower on resistance output mosfets which can drive more current would be happy into cable impedences that nobody makes.