Sharpening rise and fall time of square wave
Now that you've posted images of the input as well as the output, I'd say you've got a couple of problems that add up to cause what you are seeing on the output.


The output isn't quite capable of driving the 75ohm cable at the output.  The curved edges (both rising and falling) look to me like they've been through a low pass filter - say, the (resistive) impedance of the driver into the capacitance of the cable.  Your capacitor in place of R9 contributes to this as well.
Your source isn't capable of driving the 75 ohm cable, either.  Additionally, the rising edge of the signal seems to get slower as the frequency rises.  


The SN74LVCT145 tries to sharpen the edges of the input signal, but has its own problems driving the output load.



I don't know how you'd properly solve this.

I'd start by removing the capacitors you have in place of R1 and R9.  Just nothing in those positions.

You'll need a line driver to handle the output side.  Something like the SN74LVC3G34  

That would also handle the level translation.  It can accept up to 5V on the input, even when powered by 1.8V. 5V logic level signal in, power on 1.8V, 1.8V logic level out.

Using the SN74LVC3G34 instead of the SN74LVCT145 is only possible since you say you only need to translate the levels in one direction (you have DIR set to a fixed value in your circuit.)