What exactly limits the signal frequency on transmission lines?
It is mainly the skin effect that limits high frequency signal transmission in transmission lines. The skin effect leads to high damping (R') with increasing frequency.