Can I measure non-linearity of a line echo path?
excellent, you have PAM-4!
Send a long sequence of your least-amplitude symbol, and integrate the received power to calculate the energy of your received signal.
Do the same with the three other symbols.
Now, you can solve the equation
$$E_{RX} = \alpha E_{TX} + E_N + F(P_{TX}),$$
with $\alpha$ being the channel gain, which has to be the same for all 4 of your measurements, $E_{RX},E_{TX}$ being the transmitted and received energies (hint: normalize them to the least-power symbol), $E_N$ is noise energy, i.e. noise power times number of samples (which becomes asymptotically constant if you just average enough symbols), and $F$ is the nonlinear error term.
With $E$ known, and $\alpha$ and $E_N$ overdetermined determined from four measurements, $F$ is the reason for the error.