What could probably go wrong in PCB if output PAM4 signal my look like this?
The SMA connector grounding looks like a problem to me. There are traces going to vias, which I assume connect to a ground plane. Instead, I think there should be something like 20 vias in each pad going to the ground plane. The via-in-pad process adds cost to the PC board, but I think in this case it is required. If you add some inductance to the connector ground pin in a simulation model, you may see the problem.