PCIe gen3 over a FMC connector
FMC uses Samtec SEARAY connectors which are rated for over 28Gbps, more than sufficient for PCIe Gen 3.

In fact many vendors are producing FPGA modules or carriers that run Gen 3 PCIe links through FMC connectors quite successfully.

The main thing to be sure of if designing your own board is to take care of how you route your differential pairs. Samtec provide final inch routing information to assist in that regard.