Maximum frequency FPGA
Xilinx's, for example, have very fast giga-bit transceivers that can operate at multi-gigabit/sec rates, but they also have simple serial and parallel shift registers out at the IO edges of their dies that are typically used for the serial to parallel shifting in both directions.  In addition, to get the fastest speeds with synchronous external clocks, I typically use half-rate clocks and DDR (double datarate serializer or de-serializer settings).  Its pretty straightfoward in a series-7 Xilinx to get 1Gb/sec on each LVDS I/O pin when applying a 500Mhz external clock.  This is all done without using the gigabit transceivers.  This also involves no PLLs so it can operate smoothly from low speeds to high speeds continuously.

If you have 640MHz clock today, you may be after 640Mbps on your data lines.  My recommendation would be to use a 320Mhz clock and a 1:4 or 4:1 internal serial-to-parallel or parallel-to-serial converter in DDR mode (See Xilinx OSERDESE2 and ISERDESE2 built-in components).  This would leave your parallel clocking rates in your main logic down at 160Mhz which is OK.

There are good application notes on setting this up.  You can also use the built-in IP features of their development tools to create this component for you.

Hope that helps.