How is a signal physically routed in an FPGA?
Your interpretation is overly simplistic. Real FPGAs have a complex hierarchy of routing resources, some for local connections only, some for medium-range connections and some for spanning the entire chip. These structures have been developed over many years of studying application designs, trying to strike a balance between the area required for routing versus being able to support a wide range of application logic structures.

In Xilinx chips, for example, these resources are in the metal layers, and they connect to small blocks of active logic that function as small crossbar switches, allowing fairly arbitrary connections between the ports of each switch.

If you fire up Xilinx's floorplanning tool on your design, and zoom way in to the level of individual slices, you can start to get a feel for the actual physical layout of the chip and its routing resources. It's quite surprising how small the area is that's devoted to the actual slice logic that implements your applicaion. The vast majority of the chip area is devoted to the routing resources.