Are there any special internal Ethernet connectors other than RJ45?
You need to use a different ethernet standard than 802.3ab, that doesn't need RJ45 jacks. Like IEEE 802.3ap “Backplane Ethernet”, which supports gigabit speeds over a backplane type of connection. This standard was designed for your application in mind.

Also see On-Board connection of ethernet transceivers for 10/100 speed connections on the same board. The Micrel now Microchip app note AN-120 http://ww1.microchip.com/downloads/en/AppNotes/Capacitive%20Coupling%20Ethernet%20Transceivers%20without%20Using%20Transformers.pdf describes how non-transformer (and connector) ethernet connections are designed on board.