Can you replace a RJ45 connector with smaller connector?
Yes, there are already many ways this is done, laptop dongles are one way. HDMI has Ethernet routed in it. 

Whatever cable\connector you use needs to be able to handle differential pairs with an impedance of 100Ω