What is the best way of implementing ethernet over a custom pogo pin connector?
should I be weary of in terms of noise, routing, shielding etc

Of course.
So best thing is, after you done the theoretical calculations test your design with an FEM field simulation...
Edit:
let me state it this way: We did the development of an Ethernet capable connector in my company, Took us many month of work, simulating, measurement, etc... You need profound knowledge of the standards, top equipment like a NWA, FEM based simulations, etc... This is far beyond such a simple question... In fact the question alone shows the lack of qualification for that task. Its not I want to be mean, but realistic

Edit2:
Just to provide some more usefull information, HF-Connectors are very much defined by geometrical layout, so you need first to draw what layout in 3 dimensions the connector should have. then placement, shape, order of signal contacts is essential. From that its a lot of fine tuning if the general layout is OK. As the HF wave travels not though the conductor but the isolation material, the permittivity of material mix must be considered. step changes in permittivity or geometrical dimensions results in reflections. Wrong placement,order,shape of contacts will result in poor NEXT and or TCL performance....