RSET pin function of RTL8211E
Do you see that "not for public release" stamp on the datasheet? Also the note that says "for development partners only"? It means that you probably have to sign an NDA in order to get the datasheet from realtek and have an agreement with them about what you're building, etc. Things like gigabit transceivers are sensitive and valuable IP and so companies like to control who accesses it.

Anyway, I can hazard a guess at what RSET does based on a similar looking situation with the (public) TI TUSB7340 USB 3.0 Host Controller: It requires a precision resistor of a particular value which is used to calibrate the precise currents required for high speed differential signaling terminations inside the chip. You will also find similar resistors on Xilinx FPGAs which have the 7-series GPT Hard IP (look for mentions of "RCAL") and also on PCI Express switches. You need to find out exactly what value is required before you can even hope to get the chip to work properly.