protect RS485 from PoE
The

How can I avoid this?

Here is how I've dealt with this in the past.

Color code the connectors or label the connectors (if possible)

Use software to not turn on the POE unless the configuration is right.

Different sized connectors (which you cant do, change the RJ45 to a RJ11)


Another option is somehow do some impedance detection and don't turn on the POE unless the right cable is detected. It shouldn't be hard to tell the difference between RS485 and Ethernet, one is a transformer and the other is a transceiver. I don't have your schematic so I can't recommend ways to do this directly.
Last option: If you can change the pinout of the RS485 (not sure what you're running but if your only running (Vcc D+ D- and GND) then you could rearrange the pins so they do not use the POE pins.