ADM2483 or MAX485 Design: Modbus RS485 RTU bus in harsh environment
So as one device (PLC) has reference to mains earth, and the other device is floating (PoE by definition is isolated), they have no common ground reference, so simply two data wires of RS-485 will not work and can even break RS-485 tranceivers with large potential differences between the devices.

Standard RS-485 tranceivers such as the MAX485 allow up to 7V difference between their grounds, so this will be exceeded if the PoE device is floating. The isolated tranceiver works because it allows the isolated side ground reference to float and be set via the data wires, and it will allow higher difference between isolated ground and PC ground. Although that is not even necessary, because the PC is also most likely earthed, so there will be a common ground reference between PLC and PC, via mains earthing.

Basically, since the ESP32 is powered via PoE, you need the reference ground wire between ESP32 and PLC.