How bad can an unconnected ground be?
You should connect ground to be used as a reference point. Even in RS485/RS232, people should connect ground. The reason is that if both devices are not grounded, then common mode voltage  (Vcm) can damage the port. For example, RS485 receivers runs on relatively small differential signal (+/-200mV). However, Vcm can exists...due to noise, bad local grounds, etc. If one device is correctly grounded but the other is not, then instead of seeing +/-200mV, they might see +/- 20.2V. This is assuming the devices are not close to each other.