Is there any way a usb device can identify the usb host operating system?
The USB specification doesn't provide any way to do this. You should also consider the USB host may not be running an operating system at all if it's an embedded device.

You could design your own protocol on top of USB which allowed you to find out if you needed to.