USB device with separate switch-mode supply. Do I just connect the grounds? Are ground loops an issue?
Look at page 23 (figure 7.2) of the datasheet. This shows the wiring for a "self-powered" device. Indeed, the chip is powered from the separate power supply (not USB +5V). 

The grounds must be common, though. You may want to consider using an isolated PS for your device to avoid ground loops. That would be the easiest way to avoid potential problems.

Also note the wiring from USB +5V to VBUS_sense.