Is it always OK to directly connect grounds of different voltage sources?
It's OK if the two supplies are isolated from each other, ie. have no other common connection. 

Most AC/DC adapters are 'double-insulated' and not connected to mains Ground. Provided that you only use such an adapter to power the servo it will should be OK. If your adapter has a ground pin connected to the mains socket then its output might be grounded. This could be a problem in a circuit that is sensitive to small differences in ground potential. 

Most servos can handle a fair bit of noise on the power lines, so unless you have some high current equipment operating on the same circuit it shouldn't be affected by common ground noise.    

Putting a small value resistor between grounds may help to reduce 'ground loop' current flowing through a sensitive device. The servo should still work with 200&ohm; or more of resistance in the signal path.