Can I use two Power Source to Switch with a Relay like this?
My question is, Connecting two ground wires from DC adapter and
  Battery Ground will cause any trouble?


No, this will not cause trouble providing the battery in question is a just a battery and not some other AC / DC power supply with maybe a dodgy earth. Even then it won't burn but might throw up a few operational problems but... unlikely.

Regards Ignacio's comment, it is advisable to have a diode if the DC from the AC supply is a bit bigger than the battery voltage - this would nearly always work without going to the complication of a relay but if the voltages are somewhat similar then it's hard to say whether the battery will remain powering the target circuit when AC is on.

Another thing - when the relay changes over, there is a small but finite time when you lose power so you should have enough capacitance across the load to prevent drop-out.