Why does the GND of the Arduino oscillating as well?
I don't think your ground rail is oscillating. You may be able to test whether it's oscillating relative to the voltage rail by connecting the ground clamp to ground and the probe to the 5v or 3v rail for your Arduino.
Voltage isn't like current: you can't measure it at a point, it's almost always the difference between two points in the circuit. When you remove the ground what exactly you're measuring becomes somewhat ambiguous. What you're most likely to see is the distorted AC component of the signal at the probe mixed with a weak 60Hz (or 50Hz in some places) distorted sine wave from the mains power in the building and any other AC signals near the probe.