What is the ground port on an analog oscilloscope for?
Most oscilloscopes have a safety ground. This connects to the earth ground on the plug and also to the case. The thinking is if you wire it up wrong worst case you trip the GFI/RCD instead of shocking yourself or damaging the instrument.

As an oscilloscope is usually a bench instrument it is not isolated from the mains. You can use an isolation transformer, but this is not recommended, as it removes this protection leaving the earth ground floating.

Tektronix sells an oscilloscope which has fully isolated inputs, I don't know of any other models. It is significantly more expensive to add high speed isolation circuitry. Usually it means the whole analog and acquisition side must be isolated.

The ground should always be connected to your circuit's ground or you will get strange behaviour. Trust me. I've spent hours looking at a trace wondering where the noise was coming from, just to find I haven't properly connected the ground.