Device's ground riding on a non-ground signal
It would be best if the probe tip connects to hot, and the grounding
  clip to neutral; but swap them, and the signal will be the same.


It would be best not to connect the ground of the oscilloscope to anything that isn't ground because it could fry your scope. Ground on an oscilloscope is connected to mains ground, with not that heavy gauge of wire (on the probe maybe 22-16AWG on mine). Neutral can still carry significant current in the event of a fault. 

Good Read: How not to destroy you oscilloscope

As far as the ground plane of the scope riding on a 50/60Hz sinewave, it is ground and mains ground does a pretty good job of being close to the potential of the earth.
Even across the outlet across the room is pretty much the same potential as the ground your scope is plugged into. 




  The question is, what happens at higher frequencies?
  
  Again, if my understanding is correct, at sufficiently high frequency,
  the scope would become an antenna and this could affect its internal
  workings (the signal present on the grounding clip could capacitively
  couple to other things inside).


Lets suppose a worst case scenario, lets suppose you have an isolation transformer so that you can connect the scope to a signal generator and ride the whole ground of the scope on a sine wave with a large voltage like 20V. You also float the device that your measuring (like a PCB) and connect the ground of the PCB to the scope (both are floating on a 20V sine wave). What happens? 

Nothing, the scope still measures the same potential as it would if both the device and scope were connected to mains because it measures the difference between ground and the probe. The both have the same common mode voltage.

As far as the outside of the scope goes, it's the same potential as the scopes ground and the inside is too, because most decent scopes have decent shielding:


Source: https://www.eevblog.com/2014/03/07/eevblog-587-tektronix-mdo3000-mixed-domain-oscilloscope-teardown/


  In general, if it's not a scope, but just a device that connects to
  something else's ground, and there might be high frequencies on that
  ground, what can be done to mitigate the issues?


There can be, it can be a major problem with some devices, if the grounding between devices has a lot of inductance it can block high frequencies and cause the potential between the devices to be different (happens with shielding cables all the time). EMI radiating into a device can change the ground, using proper grounding and shielding techniques can avoid problems.