Oscilloscope measurements on device connected to a AC/DC wall-plug
My initial thought was that there may be a Y-cap between primary and GND that serves as kind of a potential reference

That Y-rated capacitor is for spurious supression/EMI, not for providing a reference. Read more about the Y capacitor here

Fact is that isolated mains supplies which do not have a mains ground connection which is also connected to the low voltage output, are unreferenced. So essentially, if there is no connection between the ground at the output and mains earth then the supply is floating. Within certain limits of course, there always be some leakage and capacitive coupling.

Since that supply is "floating" you can set the ground reference yourself. In practice you could ground it to the oscilloscope's ground.

Compare this to using a lab supply, there the supplies are also floating but normal operation is to reference the negative supply rail (assuming a single supply voltage) to the ground plug on the lab supply. This plug will (should be) be mains earth connected.

wouldn't it be even an issue to, say connect a USB cable to some wall-plug powered device, effectively connecting PE to "some" potential?

That would achieve nothing if the wall plug is the "floating" type. If it has "some potential" it is useless to connect it to that as you do not know what it is. You better connect to a proper ground/mains earth which is a potential you do know (zero).