Which algorithm could be used to compare two waveforms from an oscilloscope in XY mode?
You can't get the exact value because you don't know how log the signals spent at each stage.
Consider the case where you have two square with period 1, and shifted by $\delta$ compared to each other.
The XY plot will always be a rectangle with the width corresponding to the amplitude of the X signal and height corresponding to the amplitude of the Y signal. In other words, you have always the same input for your function to compute the correlation, while the correct output varies.
You still can get an idea of how similar two signals are, but it is not the signal cross correlation