Why isn't there a potential difference across a disconnected diode?
I think, the answer is relatively simple.
Do you know the working principle of a "Schottky diode", which is based on a semiconductor-metal junction?
Now - what happens if you connect a voltmeter (or any other load) across the diode? You create two Schottky junctions which exactly compensate the diffusion voltage inside the pn diode. Thus, no voltage can be measured.
With other words: You cannot use the diffusion voltage to drive any current through an external load.