Why does the reference electrode in potentiometry have a diaphragm?
This device is basically a galvanic cell (and an amplifier and voltmeter). Galvanic cells are quite similar to electrolytic systems, the difference being wheter one has an external voltage source.  If there is no “salt bridge” (i.e. some way for ions to wander between the two half-cells), the cell immediately stops working. This is certainly true, since one can directly observe it.

This makes sense if one considers that in (about all?) systems only one kind of metal ions is involved at each electrode. If you disconnect the two half-cells, there is a potential difference, but only between the electrode and the electrolyte. (How do you measure such a thing without intruducing another electrode??) The ions would simply stick around near the electrode until the charge makes the reaction unlikely to occur; only if the ions in the electrolyte can mix somewhat can the reaction continue. Keep in mind that the voltmeter does not have infinite resistance but "only" 10MOhms or so. This animation might be quite helpful.
In different terms: One needs a small current (at least for a short time) to measure voltage (keep in mind that the amplifier will have some capacitance) and for this to occur the circuit has to be closed, on one hand via the voltmeter, on the other via the salt bridge. 

But, since you do not want to influence the measured system too much, you only provide this almost tight diaphragm — not a large(ish) diameter hole — and almost no current flows. I see no theoretical reason why a very small hole would not work equally well, but I assume this is more annoying for technical reasons (drilling it, clogging, etc.). One could argue that very small currents can flow even without a salt bridge because separate containers are still not infinitely far apart and their fields can influence the other one, but this seems to be negligible; normal glass is already thick enough to limit capacity to a few pF.