A battery is not connected to anything. Is there a voltage between its plus and minus poles?
The electro-chemical reactions inside the battery happen only when there's a closed circuit.

For an ideal battery, this is true. However, all batteries have some, hopefully very small, leakage current, and will eventually discharge. Hopefully it takes years for this to happen, but that is not always the case.

When you place a voltmeter across the poles, then you create a closed circuit (with very low current)- the electro-chemical reactions start, and you now have a voltage drop between the poles.

The voltage between the poles existed prior to connecting the voltmeter or closing the circuit. It is the voltage between the poles that keeps the electro-chemical reactions from proceeding. If the battery is not connected to anything, the voltage between its poles exactly matches the electro-chemical potential of the reaction.

The placement of the voltmeter changes the thing you are trying to measure.

Yes.

Without the voltmeter, there's no voltage-drop between the poles, because there's no closed circuit.

This is mistaken. There is a voltage difference between the poles whenever the battery is holding a charge. There is no current when there is no closed circuit.

But the above seems completely inconsistent with how people speak of open-circuit voltage, as existing independent of actually measuring it.

Open circuits may have voltage, and this voltage exists independently of attempts to measure it.