Does RPM refer to the speed of the engine or the speed of the propeller?
Yes it is the propeller RPM.  If the engine is direct drive, the engine and propeller RPM are the same.  If the engine has a propeller reduction gearbox, then engine RPM and propeller RPM are different, the propeller RPM being the engine RPM minus the gear reduction.

Propeller RPM is controlled by a flyweight governor, like those spinning ball speed regulators you see on old steam engines.  The flyweights control the oil flow to the prop to vary the pitch. The prop control lever applies a spring preload to the governor flyweights and the amount of preload determines at what RPM there is a state of hydraulic equilibrium so there is no pitch change happening. The speed of the governor is a function of the speed of the propeller in either a direct drive or geared engine.