type3 compensator's Vref
This is the whole point of the control system: you amplify the dc error between the target (or an image of it via resistors \$R_1\$ and \$R_4\$) to meet the regulation point. The difference epsilon or \$\epsilon\$ is amplified by the op-amp open-loop gain and seen as the dc bias on its output driving the power stage. In the below example, the drive is 1 V which via the pulse-width modulator induces a duty ratio of 51%. It is slightly more than 50% to compensate for the drop in the inductor ohmic loss. It would even be larger if you would include the MOSFET \$r_{DS(on}\$:

In the below example, the static error between the wanted scaled-down 5-V output and the 2.5-V reference voltage is 34 µV and amplified by the op-amp gain which exhibits an open-loop gain of 30000. If this open-loop gain would theoretically be infinite, the system would have a zero static error.
You are correct in your remarks that the op-amp could rail up or down in some cases. For instance during the start-up sequence, when \$V_{out}\$ is rising and far from the target: the op-amp asks for the maximum power and drives the duty ratio to the maximum value by hitting its upper rail. During a sudden load release also, as the output can potentially overshoot. In this case, the op-amp will rail down considering the large static error and recovers a few moments later when \$V_{out}\$ returns to its steady-state value.