Why would this (simple) over voltage protection circuit raise the output voltage?
R4 is needed to form the reference voltage with D2 by providing the bias current. The intersection of the V_in, R4 with the I-V curve of the Zener diode determines the bias point of that zener. If R4 was not present, D2 is only connected to the base of the BJT through R5, which would require negative base current.

Regarding your observation of a higher voltage at the output than at the input. From a theoretical standpoint, it should not be higher. Troubleshooting is all about breaking the problem down into smaller blocks and checking your measurements against your expectations. 

Here's what I would do to troubleshoot the system.


Disconnect everything from the input, check V_in, does it match your power supply setting?
Double check the orientation, connections, and component values of all components just as a sanity check.
Measure the net between R4 and D2 while disconnected from the R5, Q2, Q1. What this is doing is checking where the reference level is at. This should be near 2.5 V assuming your input is 2.5.   
Measure the base of Q1 and R6. This should be at 0 V as Q2 is off and R6 is pulling to ground.


If any of these voltages are off, you can look deeper into that area and try to determine what is going wrong.

Regarding charging supercapacitors. You can charge supercapacitors in many ways, constant current, constant power, etc. The design of the charging circuit should protect against overcurrent and reverse voltage.

A simple Shottkey diode and current limiting resistor is will work. This isn't the fastest way, but it gets the job done. Reference [1] shows an example of this circuit in Figure 2a. Size the current limiting resistor such that V_in / R is less than the maximum current rating of your supercapacitor. 

If you want the overvoltage protection, you can put the Shottkey diode in front of your overvoltage circuit and the resistor on the output. The Shottkey will protect the overvoltage circuit as well. Before you do this, solve whatever problems you're having with the overvoltage circuit first. Don't add more complexity to troubleshoot.

[1] http://www.ti.com/lit/an/slva920/slva920.pdf