Designing an High Amperage, Low Voltage AC Voltage “Breaker” for Model Train
I'd be tempted to go into the digital domain, but if you don't want to do that, what I'd suggest is to divide down the measured voltage with a voltage divider, rectify it, low pass filter it, and then compare it with a DC reference. An AC reference will be out of phase!

You can then change the voltage divider to set the target voltage, rather than having to set the reference. LEDs can be used as budget references since the whole thing doesn't need to be too accurate.

This has the added advantage that you can feed the measured voltage to a DC panel meter and draw a red line on it at the trip threshold.

Consider also having a latching relay that needs to be manually cleared, or is cleared when the voltage goes below a much lower threshold. That way it doesn't start up at full voltage.