Fuse 125V/4A can be replaced with 250V/4A?
Fuses are used for over-current protection, first please diagnose the mistake what causing the fuse to blow (like any shortages, overloading or some faults) and solve that before you replace new fuse.

As long the voltage of fuse is more than the system voltage. you can use it.

Since, what you are saying is right.. fuses protects the system/equipment from over-current not over-voltage.