Is my water pressure reducing valve faulty?
Yes, that sounds defective. 

A pressure regulator (which is what a PRV is) should stop flow when the pressure exceeds the setting. This sounds like you may have a small leak, which could be from debris trapped in the valve, a scratched valve seat or something like that.

Before spending money on it, determine if the PRV is your responsibility or the water company's responsibility. If it's theirs, they may just come swap it out for you.

Your described behavior (particularly the speed of it) would seem to be a leaky PRV. A slightly different case arises when there is a PRV (which also acts like a one-way or check valve) and a hot water heater, with no expansion tank - then the water pressure can rise when cold water is heated and no valves are open.