What effect would a afterburner have on the compressor if the nozzle outlet is not increased?
Yes, when the AB is lit, the nozzle needs to be opened, because the increase in exhaust temperature causes a decrease in density (pressure remains about the same), and so the same mass flow needs a bigger nozzle area to flow through. In AB, the engine mass flow is about the same as in max dry operation. The AB basically sits behind the core, and operates more or less independently, because the change in nozzle area compensates for the change in exhaust temperature.

And you are exactly correct, if the nozzle wasn't reduced when the AB is turned off, the spools would over speed.

If the nozzle area wasn't increased, the sudden lighting of the AB would most likely stall the engine - probably the fan as the back pressure wave can easily travel back up the bypass duct. AB stalls were common on first generation low bypass military engines. If you could light the AB slowly, it should just cause the spools to slow down, and mass flow to correspondingly decrease. Modern ABs light a bit progressively, to give a "soft" light. Older designs weren't quite so refined.

E.g. See page 208:


  The TF30 ... still had compressor problems going in and out of afterburner..


And here about the F100 saying:


  Some stagnation stalls were caused by "hard" afterburner lights, which were mini explosions that took place inside the afterburner when it was lit up. The pressure wave from the explosion then propagated forward through the duct to the fan, causing the fan to stall and sometimes even causing the forward compressor stage to stall as well. These types of stagnation stalls usually occurred at high altitudes and at high Mach numbers.


Also read Goodbye F100 Stagnation Stalls, see:


  To reduce the influence of pressure waves traveling back up the fan duct, the compressor inlet lip has been moved to bring it closer to the rear of the fan.