How does the nozzle diameter affect the thrust of a ducted propeller?
Welcome. I'm afraid your theory wasn't actually working. Reducing the duct's exit diameter led to an increase in its internal pressure, increasing load on the propeller, and likely even causing some reverse flow.

Nozzle design is a complex subject. I can't think of a way to condense it, even limited to one specific case, into a suitable answer; maybe I don't get it enough myself. Keep that in mind; the below is just one small shard of the whole and by no means the full picture. 

In general, the job of a nozzle is to match the pressure at jet engine exit to that outside it. If the pressure is different, it gets matched outside the engine, where it doesn't produce thrust. 

When the engine is a rocket, which creates high pressure, the nozzle needs to expand the gas, converting pressure to thrust through acting on the nozzle. When the engine is a cold fan, which accelerates air, it's the opposite - the nozzle needs to compensate for the loss of pressure with a bit of compression, so that the air stream can exit without fighting the pressure of outside air at the back.

It's important for a convergent nozzle not to compress the exhaust to a higher pressure than the outside air, else it will destroy thrust. That was your case, the nozzle was too narrow, so it compressed the air to above ambient - which caused it to try and blow back through the fan.

To give a practical answer, an optimal nozzle at these velocities would be very similar to a simple duct, narrowing just a percent or two at the end, with a smooth exit shape.