Why does the rotor thrust decrease at smaller tip clearance for shrouded rotor?
This is the reason why a jet engine needs a shroud around its core: to reduce blade tip loss. When you decrease the wingtip clearance, you are curbing the wingtip vortices or the wingtip pressure bleed, and in doing so you are making each blade more and more effective at approximating its ideal i.e. infinite span high-low pressure spatial distribution under and above it, and this more ideal pressure difference accelerates the airflow better, which results in more inflow, which reduces the angle of attack of each blade a little, but increases the overall thrust produced. Think of the shroud as a big end plate for each and every blade. Think of this setup as the fan in a high bypass ratio turbofan: less fan tip clearance = more thrust.

EDIT: I'm not sure the author understood it as I do in his paper. He never clearly explained it in his writing. Plus, just why is there so much lift distributed at the wingtip? I've never seen anyone design a propeller with so much bending moment. He is making a propeller much heavier than it really is because the propeller could be less rigid and lighter and still providing the same thrust. If you really want an efficient prop, put the majority of lift near the midsection, and let the lift taper when approaching the tip.

EDIT: See how shroud works on a bona fide jet engine compressor: