Solder mask for 0.5mm pitch TQFP
You should ask your board house what they need to account for registration errors. My board house needs 0.003" on each side of the pad. As for the paste layer, it depends on the pasting process. If it's the classic stencil and squeegee process, you need to be mindful of the thickness of the stencil. I recently did a 0.5mm TQFP, and with 95% coverage and a 0.005" stencil, did just fine. Having leads for excess solder to climb helps a great deal. I'm not aware of any hard, fast rules here.