Build a proper Cycles shadow pass in the compositor
I was able to solve it in the end by darkening certain passes before combining them. The usual combination method is Direct Pass + Indirect Pass multiplied by color pass. The Direct+Indirect contains all the lighting and shadow information, but does not have any material colors to throw off the ramping.

I took the direct+indirect, ramped it to get a mask of the desired darkening, then multiplied the direct+indirect by it. Then I multiplied by color, completing the combined pass. This was also done for the Gloss using a different ramp for more artistic control.