Rendering: Excluding a layer while keeping it's influence (Masking?)
It should work if you replace the transparent with a holdout shader. However that doesn't solve the Memory issue. 
There's a fairly easy solution, which of course doesn't work in all cases.
In post you can use the Z-pass in order to combine renderlayers. 
Compositor -> SHIFT A -> Color -> Z-Combine.
It is intended to do what you want, but it will produce strange results when MB or DoF is involved.