XT 11 speed derailleur (RD-M8000-GS) rubbing 46T cog with b-screw all the way in?
As the comments suggest, you need to install the chain to adjust the rear mech. You can test this yourself by manouvering the shifter to mimic having a chain and you should notice that the mech cage can move up and down.
The proper intaliation process is:

Install derailleur with b screw half way on the hanger
Install chain, length calculated using the manufacturer's instruction
Adjust b screw according to manufaturer's instructions
Fine tune the shifter/ cable
Re-check the b screw as the fine tuning of the cable can change it

On a side note, I tried this same setup, but with xtr 11s mech and xt 10s shifter, and the middle gears where always off and a little jumpy. So I just bought the 11 shifter and I wouldn't recomment mixing it.
Whether the mech is direct mount or not, shouldn't affect this, as long as the mount is not bent or damaged.