Nexus 5, Marshmallow, and TWRP recovery - Unstable, inaccesible recovery
The same thing happened to my Nexus 5 as well,when I upgraded to android M and tried rooting and flashing recovery.The process went perfectly fine but when I tried to enter recovery mode it didn't happen.
For which I tried using the Nexus Root Toolkit,which has several functionalities required.I tried rooting my phone and then flashing the stock recovery,everything worked like a charm for me. So I would suggest you to download the Nexus Root Toolkit by wug fresh then first try to flash the stock ROM again there after Root followed by flashing the custom recovery.