Unable to boot my Nexus 5 into recovery mode I was having Marshmallow stock ROM rooted with TWRP installed. The way I had installed TWRP recoery made the monthly security updates uninstallable. 

After learning the mistake, I decided to download the MOB30D build and flashed as I said in my answer.

After installing the phone was working normally. I tried to install TWRP through ADB command line and there was no error messages.  When I rebooted the phone in recovery mode I find a red triangle.  After some fiddling with power and volume buttons I saw the following recovery screen and used it for rebooting only:


I tried to flash the latest TWRP version, downloading again thinking the download might be corrupted. I then tried several old version of TWRP and same result.  I tried flashing stock recovery, doing all these again with erasing the recovery partition first - all results in red triangle.

I then tried Nexus RootKit flashing stock and TWRP image (downloading again); still no use.  I tried Nexus Rootkit's "Boot into temp recovery image" which did not work well but suddenly worked.  I flashed latest TWRP image through the temp TWRP and it too says all okay, but no joy:


Question:
I am not sure what else I can do to install TWRP. I want TWRP basically to root and for emergency purposes.  What I can do further to install TWRP and if not at least root without a recovery partition.

Update
I tried CWM recovery as well and same result. I tried to lock the bootloader and unlock again there by resetting everything. Still no use.
The problem is your not flashing SuperSU or deleting /system/bin/install-recovery.sh. Android has safeguards which revert to the stock recovery thru the install-recovery.sh, SuperSU by default disables this function so I'd recommend flashing it after you install TWRP.

Steps


fastboot flash recovery twrp.img or fastboot boot twrp.img
flash twrp.img inside of recovery if you just booted it instead of using fastboot flash
flash SuperSU-2.79-SR3.zip


OR


tap the mount tab
tap System 
go back to main screen and tap advanced then file manager
navigate to /system/bin and delete install-recovery.sh


I recommend using the SuperSU route but whatever floats your boat.