What protocols and tools does Arduino Due use for flashing and debugging?
The Due (more specifically the SAM3X) uses its own proprietary protocol. It's all embedded in the bootloader, which is hard wired in the ROM and can never be changed (idiotic if you ask me).

I know of two systems for communicating with it.  Atmel's SAM-BA program, and the open source BOSSA.  The command line version of BOSSA (bossac) is used by the Arduino IDE for programming.