SWD and i2C hardware layer differences
I2C spec: http://www.nxp.com/documents/user_manual/UM10204.pdf

The main difference is in the handling of multiple devices on the same pair of wires. I2C allows this, SWD does not. As a result, I2C was designed to use open-collector logic with pull-up resistors: this is robust (no way one device can damage another by fighting over whether the line is 1 or 0) but it drastically limits the speed. SWD is single-master: the debug host is always controlling the bus, and occasionally reading data up from the slave. I2C is multi-master: any device can start talking at any time.

In fact, the 5MHz "ultra-fast" version (see section 3.2 of the spec) gives up multi-master and bi-directionality in order to achieve that speed.