Cancelling extrude still leaves extra geometry
Pressing E is really calling extrude_region_move(), which is a two part operation. It extrudes the selection and then moves it as if you had pressed G.
Pressing RMB or Esc will cancel the move part of the operation, but not the extrusion.

This is actually useful behavior, as it allows you to easily keep the extruded geometry (or loop cut) in place (maybe you want to scale it or something). If you don't want the extruded geometry, you can of course undo it with CtrlZ.  

This goes for all two part operations, such as CtrlR loop cut and slide (loopcut_slide).

Note that there are actually one part equivalents for these, but there aren't shortcuts for them (to my knowledge).
If you press Space in edit mode and search for e.g. Loop cut:



Pressing Loop Cut will put you in modal operator for cutting loops, just like pressing CtrlR, but it won't slide them like CtrlR does once the cuts are confirmed (and if they are not confirmed, e.g. by pressing Esc there will be no cuts made at all).