Nexus 4: Can't get phone into vibrate only mode
Try holding the volume-down button as you do now, and then after the volume slider reaches the bottom, tap the volume-down button once more. This goes into silent (not vibrate) mode on my Nexus 7 running 4.3, but I think that's because it can't vibrate.

Alternatively, hold down the power button as if you're going to turn the device off. The power-off menu has on-screen buttons for noisy, vibrate, and silent modes.