in the selection mode "by pressing button b", how to deselect previous vertices? I'm confused by the selection mode triggered by pressing the shortcut key "b".

In this mode, it by default keeps my previous selection, and when I select a new vertex, for example, for most of the time, this new selection will be added to my old selection. But for some other times, my new selection will deselect the previous selection, and becomes the only selected vertex. 

I'm curious what alters this behavior. For most of the times, I want my new selection to deselect the old vertices. How do I do it?
When activating border select(b) middle mouse will deselect while left mouse will select.

-eh five