How To NOT extend selection with Box Select?
Your assumption is understandable, yet you're approach is impractical in Blender.

Think of it this way.

When you hit B, you are executing an action: border select. This action can be aborted with Esc and you can choose mouseactions for deselect or select.

You are not using a tool. Once you completed the border select action, you are still using your previously selected tool (e.g. selection context). You will have to press B again to enter the border select action.

What you can do is setting the border select with extension to Shift + B and setting the border select without extension to B.


Press Ctrl + Alt + U to open the Blender User Preferences. Go to the Input rider. You can edit the hotkeys in this context. Expand the 3D View > 3D View (Global) area.

Locate the entry Border Select, expand it and uncheck Extend. Now the hotkey B will not extend the selection with the border selection, but replace it.

Locate the entry, which is assigned to Shift + B. In the default case this will be Zoom to Border and Set Render Boder. Disable it by unchecking the box at the start of the row. If we don't do this, the Shift + B will call this function first and ignore other assigned actions.

At the end of the 3D View (Global) List, press the button Add New. Enter view3D.select_border as the command. Set the hotkey to Shift B by clicking on the hotkey assigning area and pressing Shift + B.


  

You will now have a B and Shift + B border select action, which will work as you explained. Note that this is set only in the 3D area. In other contexts you will have to change the hotkeys as well. If you want to keep these hotkeys for the next time starting Blender, click the Save User Settings button on the bottom left of the Blender User Preferences window.