How can I force Composer to install a dev branch over a stable release?
While it will install the latest dev release of the Drupal module/theme to simply do the following:

composer require drupal/bootstrap:3.x-dev


I would argue a best practice when installing a dev version of a module is to make sure you are going to get the commit you want (which might not be the latest), so I would do the following:

composer require drupal/bootstrap:3.x-dev#5670922dea05fffb6506a6cd5d53546b9fae8df9


Or you can manually add it to your require in composer.json (as below) and then on the command line run composer install

"drupal/bootstrap": "3.x-dev#5670922dea05fffb6506a6cd5d53546b9fae8df9",


To get the commit you should go to http://cgit.drupalcode.org/bootstrap/log/ and find the commit you want (the commit hash is the id on the commit's page).
btw: the above hash is not a real commit on the Bootstrap project