When does civicrm-webform remove you from groups?
Webform-CiviCRM will remove you from a group:


Only if you are logged-in or following a checksum (or otherwise selected as an existing contact)
Only if the group is presented on the webform as a checkbox/multiselect field and you deselect it.


In other words, the user has to make an active choice. Filling out the form as an unknown user, you're not consciously choosing to un check the box (it just defaults that way) so in that instance the module will only add, not remove you from groups.

BTW it works that way for all webform fields. The module will not delete any data from a contact record unless that contact has already been identified and the data was pre-filled on the form and the user made an active choice to erase it.