Only Exposing CiviCRM Contact from Active Group in Drupal Views?
When you remove someone from a group, the status of their group membership is changed from 'Added' to 'Removed', so try adding another filter on status:

CiviCRM Groups: Contact Status (= Added)