Unflagging (by rule) doesnt show rightaway
I am assuming you are using flag's javascript toggle to set these flags, and if that's correct, you could hook into flag's flagGlobalAfterLinkUpdate event and refresh the portion of the page where your flags are set.  More details are here:  http://drupal.org/node/336122.  If you don't want to do this, you could always set the flag link to a normal link, which would refresh the page for you in of itself.