How can I implement a flag module with caches but won't affect cache the flagging nodes?
Views Content Cache has a patch for flags support, although I don't know how well it will work with Boost.

Try using the Views Flag Refresh module, it's supposed to tell the view when to refresh, don't know if it will play nice with the other modules though.