Object oriented design for access control
AccessGroup should contain a Set of permissions to be extensible, rather than have the Read permission and Write permission directly attached. Otherwise you end up with a dozen attributes hanging off AccessGroup in a year's time.
(this also means you can just union the permissions on a user to see what they can do, instead of messing with AccessGroup.getXXXX attributes)
You're overloading the terms 'add' and 'remove'. You should createUser and deleteUser
The User class is quite anemic. Perhaps you could use composition to hide the entire AccessManagement class inside it, maybe by having an AuthenticatedUser subclass that knows about its groups and permissions.
Your mediator should have specific methods to swapIn and swapOut. TBH I'd model it as an event stream so there would be no updates, only inserts.