Delete fieldgroup when deleting plugin
You can see what Craft does when uninstalling a plugin by looking at craft/app/services/PluginsService's uninstallPlugin method.

Craft will delete any custom elements the plugin has defined from craft_elements and the cascading deletes across foreign keys to that table will trigger.  It will drop any tables defined by any plugin records. It will remove the plugin row from the craft_plugins table and any migrations from the craft_migrations table.

craft_field and craft_fieldgroups won't get picked up because they don't have any indirect FK connection to craft_elements, but you can add an onBeforeUninstall method in your base plugin class and do any custom cleanup you want there.