Size still the same after removing groups
One way in which you could be left with a bounding box larger than the remaining geometry would be if you deleted the objects while in Edit Mode. Then you would be left with objects with no geometry, but the objects would have their location, scale and rotation information still there.  If this is the case you would need to go to the Outliner to find the mesh-less objects and select them there and then Delete. If your deleted groups had parents, you'd need to open out the items in the Outliner in order to find them.