Restoring Contacts groups from Time Machine
Over the last couple of weeks, I’ve been dealing with trying to selectively restore certain data types from TM. Upon disabling the iCloud on a few devices, I lost contacts, calendar, reminders and notes. 

I worked with a senior Apple advisor to restore contacts and calendars from TM. (We were unable to get reminders and notes - argh.) I experienced what you have, some restore attempts did not bring back groups. I use groups heavily and so was very determined to find a solution that brought back both contacts and groups. 

When I saw your post yesterday, I contacted the Apple advisor to confirm the steps we took to restore contacts, that included groups. He says that it was going to ~Library/Application Support/Address Book, and restoring that entire folder was the solution. I can’t remember for sure (we were trying lots of different things), but he says that was the one that worked. I have to assume he is correct.

In your message, you indicate that this is the route you tried without success. Keep trying...I want to let you know there is a solution, I was able to do so. 

C.