Contacts sync iCloud/iPhone/Mac after clean OS X install
I had the same issue with certain contacts and contact groups not coming back from iCloud after a clean Yosemite install. 

Today I found out that all this contact groups and contacts are starting to sync once I edited them, e.g. add a contact to a group, update a contacts phone number. For me it worked to edit them in icloud.com (webpage) or on an iOS device.