What exactly does "Sync Google Contacts" checkbox in iTunes do?
The UI for configuring Google Contact Sync in Address Book and in iTunes is intended to support the same feature. If it is enabled in one place and not enabled in another place, it is a bug in the Apple software.

Pre-iCloud, all contact syncing worked through a service called Sync Services. Google Contact Sync is one of the clients of contact syncing via Sync Services, and both Address Book and iTunes provide a user interface for enabling Google Contact Sync.