How do I go invisible in Messages (OS X)?
You don't get invisibility using Apple's Messages app on OS X. According to Google, their client also does not support invisible status, so perhaps it's a limitation of their API and only something you can set using their web app. They get to serve you ads when you are on the web, but not on a client so perhaps that was a factor in deciding to code that function in one place over the other.

Apple's Messages app on OS X will set an invisible status for AIM accounts, but not for Google Talk, so it's less like a Messages oversight as an account/API design issue from Google.