Adding a Google account in iMessage appears to be stuck
As of some time in 2017, gTalk is no longer available. Messages/iMessage/iChat used the gTalk service to send and receive hangouts messages. It seems Messages 11.0 (the one I'm using with 10.13.5 on late 2009 MacBook) no longer has support for any functionality whatsoever with Google services. gTalk (2005-2017) was the precursor to Hangouts though it seems Hangouts later had backward support for gTalk which is why IM clients using gTalk still worked up until some time last year.

I submitted a feature request to add support in a future version of messages. It seems all the API technology is already available and that Messages and other clients merely need to add support in future versions of the messaging clients. You can submit a feature request to reinstate some method of Google service chat IM here. I bet if enough people request the feature, we'll see client support for Google IM return in a future version of Messages. I really miss it because I hate having to visit hangouts.com or mail.google.com every time I want to interact on hangouts.

Most of the information in this post is obsolete as it seems Messages had a bug prior to gTalk being discontinued. However, it means nothing now since the API no longer exists.

Update August 2018:

I found a third-party Github project that works really well. Very nice interface. It seems that Apple is just too lazy to add support for the new API and, instead, have simply removed the ability to add other accounts in Messages. Anyway, you can find binary installers here for all operating systems (macOS, Windows, or Linux).