SQL Availability Groups: Can clients use cluster name instead of listener name(s)?
First, when using AGs, you generally are not using a clustered instance of SQL Server, you are using multiple standalone SQL Servers. You are able to technically connect to each individual SQL Server in the Windows Cluster Separately. Each having their own configuration and resources. 

Secondly, each group has its own name in AD as a computer object and its own IP address. These are registered in DNS. These are different than the cluster name. The listener name is a special object to route traffic based on incoming query type. For instance, if you configured your AG to use a secondary node for read only queries, the listener name is what routes that query to the secondary node, not the cluster group. Backups are also attached to that listener name for routing traffic from the primary to the secondary node. 

If you are monitoring AG health, it will be based off of the listener name/object, not the cluster name. It is possible to have a 3 node windows cluster, but only have an AG involving 2 nodes. The cluster name would not properly handle that. 

Adding further complexity to this question. With SQL 2016, you have distributed availability groups which uses separate clusters on different networks or domains. Those separate clusters are clusters all to them selves and are only joined via the listener name for failover purposes.

Lastly, if you chose to extend your AG to Azure, you will need to use the listener when connecting to the "active" database so your query can route to the appropriate node. The cluster resource name does not know about the Azure secondary node.