Listener does not connect to the correct instance
By default a standalone SQL Server instance (ie non Failover Cluster Instance), will listen on its port on All IP Addresses.  See  TCP/IP Properties (IP Addresses Tab).

And this means that the instance will accept TCP connections on its designated port on all current and future IP addresses.  So when the Cluster brings the AG Listener IP address online, it won't just be the SQL Instance that owns the AG that listens on that IP address.