Troubleshooting slow connection establishment
Since the connection is fast from sqlcmd, but slow from the .NET Core SqlClient driver, it's most likely this problem is on the client driver side (rather than an AG configuration issue).
You should add MultiSubnetFailover=true to the connection string to speed things up, as it shortens the retry interval. See the docs for details:
Connecting With MultiSubnetFailover

When MultiSubnetFailover=True is specified for a connection, the client retries TCP connection attempts faster than the operating system’s default TCP retransmit intervals. This enables faster reconnection after failover of either an AlwaysOn Availability Group or an AlwaysOn Failover Cluster Instance, and is applicable to both single- and multi-subnet Availability Groups and Failover Cluster Instances.

This isn't needed anymore on .NET Framework 4.6.1 and higher because TransparentNetworkIPResolution is on by default (which causes the driver to attempt connections to all the replicas in parallel, and use the first one that works, rather than trying them one-by-one).  But it looks like that enhancement hasn't made it to .NET Core yet.