SSMS connection to Availability group listener doesn't failover
What you describe is normal behavior for SSMS, at least all the times I've done failover testing.
But Management Studio is a poor tool to test this type of thing.  Best practice for an application is to have retry logic built into it that will upon error dispose of the current connection and re-establish, which will connect to the now primary side.  And SSMS does not do that.
I've never seen or caught SSMS auto-magically re-connecting to the primary side once it's connection is established.