Slow Connection to Azure SQL Database using SQL Server Authentication
Here is the Microsoft explanation of this issue:
https://techcommunity.microsoft.com/t5/azure-database-support-blog/lesson-learned-132-delays-connecting-to-azure-sql-database-from/ba-p/1502030
They describe two types of authenticated users.  Ones with instance LOGIN's and ones that have only access to a single database (&quot;contained&quot; database users).
The problem with the slow SSMS connectivity only affects the first type of authentication (instance LOGIN's when they have access to a database but have no rights to the &quot;master&quot; database).  If you are able to switch to &quot;contained&quot; database users then you may avoid the problem altogether.