What is the SQL Azure equivalent of the server "public" role?
After much experimentation, I stumbled upon an answer on the Secure your Azure SQL Database. 

Please correct me if I am wrong here!

It is strange that the documentation on securing SQL Azure does not mention the following in any detail.  The clues were here (emphasis mine):


  SQL Authentication, which uses username and password for logins and
  users that are valid only in the context of a specific database within
  a logical server.
  
  -snip-
  
  By default, the user can connect to the database, but has no
  permissions to read or write data. 


In SQL Azure, it is possible to have database-contained users with a password (which is different from the on-premise approach of creating a user against an existing login account).  This makes sense, because the all-in-one contained approach allows for a seamless failover.

The solution is to use SSMS and not the Azure portal:

-- After connecting to SQL Azure within SSMS (as dbo)
CREATE USER [UserNameForApplication] WITH PASSWORD=N'543rg.....ehgdGDF';
GO
ALTER ROLE [MyExistingDatabaseRole] ADD MEMBER UserNameForApplication;
GO


My application can now access the SQL Azure database with zero additional rights that would otherwise be granted via the Azure portal.