Azure Readable Secondary - How to: Different Access Policy to Primary?
The only way to have different access policies. e.g A reader (by whatever principal - AAD group user etc) can only access the secondary-

Make the 'Active Directory admin' of the logical server hosting the secondary replica an AAD group.
Make the user or group that needs to read from the secondary replica is a member of the group mentioned in step 1.
As a pre-requisite you need to set the properties of secondary to read-only or all.