Grant access to only one key in azure keyvault
Permissions for keys are at the vault level. In order to get the granularity that you want, you would have to create another key vault.

From the documentation (emphasis added):


  Key vault access policies grant permissions to keys, secrets and certificates separately. For example, you can give a user access to only keys, but no permissions for secrets. However, permissions to access keys or secrets or certificates are at the vault level. In other words, key vault access policy does not support object level permissions. You can use Azure portal, the Azure CLI tools, PowerShell, or the key vault Management REST APIs to set access policies for a key vault.