The smart home field has witnessed rapid developments in recent years. Internet of Things applications for the smart home are very heterogeneous and continuously increasing in number, making user management from a security perspective very challenging. Moreover, the resource-constrained nature of most of the devices implies that any security mechanisms deployed should be lightweight and highly efficient. In this article, we propose an authentication scheme based on symmetric key cryptography, combined with a capability-based access control system, to provide the different stakeholders (residents, recurring guests, or temporary guests) end-to-end secure access to the Internet of Things devices in a smart home, managed by the home owner in an anonymous way. The operations in our scheme only include a small number of communication phases and protect the identities of the entities involved (i.e. stakeholders and end-nodes) from any outside entity. The proposed scheme ensures that even if the stakeholder's device or the Internet of Things device is attacked, the system remains secure.