How to set permissions on an angular page
I found a solution for this. The developer docs (https://docs.civicrm.org/dev/en/latest/framework/angular/loader/) mention how you could create your own angular page, which is a regular civicrm page but at a different url. 

Because it is a regular page I could set permissions for that just as I was used to do. See https://docs.civicrm.org/dev/en/latest/framework/routing/#access_arguments for how to do that.