Is it possible to assign a "custom group" to a specific roles of users?
Use 'Access Control Lists'. You can access it by 'Administer->Users and Permissions'. Then go to 'Manage ACLs' to and define a ACL for a Custom Data Set.