Cannot access plugin controllers when plugin does not have a "cp section"
If you don't want to give the user specific access to your plugin, I think the only solution is to allow anonymous access to the controller action. But of course, this would make the controller accessible to alll anonymous users, also outside the control panel.