Granular role access on a per Entity basis
but that doesn't seem like a very good solution.


Yes, it is.

What you're saying makes perfect sense, just do it :)

access() (or checkAccess()) is called for each entity separately, you can implement any imaginable logic in there, that's completely up to you.