CiviCase dashboard shows cases for invisible contacts
You're correct that the case dashboard permission is a bit hardcoded to just "all" or "my" cases, e.g. https://github.com/civicrm/civicrm-core/blob/8036d43973a5d42b0722cc45cba3a9f88e9ce182/CRM/Case/BAO/Case.php#L695