How to implement per user caching when restricting display of comments by author
As Clive said in the comments, all I needed was the following.
I won't mind if people down vote the question for being too stupid…

// was user.permissions, but that means there will only be a single
// cache shared by all reviewers, and we must cache according to authorship
$build['#cache']['contexts'] = ['user'];


Caveat: the view block cache setting also needs to be 'none'.

Drupal docs - cacheability of render arrays