How do I change what blocks show up on a user page depending on the user role of that page? It is possible to filter blocks based on the user role of the current user, but I want to be able to change blocks based on roles of the user whose page I am visiting.

User 1 is an administrator so visiting 

 /user/1


users see 

  Administrator block


no matter what user role the current user logged in has.

if User 1 visits

 /user/4


where user 4 is RoleX they see 

RoleX Block


Where RoleX block and Administrator block are both created in views. 

Is there a way to do this out of the box or do I need to write my own contextual filter
You need to expose that entity as a context to Drupal, then the visibility settings should automatically let you select between the current user and the user on the page.

Drupal core only does that for nodes at the moment, the logic for that is in \Drupal\node\ContextProvider\NodeRouteContext with its service definition:

node.node_route_context:
class: Drupal\node\ContextProvider\NodeRouteContext
arguments: ['@current_route_match']
tags:
  - { name: 'context_provider' }


So you need to duplicate that class and basically do a search and replace for node => user, remove the node.add part and put that in a custom module.