Is it possible to remove Route objects from a RouteCollection object?
There is also a RoutingEvents::ALTER event, which runs afterward.

You can think of the difference between RoutingEvents::DYNAMIC and RoutingEvents::ALTER as similar to the difference between hook_menu() and hook_menu_alter().

There is a Views issue that discusses replacing routes, as opposed to removing and adding them, perhaps follow their example.