How can I change which view wins over two using the same path?
Use the internal path of the term:
/taxonomy/term/123

Route matching operates only on internal (unaliased) paths and route #2 having more fixed parts wins.
See https://api.drupal.org/api/drupal/core%21lib%21Drupal%21Core%21Routing%21RouteCompiler.php/function/RouteCompiler%3A%3AgetFit/9.0.x