How do I build Breadcrumbs for a certain node?
I eventually solved it by creating my own subclass of the PathBasedBredcrumbBuilder from core. This class unlike core's version actually uses the passed in route_match and returns breadcrumbs based on that which makes it possible to get breadcrumbs for a certain node (or any other given route). 

I still need to calculate the route match from the current node to pass to the build method (as shown in the question). It feels a bit awkward since the route_match is converted back into a url object in the build method but  it works and it is the cleanest solution i've come up with so far.

Below is the subclass:

namespace Drupal\contextual_breadcrumbs;

use Drupal\system\PathBasedBreadcrumbBuilder;
use Drupal\Core\Routing\RequestContext;
use Drupal\Core\Routing\RouteMatchInterface;
use Drupal\Core\Url;

class ContextualPathBasedBreadcrumbBuilder extends PathBasedBreadcrumbBuilder {

    /**
    * {@inheritdoc}
    */
    public function build(RouteMatchInterface $route_match) {

      /// Set request context from passed in $route_match
      $url = Url::fromRouteMatch($route_match);
      if ($request = $this-&gt;getRequestForPath($url-&gt;toString(), [])) {
        $context = new RequestContext();
        $context-&gt;fromRequest($request);
        $this-&gt;context = $context;
      } 

      // Build breadcrumbs using new context ($route_match is unused)
      return parent::build($route_match);
    }

}