How to do a custom taxonomy breadcrumb
The taxonomy module provides a default breadcrumb service and if you have a look at the declaration of this in the taxonomy.services.yml file you will see it has a priority of 1002. 

Setting the priority of your breadcrumb service to a value higher than this should get Drupal to use your breadcrumb rather than the default one.

Something like this should do the trick:

custom_breadcrumb.services.yml

services:
  custom_breadcrumb.breadcrumb_blog:
    class: Drupal\custom_breadcrumb\Breadcrumb\BlogBreadcrumbBuilder
    tags:
      - { name: breadcrumb_builder, priority: 10001 }