What is the alternative of hook_custom_theme in Drupal 8?
That is a service now. See \Drupal\user\Theme\AdminNegotiator for an example, which is registered as a tagged service:

theme.negotiator.admin_theme:
  class: Drupal\user\Theme\AdminNegotiator
  arguments: ['@current_user', '@config.factory', '@entity.manager', '@router.admin_context']
  tags:
    - { name: theme_negotiator, priority: -40 }