How can I use different database backend for certain tables?
That's actually more than one question because the answer is different.

To use redis as the cache, you can use our port on github: https://github.com/md-systems/redis

The README isn't been fully updated yet but the quickstart part tells you how to configure it (after enabling the module):

  $settings['redis.connection']['interface'] = 'PhpRedis'; // Can be "Predis".
  $settings['redis.connection']['host']      = '1.2.3.4';  // Your Redis instance hostname.
  $settings['cache']['default'] = 'cache.backend.redis';


To use redis for the cache tags checksum service, which most cache backends use, you need to override a service, for example in sites/default/services.yml, add the following:

To use the redis cache tag checksum implementation, put this in
sites/default/services.yml or any other yml file that is loaded after
core.services.yml:

services:
  cache_tags.invalidator.checksum:
    class: Drupal\redis\Cache\RedisCacheTagsChecksum
    arguments: ['@redis.factory']
    tags:
      - { name: cache_tags_invalidator }


In the same way, you can replace any service that is defined in the core or one of the module services.yml files with a different implementation, for example some of the mongodb service implementations provided by https://www.drupal.org/project/mongodb.

In general, you don't think in terms of tables but services and API's, and you either replace them directly or by configuration he factory service (cache/key value/queue/...)