Is it possible not to use utf8mb4 as encoding?
MySQL (and libmysqlclient) have a minimum version requirement of 5.5.3 for Drupal 8 (see https://www.drupal.org/node/2558615, https://www.drupal.org/node/2473301, etc).

It's not often that requirements are applied arbitrarily, so there will almost certainly be something in MySQL >=5.5.3 that core/vendor code requires.

As such, I would have to assume that no, it's not possible to change this without hacking core. Maybe not even with hacking core, as vendor supplied code may rely on it.