Proper way to create custom tables
hook_schema() is not used differently from how it was used on Drupal 7, the only difference is that it is not used to create database tables for entities. (See, for example, node_schema().) In particular, you don't need to call drupal_install_schema()) in hook_install() to create custom database tables, which was instead necessary in Drupal 6.

.schema.yml files (for example, user.schema.yml) are used to define the schema for the configuration used in the configuration files from the module. Its purpose is different from defining a database table schema, and they cannot be use for defining custom tables used from third-party modules.