What is the correct way to document the schema for a YAML file? As a module developer, I'd like to convert some of my Drupal 7 info hooks to YAML discoveries in Drupal 8, but it's not clear to me how to document the schema of a YAML file for consumers. For a hook, of course, I would define hook_whatever() in mymodule.api.php. What do I do for *.whatever.yml? What is core doing?
There is no "documenting" of hooks in D8 except the mymodule.api.php as you've already mentioned(+ impleneting hook_hook_info).

The YAML files are only for:


services
routes
menu local tasks
menu local actions
libraries
module info
simple configuration
simple configuration schema


Now you've mentioned "info" hooks. In Drupal 8 these are Annotations and their implementations are Plugins. If this is assumption correct then see the documentation in here https://drupal.org/developing/api/8/plugins for plugins - what they are, how they work and how to use them.