How to add a Custom Operation in the Book Admin Edit Form's Operations?
You can override the bookAdminTableTree() method in the BookAdminEditForm class by extending core/modules/book/src/Form/BookAdminEditForm.php with your own custom form class. You can copy the bookAdminTableTree() method's code and then add an item to the $form[$id]['operations']['#links'] array for 'add-sub-page' (or whatever you want to name the new operation.)

Then alter the existing /admin/structure/book/{node} route (book.admin_edit) in book.routing.yml with a route to your new custom form class.  See www.drupal.org/docs/8/api/routing-system/altering-existing-routes-and-adding-new-routes-based-on-dynamic-ones and the setDefault() method in the Route class: api.drupal.org/api/drupal/vendor%21symfony%21routing%21Route.php/class/Route/8.2.x

This is a helpful tutorial that describes the overall approach: https://www.jaypan.com/tutorial/drupal-8-extending-core-configuration-extending-core-forms-and-overriding-core-routes