Properly override template_preprocess_book_navigation function in Drupal 8 Book module
You can put a template_preprocess() function in a custom module or your theme mytheme.theme file.
It depends on what you want to achieve.
If this preprocess function is specific to the theme (for example, the results are used in a theme template), then you should put it in mytheme.theme.
But, if you put the code in mytheme.theme and then change themes, your book navigation will no longer behave as expected.
So, if the preprocess function does general processing that you might want to use in multiple themes, and it doesn't do anything theme-specific, then it's better to put it in a custom module.  (This is probably your use case.)