Editing the Book Template
The theme hook you'll want is menu_link__book_toc_123, where 123is the node ID of your book's outline node.
For the wrapping markup, use menu_tree__book_toc_123.

You can also find the ID by inspecting the navigation list: