Display single field from argument taxonomy term in Views header area
I've included my original response below but it turns out I'm an idiot: All you have to do is create the taxonomy-term as a relationship and then you can just pick out the field you want. D'oh!

~~~ my original answer

Yeah, so this is a problem that is currently unsolvable without code as of Drupal 8.6.0. Unless you deploy Display Suite, which is using a sledgehammer to crack a nut.

There's a lot of discussion on drupal.org about the lack of control on title fields in respect to nodes but the issue goes deeper. It applies to Taxonomy as well: You cannot stop the title field from appearing in any view mode because that's the way the template works.

The only solution is to create a new template that doesn't display the title (in my case taxonomy-term--book-series.twig.html) plus the supporting hook_theme() and hook_preprocess_taxonomy_term__book_series() all of which is a PITA.

Even then it's not great because there are no (default) theme suggestions for taxonomy terms that include view modes. It would have been much nicer if I could have designed a view mode called "description" and then had a theme suggestion of taxonomy-term--description.twig.html because that would then work in all cases. Instead I have to create the same template and support code but for a different vocabulary.

All-in-all, not the best experience.