How can I fix the node list page broken by changing the reference field from a term reference to an entity reference
I got it.  I had initially been trying to create  a new view to operate on the /pub/by/publisher/% path, but I realized by inspecting the view, that the view I already had on /taxonomy/term/% was handling the route because links to one of the pathauto generated routes end up being to the equivalent of /pub/by/publisher/term-name?destination=/taxonomy/term/tid

The problem is then to get the existing view to show the results I want.  In my case, since the existing view was showing no results for any publisher vocabulary term, and publisher terms should be the only terms that return results for my new view; I had the option of adding an attachment or other view in the no results section or to just create an attachment attached after the main view.  I added an attachment and added it after the main view.  Since I needed the results of two entity reference fields I needed an OR condition between my contextual filters.  For this I used Views Contextual Filters OR.

One additional problem I faced was that my filters and contextual filters were being joined with an OR condition rather than an AND condition, meaning that I was effectively querying for public nodes or the right type OR the nodes I actually want, which returned way more results than intended.  The solution for this, was to add a new empty filter group to my attachment, joined to the original filter group with an AND condition, this caused it to start joining my contextual filters with AND as well.

Finally, my main view had some exposed filters on it, but thought I had inherit exposed filters checked on the attachment, those exposed filters were not working.  It turns out that you need to add the same exposed filters to the attachment before the inheritance on them will work.

I doubt many people will have exactly the same problems I had, but hopefully it will provide a clue for people with something similar.  As seems to always be the case @no-sssweat was instrumental in providing the hints I needed to get to the answer.  Thank You!