Algorithm to Count Zeros Outside Unit Circle for FIR Filter
Here is one answer, if someone can improve on this I will select it as the "right" answer (also comments very welcome on obvious flaws with this approach):

Given Cauchy's argument principle, the number of zeros outside the unit circle is given by the number of encirclements of the origin for the frequency response of the filter as plotted on a complex plane. Each encirclement causes the unwrapped phase to go further than $\pm \pi$ in any interval $\pi$ in normalized radian frequency. 

Thus the algorithmic approach would be to unwrap the phase, remove the initial phase at $\omega=0$ and then count the number of crossings of the horizontal lines given by $\pm \pi$, using the Bentley-Ottman algorithm to efficiently count the crossings: https://en.wikipedia.org/wiki/Bentley%E2%80%93Ottmann_algorithm. 

Further details on algorithmic phase unwrapping given here:

https://www.ljmu.ac.uk/~/media/files/ljmu/about-us/faculties-and-schools/fet/geri/onedimensionalphaseunwrapping_finalpdf.pdf