Realization of network function and positive real function
The poles are the ones that must be in the left-half plane, which means the denominator must be a positive real function. Since the zeroes need to be reflected along the imaginary axis, then the polynomial at the numerator cannot be a positive real function, anymore. And this is completely fine, since the zeroes can be anywhere, they don't influence the stability. It's the realpart of the poles that affect the exponential term (transient).
Let's analyze two generic, allpass transfer functions, for a 1st and a 2nd order system:
$$\begin{align}
H_1(s)&amp;=\frac{s-a}{s+b} \tag{1} \\
H_2(s)&amp;=\frac{s^2-a_1s+a_0}{s^2+b_1s+b0} \tag{2}
\end{align}
$$
Applying the inverse Laplace transform results in the impulse response. For the 1st order t.f. there is only one expression for the impulse reponse:
$$h_1(t)=-(a+b)\mathrm{e}^{-bt} \tag{3}$$
The presence of \$\mathrm{e}^{-bt}\$ shows that the function is of the form \$\mathrm{e}^{-x}\$ and thus will always be convergent. It's also dependent on the \$b\$ from the denominator, since \$-(a+b)\$ is just a constant.
For the 2nd order t.f., the impulse response will depend on whether \$b_1^2-4b_0\$ is greater than (overdamped), equal to (critically damped), or lesser than zero (underdamped). But all three will have one term in common: \$\mathrm{e}^{-b_1t}\$. Notice that it's identical to the 1st order t.f., except there is\$b_1\$ instead of \$b\$. Also notive its sign, and the fact that it's only a function of \$b_1\$, not any of the numerator terms.
Now, a 2nd order equation is very well defined in terms of it's roots \$r=\pm\alpha\pm j\beta\$ as:
$$s^2-2\Re(r)s+|r|^2=s^2-2\Re(r)s+\Re(r)^2+\Im(r)^2=0$$
This shows how, as long as the real part of the roots of the denominator are negative, the equation will result with a positive \$b_1\$ term, and \$\mathrm{e}^{-b_1t}\$ will always be convergent. Again, no term from the numerator.
Let's put the theory to practise. Here's a simple example in Octave of a 4th order allpass with random roots:
r=-randi(10,1,2)+1i*randi(10,1,2); r=[r;conj(r)](:), ...
num=poly(-r), ...
den=poly(r)

r =

  -9 + 3i
  -9 - 3i
  -3 + 2i
  -3 - 2i

num =

      1    -24    211   -774   1170

den =

      1     24    211    774   1170

The numerator is simply the poles with their real part positive. If what you say is true, then the system's step or impulse response should explode (i.e. should not be convergent):

Eppur si muove.

Since the question was not about building a SPICE example, I didn't think I should insist on that part, but, as per jonk's comment, I'll extend this answer for future prying eyes.
In case the SPICE approach is obscure, there are three things that are at the base of it:

the voltage across a capacitor is \$v=\frac1C\int_t{i\,\mathrm{d}t}\$
in LTspice (since this is what I used), current sources are a lot more convergent-friendly than voltage sources
in LTspice, Laplace sources are very reliable in .AC (frequency-domain) analysis, but can be quite awful in .TRAN (time-domain)

With these in mind, a current source with a capacitor in parallel will act as an integrator for the voltage across it. It will also replace a \$\frac1s\$ Laplace expression. Using a VCCS will allow a voltage feedback, so now the first canonical form of a transfer function can be built:

The difference between this and the first schematic is that in the latter, the nodes have summers, while the former shows a chain of integrators, which means they're multiplying. The reason for the SPICE version is simplification: why use summer and gain and integrator, when you can use one Swiss-army integrator? However, this means that the terms, as shown in the canonical form, if left untouched, will come out at the end of the chain all multiplied in the SPICE version. To circumvent that, take advantage of the fact that it is a chain, and so the last term can be divided by the previous one, then the previous one to the one before that, and so on. Now, when multiplied, they will simplfy on the way to the output leaving the terms clean. That's why, in the SPICE schematic, I left all the terms visible (i.e. I didn't just use the result of those divisions), so that even a value like {24/1} would make it clear that there are two terms in there, and how they got there.
Usually, long transfer functions are split into 2nd order sections. Here they would have resulted in \$s^2\pm 18s+90\$ and \$s^2\pm 6s+13\$, but I left it as a whole for better viewing of its terms.
As a last note, G13 and L1 (bottom right) form a differentiator, since \$v=L\frac{\mathrm{d}i}{\mathrm{d}t}\$. The input is a step function, and V(s) gives just that. A step function, in Laplace domain, is \$\frac1s\$; this is multiplied with the transfer function. Differentiating this results in a clean transfer function, whose inverse Laplace is the impulse response. Thus, differentiating the step response gives the impulse response, V(d).