Bireciprocal lattice wave digital filter
In the meantime someone gave me the answer: the first formula, \$\gamma_i=\frac{\sigma_i-1}{\sigma_i+1}\$ is correct, but the determination of the poles in the s-domain is wrong, in that all the four parameters, \$A_s\$, \$A_p\$, \$\omega_s\$ and \$\omega_p\$, need to be specified such that the resulting \$N\$ is closest to integer, without using ceil(). In other words, the best approach is to impose \$N\$ and deduce one of the four parameters from the other three.
With the same example from the thesis (p.27) and a quick change in my formulas, I get this result:


where the blue trace is from the thesis and the black one - mine. Even if there still are differences, they are minor, closer to the truth and probably due to one of the other 2 possible optimizations (which are too difficult to achieve with LTspice). Either way, this is the right path.