problem : find the coefficients of linear system used to build a digital oscillator ..?
For a 3rd order TF, stability is easily determined from the relative values of the denominator coefficients.

For the denominator, \$ds^3+as^2+bs+c\$, if:

\$ab&gt;cd\$ the TF is stable,

\$ab=cd\$ the TF is marginally stable (i.e. a sine wave),

\$ab&lt;cd\$ the TF is unstable.