Title: Radial orbit instability in systems of highly eccentric orbits: Antonov problem reviewed,
Abstract: Stationary stellar systems with radially elongated orbits are subject to
radial orbit instability -- an important phenomenon that structures galaxies.
Antonov (1973) presented a formal proof of the instability for spherical
systems in the limit of purely radial orbits. However, such spheres have highly
inhomogeneous density distributions with singularity $\sim 1/r^2$, resulting in
an inconsistency in the proof. The proof can be refined, if one considers an
orbital distribution close to purely radial, but not entirely radial, which
allows to avoid the central singularity. For this purpose we employ
non-singular analogs of generalised polytropes elaborated recently in our work
in order to derive and solve new integral equations adopted for calculation of
unstable eigenmodes in systems with nearly radial orbits. In addition, we
establish a link between our and Antonov's approaches and uncover the meaning
of infinite entities in the purely radial case. Maximum growth rates tend to
infinity as the system becomes more and more radially anisotropic. The
instability takes place both for even and odd spherical harmonics, with all
unstable modes developing rapidly, i.e. having eigenfrequencies comparable to
or greater than typical orbital frequencies. This invalidates orbital
approximation in the case of systems with all orbits very close to purely
radial.