Moving from mathematics to astronomy/astrophysics
There are fewer placements than qualified applicants so competition is strong.  Moreover, suppose this was the other way round:  A person doing a PhD in (say) planetary nebulae comes to the maths department saying that they are really interested in maths, they're good at equations and calculus and really like all that topology stuff (you know, rubber sheets and so on) so can they start doing cohomology of differential graded categories?
It's not impossible, and a strong foundation in maths is a prerequisite for progress in Astronomy but the question you should ask is why should a university hire you, when there are 5 others with PhDs in astrophysics?
Talk to your supervisor and careers advisor before burning any bridges.