Second PhD in Astro after a PhD in CMT (Physics)
Don't plan on doing two PhD for all the reasons in the comments.

Whether to stay and throw yourself fully into the best and most active and dynamic CMT group (try to be a CMT star for goodness sakes)  or leave for another university depends on being honest with yourself about your reasons for choosing one over the other. (Is it 'liking' to call yourself an astrophysicist? ) Intellectually - they are both fascinating fields. You will find the smartest of the smart in either. 

You talk about losing work, but did not mention being involved in any research projects with either of the astrophysics faculty who are leaving. If you had, they likely would be already talking to you about whether a plan could be put in place to help you finish your project somehow.

It wasn't clear if you were in UK or US. In the US- those 2 years are typically core courses and preparation for the qualifiers. If so, those core courses will transfer pretty easily to any similar department, and if you did stellar in them, will probably make it easier to get into a department that matches your interests. You will not have to take the core again and in particular, the pressure will be that you should hit the ground running and  get involved in a significant way in a project (even if you have less time because you are supported initially on TA instead of RA). If you can pass the qualifier will be the main issue. 

As an additional note - most faculty understand if a student wants to switch schools because the subfield of interest is not represented. They will write recommendation letters without qualms and wish you the best.