Astrophysics after Aerospace Engineering?
It's doable. Won't be easy, but doable. Get as much background in physics as you can during your undergraduate studies. You probably won't be able to take e.g. Quantum Field Theory, but it doesn't sound like you're aiming for these theoretical branches of astrophysics anyway.
Example from the MIT Physics PhD webpage:

Must I have a degree in physics in order to apply to this graduate program?
Prior to starting their Physics doctoral studies here, our successful applicants generally hold a Bachelor of Science degree in Physics or have taken many Physics classes, if they have majored in another discipline. The most common other majors are astronomy, engineering, mathematics and chemistry. Bachelor of Science degrees may be 3-year or 4-year degrees, depending on the education structure of the country they are in.