Choosing PhD advisor in a field different from your undergrad research?
You are talking about what you did as an undergraduate. When talking to a potential supervisor, you have to remember that you are still going to be a student. You will be learning things. A supervisor will see your previous research and your potential for additional research.

As an undergraduate, I'm pretty sure that you did not become a world expert in astrophysics. So it's not like that you will have "wasted" knowledge, now that you will switch to a different topic.

Also, it's not like you're changing from biology to physics. You are staying within the realm of physics, and any skills that you gained doing your ("unrelated") undergraduate research will only benefit you when doing your graduate research.