Is it worth it for me to try at this point?
I think this is a simple answer.

Stop thinking and start doing.

If you want a graduate degree at an Ivy League school and a successful academic career (perhaps even in the field of astrophysics), then pull your socks up, set definite, measurable and realistic goals and start achieving them. The path will not be easy but always remember that you are not the first and neither will you be the last.

Note that a graduate degree at an Ivy League school and a successful academic career are not necessarily mutually inclusive. To do one does not mean that you will do the other and neither does it mean that there is a causal relationship between the two.

Personal anecdote:

I had a rather forgettable start to my academic career as well. I recovered enough to get myself admitted to an "Ivy League" graduate program. I am still on my journey and I dare say the destination is years away. Not everyone is a rising star. Some folks are late bloomers. 

I reiterate again. To achieve your dreams, just keep trying sincerely and honestly. It will happen in due time.