Switch to Physics from CS Career
Yes you can leverage.  At my university, these people work in the EECS department.  You could just apply to be a graduate student.
Study vector calculus and linear algebra.
Not really.  However, "big data" is quite popular in astro/astrophysics currently, so maybe you could get hired as a programmer.