3 years into my phd program, want to shift to theoretical physics
You can find many areas in microelectronics which are closely related to the condensed matter physics (i.e: Electronic band structure, semiconductor, Conductors,  Superconductor, Ferroelectric, etc). My recommendation is to find a specific field in condensed matter that is also interesting for your supervisors, and pursue your PhD.

In your case, moving to another department is just burning two years.