Transitioning from Electrical Engineering to Applied Physics/Material Science
I won't comment on Europe.  The difference between a PhD in electrical engineering, applied physics, or solid state physics is often very small.  There is no reason not to try switching between those fields when you start your PhD.
To successfully apply for a PhD program, you should fill gaps in your knowledge.  You will likely need to obtain a good Physics GRE score and pass qualifying exams for programs in the US.  Exact practices vary, but you'll find most students have gaps in their knowledge that they need to fill.  Also, many of the students will have forgotten quite a bit of that material before they finish their PhDs.