Should I apply to PhDs in the US (theoretical physics)?
I think that the only issue here is that you intend to change fields. You will need to show a US institution that you have adequate background in needed physics topics to begin advanced study. But you may be fine there also, depending on your course of study. 

The reason that the US degree takes longer is that it makes fewer assumptions about the candidate's background on entry. A doctoral program (here) normally requires some advanced coursework leading to comprehensive (qualifying) examinations. So, there is some opportunity to make up for missed subjects, provided that you can convince a department to take you. Letters of recommendation might be essential, especially if they can speak to your readiness to switch fields. 

And, you might be able to get into a top program (depending), since the entrance requirements are a bit less than for Oxbridge, which has a different educational system. In particular, an undergraduate degree in UK is normally more specialized than one in US, whereas most students here can enter doctoral programs with only that less specialized undergraduate degree. 



Following your edits, I can only add that the decision about whether it is "worth it" is personal and can only be answered given your own preferences. Weigh the plus and minus aspects of each decision as you would for any important one. You are unlikely to suffer any major setback to your career no matter which way you jump.