will lack of experience make it impossible to change my subject of study?
Age won't matter much. 

"Experience" may matter. Any program has prerequisites, and there's really no way to jump to PhD without that essential knowledge.

I don't think you're going to get into the PhD program without first getting a Masters or having equivalent work experience. The Physics background plus independent study may be able get you into a Masters program.

I suspect you'll have better odds if you do the Masters in your own country, and then apply for a PhD program (in the US or elsewhere).