How to apply to a prestigious American university as an international student?
I am planning to do a Ph.D. in theoretical physics in the US, more specifically California


Many American prospective graduate students limit their search with severe geographic restrictions.  I view this as being a poor investment in a future academic career, but if someone truly prioritizes being an easy drive from their family and childhood friends, that's certainly up to them.  If you are coming from abroad it makes no sense to me to concentrate your search in "California".  California has some excellent institutions, but certainly only a minority of the nation's best institutions are in California.  Moreover, if by chance you have family living in, say, Los Angeles, going to graduate school in San Francisco is not going to put you close enough to them to be helpful to you.


  I understand that an American Ph.D. program is very different from a European one in that it usually takes around 5 years to complete and involves coursework.


First of all, American PhD programs can take less than five years to complete -- though it is quite rare to see a student graduate in less than four -- and also more than five years to complete.  In my field (mathematics) the time to completion is certainly less than the overall average, but I think there are very few, if any, math PHD programs where the average time to completion is 5.0 years or less.  (I went to one of the top programs in the country.  I did graduate in five years, which was most common in my program.  I think though that more students took more than five years than less than five years.)  


  If a master's degree was already obtained, do American universities allow for a more European style Ph.D. completed in around 3 years?


I think you may be misunderstanding the system slightly.  There are (virtually) no length requirements in American PhD programs.  You probably do have to complete a certain amount of coursework, but this is just what you would automatically get by spending 2-3 years in the program.  In most cases you can get "course credit" for your thesis work, so the amount of actual courses that PhD students take varies considerably and to a large degree is up to them. 

Entering a PhD program with a master's degree makes much less of a difference in an American PhD program than you might think, and the better the program the less it matters.  For instance, I had a sort of "secret master's degree" in that I got it along with my bachelor's degree.  I don't think having this master's degree impacted my studies at all (except of course to convey that I had roughly equivalent training to the excellent foreign students who were admitted to the program).  At a certain point in our PhD program we "earned a master's degree" and could, with a little paperwork and a small fee, formally collect it.  I seem to recall a few people did that, perhaps to get a leg up on some summer job.  I already had one; I could have gotten another, but what PhD student needs two master's degrees in the same field?

My present PhD program (mathematics, UGA) is around the 50th best in the country, and things work differently.  Up until a few years ago having a master's degree was actually a formal disadvantage to our entering students in that it conferred one year less of guaranteed PhD funding.  But we observed that students with entering master's degrees were not in general better prepared than students without, so we rolled back that requirement a few years ago.  In this regard let me say: in the American system a master's degree is kind of a "no man's land": we have absolutely no agreed upon notion of what skills and work a master's degree should convey.  If you get a master's degree from a low quality institution, you are probably less prepared in every way than if you get a bachelor's degree at a solid institution.  Conversely, a master's degree at a pretty good institution is worth something: of my first four PhD students, they only one who graduated in five years came in with a master's degree from a good place.  This counted for something: in fact he started working with me in his first year in the program, whereas many PhD students spend a year or two or three entirely focused on coursework and exams with no research in sight.  


  Does a Ph.D. necessarily involve paying the university's tuition fees?


Nooooooo.  As a good rule of thumb, if you are in a STEM field, you should not enroll in a PhD program unless your fees are being paid.  


  Ph.D. students in Europe are employed by their university, receive a monthly salary and don't pay tuition. Does this model exist in the US, i.e. is there any way to avoid paying tuition fees as a Ph.D. student?


Yes!  Most PhD students in STEM fields not only do not pay tuition, but also receive a livable stipend (which is some combination of money for teaching, for research assistance and outright scholarship money).  There are some circumstances in which one might want to consider enrolling in a STEM PhD program without such a stipend, but they are the exception, and you should think very carefully before agreeing to this.