About recommendation letters
Usually a PI would have tons of students asking for recommendation letters


...which is why the experienced ones will say no to anyone they don't think they can honestly give a strong recommendation.


  it's still a big favor.


No, it's not.  It's their job.


  How is a great letter composed?


Most successful reference letters follow the same general outline:


A short blurb describing how long the writer has known you, and in what capacity (student in class, independent study, senior thesis, lab slave, coauthor, squash partner, etc.)
A detailed description of why (not just whether) the writer thinks you are a strong candidate for PhD admission, and in particular your potential for independent research in direct, personal, technical, and credible detail.  This part is different for every letter.
Direct comparisons, by name, with other students the writer has worked with and/or recommended in the past.
A short blurb describing the writer's credentials and experience.



  Are the writers really aware of what they have to write?


In general, yes—after all, they read recommendation letters themselves (or they're the wrong people to ask)—but the only way to really be sure is to ask them directly.  If they are anything but confident about their ability to write you a strong and effective letter, ask someone else.


  is the same letter suitable for two PhD programs?


At least within the US, yes, definitely.


  What if they're in different countries?


Ask your references directly whether they understand the cultural expectations in those countries.  But in my experience (in computer science), these differences are shrinking rapidly.


  Then, should a pre-PhD student work in both continents first and then apply?


Nobody can tell you what you "should" do.  There's an obvious tradeoff between breadth and depth in your pre-PhD research experience.  The choice depends on which is more likely to provide evidence of your potential as an independent researcher.


  How to deal with the fact that the PIs may be unknown?


There is no such thing as an unknown PI; everyone has a professional web page, and everyone can use Google.  (Corollary: If someone doesn't have a professional web page, do not under any circumstances ask them for a letter.)  But more junior writers do need to provide more narrative detail in their letters, to make up for being less known and less experienced.


  How to deal with the fact that you need many different PIs writing you many different letters?


What is there to "deal with"?  With rare exceptions, you need three our four PIs to write one letter each, possibly with some very minor customization.  Ask them if they are willing to write you a strong letter.  (Use the word "strong" when you ask.)  If they say yes, give them everything they need to write a strong letter, including time, and then get out of the way.


  How to ask your PI a recommendation letter to "leave" him/her?


Directly—just as if the PI were actually a responsible, mature, adult human being—and far enough in advance that they can plan for your departure.  Anything else would be incredibly disrespectful.

(If they get huffy about the idea that you might pursue opportunities elsewhere, then be very happy you asked—you really don't want to work for them.)