Picking referees for a staff review?
tl;dr: You want a letter from God.  Until then, do not work with God.

Reference letters for promotion and tenure cases are supposed to be impartial evaluations of the quality, importance, visibility, and impact of the candidate's research, their standing in the research community, and their promise of future intellectual leadership (or in the case of full professor promotions, their proven intellectual leadership.)  So the people who write those letters must be credible independent judges of those qualities.

Here are the rough guidelines applied at my university.  Specific standards my be different elsewhere, especially outside the US — Ask your senior faculty mentors, department chair, and/or dean! — but I suspect the general principles are common everywhere.


Referees must be familiar with the candidate's research.  Promotion letters must discuss the importance and impact of the candidate's research in detail.  It is especially helpful if the candidate's work led to a breakthrough in the referee's research.  (Obviously, you can't dictate the letter; you have to choose a referee that can write such a letter.)  It doesn't matter whether the referee knows the candidate personally.
Referees must be experts in the candidate's research area.  For successful cases, this criterion should immediately imply the first.  However, "research area" must not be defined so narrowly that there are only a dozen experts in the entire world, or the impartiality of the referees is not credible.  Editors-in-chief of major journals, committee chairs of important conferences, and officers in relevant professional societies are all good choices, but for the most part, the expertise should be visible from the language of the letter. 
Referees must be obvious experts in the candidate's field.  Here by "field" I roughly mean "department".  The letter-writers will be scrutinized by faculty committees at several levels (department, college, and campus), most of whom don't have the expertise to make a direct judgement; they have to rely on the trappings of success instead.  At my university, it is a de facto requirement that all promotion references must be full professors (preferably with chaired positions) at top departments, in top colleges, in top universities.  Professional society fellows and major award winners are even better.
Referees must be obviously impartial.  Thesis and postdoc advisors are right out.  At my university, any faculty member from the candidate's PhD department is barred, but this is not universal.  Frequent and/or recent coauthors/coPIs are out.  It's okay to have a few letters from former collaborators, as long as the coauthored work is far enough in the past and represents a small fraction of both the candidate's and the referee's research record.
Referees must be familiar with the promotion standards of the candidate's institution. At my university, this means that almost all letters must come from faculty at top American universities.  I was explicitly warned not to ask for letters from European researchers, because they tend to write blunter ("more honest") evaluations than Americans, and in any case, European promotion systems are different.  This is another reason for needing letters from top institutions; a supportive letter from a significantly lower-ranked school will likely be viewed with suspicion, because the writer "doesn't understand our standards".


Multidisciplinary research makes this much more difficult.  The strategy I would recommend (and have applied myself) is to find referees that cover most of the different disciplines of the candidate's research.  To pick a random example, a candidate studying molecular simulations might request letters from other molecular simulators, from other computational scientists, and from polymer chemists.  This may require a higher than usual number of letters.

An additional complication at most US universities is that a majority of the referees must be chosen by the candidate's department, without the candidate's knowledge or direct input.