Co-organizing a workshop, does diversity matter?
EDIT: To add after an edit focussed the question:

The convener line-up will be important to attract people to the workshop so getting good names is not a bad idea as a starting point. Having conveners from several institutes may be beneficial since it shows prospective participants that the workshop has wider support. It may also mean that the advertisement can be improved, particularly if conveners come from different continents. Such signals should not be udnerestimated but will highly depend on the field and where research is carried out (geographically). There is of course nothing wrong with having conveners from only one institute. In the end it will be other factors such as convenience and interest that will determine the line-up of conveners. 

Answer to the original question as it was interpreted: 

The first question you should ask yourself is what is the goal or purpose of the workshop. I am used to workshops focussing on specific questions, perhaps leading up to common paper or document detailing  a problem formulation or providing recommendations for the future. In any case, the persons to invite would be the most suited to provide input to the goal. I have run very small workshops where I could only accept one person from any department/research group due to limited physical space. In other words, getting representation was more important than getting certain names in.

So depending on the goals you have you will probably see how you distribute invitations. If you have the space and time then it may be open but that may not be the most efficient way and invitation only may be the way out. That requires careful thinking so that you can achieve the set goal(s).