Content type with multiple users - create a view of all users that relate to content of that type
I'm not entirely positive this is the type of relationship you want for this data.  If you use a node_reference field on the user rather than a user_reference on the node it would be pretty straightforward to create a view of Users who party.

Query all users and then add a Relationship to the field_party node reference.  Require the relationship with a Delta of 1.



This type of reference would seem to make more sense for your use case as well, since this would enable any user to administer their own party affiliations rather than the party manage the list (but then this is an analogy and I don't know the particulars).

Either way you are hitting up against the boundaries of what is possible using references.  I'd encourage you to take a look at the Relation Module which is designed to handle these kinds of bi-directional relationships, e.g. Display the members of the Party, Display the parties an individual is a member.