Human ranked/sorted items I'm trying to derive an algorithm or procedure I would use to have a human interactively rank a series of items based on importance by comparing a random-pairing of items at a time.

So, given an array of let's say 10 "things", I want to present two of the things, and have the user specify which is the more important of the two.  What I don't know is:


What is the proper name for this procedure/concept?
Would I use some form of sorting algorithm?
How could I determine when to stop?  When is the ranking "done"?
And, importantly, I would prefer not to present the same "thing" to the user in a row.  That is, if comparing Item B and Item K, user selects Item K as more important, the next iteration should not present either Item B or Item K.


What I'm after seems similar to this question: Nearest neighbour based on subjective human comparison - is this a thing?

... but I'm not interested in similarity, just relative importance.  Think prioritizing a to-do list or subjectively ranking vacation destinations, etc.

The Elo Rating System may be similar to what I'm after, as well.
There are statistical models for relative ranking.  I'd suggest taking a look at the Bradley–Terry model.  The data is sometimes known as paired comparison data.

See also Analysis and classification based on data points,  Peer grading design - choosing a graph, to get accurate rankings/ratings, https://stats.stackexchange.com/q/83005/2921.