Consensus algorithm for imperfect annotators
Technically speaking, you can't.  If you flip all of the oracle outputs, then it maintains the same level of agreement but reverses the ranking.
But this is probably a trivial case that's unimportant in practice.  Suppose we can assume that every oracle has accuracy that's at least 50%.  Then one approach would be to take all questions that are rated by at least three oracles, take the majority vote among all oracles that rated it, and then measure agreement with that majority.  I think that should give a correct ordering of the oracles, given enough data (i.e., in the limit, as the number of data points goes to infinity, it should converge to the correct ordering).
In practice you could consider using a more sophisticated model that captures both differences in the accuracy of each oracle and differences in the difficulty of each question.   I'm not sure whether this will lead to a major difference in outcomes significantly, but there is tons of work in the research literature you could read about to see what methods have been studied.  Here are some example papers:

Online crowdsourcing: rating annotators and obtaining cost-effective labels, Welinder et al.

Lean crowdsourcing: Combining humans and machines in an online system, Branson et al.

Get another label? improving data quality and data mining using multiple, noisy labelers, Sheng et al.

Max-Margin Majority Voting for Learning from Crowds, Tian et al.

Active visual recognition with expertise estimation in crowdsourcing, Long et al.


and probably many more - I'm not expert on this topic, so I have no idea whether these are the most important or useful papers, so I suggest doing a literature search of your own.