The (Cohen) kappa metric used to rank models in the challenge was first described in 1960 by Jacob Cohen. Kappa was introduced as a coefficient of agreement for the voting of multiple judges for nominal, though non-ordered, categories. It differs from the simple summation of identical votes by respecting the amount of agreement that is expected to occur by chance. In 1968, Jacob Cohen extended the theory and described the weighted kappa coefficient, which takes the magnitude of disagreement into account. The Quadratic Weighted Kappa (QWK) metric used for the challenges works with a quadratic distance function between categories to enhance the sanction of higher disagreements (i.e., low vs higher accuracy). Like all kappa coefficients, it ranges from +1 for an exact agreement throughout all elements and -1, which would indicate a systematic disagreement. A value of zero shows an alignment, which would occur at random considering the categorical frequencies.