Which correlation test can be used on low count contigency table?
I don't think it's worth doing any kind of significance test for such a small sample.

option 1: pure categorical values

You could just count the number of identical ratings between the two ratings and divide by the total number of players

option 2: take the order into account: weak &lt; moderate &lt; strong

Define a similarity measure such as:


sim(X, X) = 1
sim(weak,moderate) = 0.5
sim(moderate,strong) = 0.5
sim(weak,strong) = 0


Then use the same process as option 1 except that for every player you add sim(ratingA, ratingB), and divide by the total. This will also give you a normalized score between 0 and 1 which is a bit more precise (the value will be higher or equal than in option 1).