Bibbity-Bobbity Sonnet lives rent free on my property
(I don't accept an Anthropic defeat on AidanBench)

So guys, you see this? This means Sonnet is the GOAT.

Here's what I did:

Let's say we don't care about peak-performance and outliers but on consistency.

So instead of rewarding models like GPT-4.5 for getting 1000 points on a single question (1/3rd of it's score from a total of 63 questions) because it memorized all non-poisonous recipees, we just look at the ordering of the models.

The model with the highest number of answers on a given question is awarded with "NumberOfLLMs" points. So if there are 10 points, rank 1 get's 10 points.
You do this for each question and end up with the plot above. This is called Borda Count and is a ranked voting system.

Let's look at the individual questions in this matrix:

As you can see, o1 is by far the model with most number 1 spots which means its peak performance is super high.

But overall, Sonnet 3.5 (new) clutches, because it's consistently among the top models.

Now don't get me wrong. A score of +1000 on a single question by GPT-4.5 is impressive, as it requires creativity, good instruction following and long context understanding. But I don't think 1 question should be 1/3rd of the total score. Especially if that model never reaches rank 1 in any other question and this score is the largest overperformance of the entire dataset.

The average performance has to be considered too.
Otherwise it just screams memorization.

Surely, it's possible to fine-tune a model that it excels in answering one question.
So I think a "sensitivity analysis" is warranted.