Is the AI Effect caused by bad tests of intelligence?
I think it is mostly right. But not that intelligence is hard to define. In my opinion, it is simple: A is more intelligent than B if A achieves some purpose in less steps than B. It is functional/algorithmic efficiency.

What is difficult to define is human intelligence.

But when someone says, "No, X is not real intelligence," what they mean is that it does not satisfy what we would consider real human intelligence.

So when people discount new and amazing discoveries in machine intelligence, it is not because they are not amazing in their own way, but because those discoveries, while exhibiting intelligence, are actually not replicating human intelligence - which is what many people actually mean when they say "that thing isn't really intelligent."

In truth we are very far, in the science of artificial intelligence, algorithmically speaking, from an artificial human intelligence (AHI).

Additional note: What's funny is that we don't call the science of artificial intelligence just 'the science of intelligence.' That we add the "artificial" qualifier by necessity pegs the science to what the artificiality implicitly emulates: human intelligence. In other words, "artificial intelligence" must be by definition more specific to the thing it allegedly artificializes than a more general science of just "intelligence."