Are current AI models sufficient to achieve Artificial General Intelligence?
I am focusing on what you have posted here without going through (or having read) the whole essay you linked:

In the classic ‘brain in a vat’ thought experiment, the brain, when temporarily disconnected from its input and output channels, is thinking, feeling, creating explanations — it has all the cognitive attributes of an AGI. So the relevant attributes of an AGI program do not consist only of the relationships between its inputs and outputs.

The classic brain in a vat experiment does not disconnect the brain from all inputs and outputs but replaces &quot;real&quot; connections with its environment with &quot;fake&quot; connections, e.g. by connection the brain to a computer. This is what Wikipedia says:

In philosophy, the brain in a vat (BIV; alternately known as brain in a jar) is a scenario used in a variety of thought experiments intended to draw out certain features of human conceptions of knowledge, reality, truth, mind, consciousness, and meaning. It is an updated version of René Descartes's evil demon thought experiment originated by Gilbert Harman.1 Common to many science fiction stories, it outlines a scenario in which a mad scientist, machine, or other entity might remove a person's brain from the body, suspend it in a vat of life-sustaining liquid, and connect its neurons by wires to a supercomputer which would provide it with electrical impulses identical to those the brain normally receives.2 According to such stories, the computer would then be simulating reality (including appropriate responses to the brain's own output) and the &quot;disembodied&quot; brain would continue to have perfectly normal conscious experiences, such as those of a person with an embodied brain, without these being related to objects or events in the real world.

Following that, the argument of the author referring to the brain in a vat scenario does not hold.
If you ignore that problem for a moment the next inconsistency arises: The author assumed that because a brain would act &quot;so and so&quot; an AGI would need to act &quot;so and so&quot; as well. That however is not in line with how artificial intelligence is usually defined. Even if you consider the broad range of definitions:

(source: &quot;Artificial Intelligence: A Modern Approach&quot;; Russel, Norvig, 3rd Ed, 2010)
None of these definitions directly references AI to the brain. Therefore, the assumption that an AGI (as a subset of AI) would need to act like a brain in a vat is, without further assumptions,  generally flawed.
Another problem with the author's argumentation lies here:

to address some outstanding problem in theoretical physics — say the nature of Dark Matter — with a new explanation that is plausible and rigorous enough to meet the criteria for publication in an academic journal.
Such a program would presumably be an AGI (and then some).

The assumption that &quot;creativity&quot; (in quotation marks since we actually need to precisely define that in the first place) requires an AGI does not hold either. Let's stick to the authors example related to dark matter:

This is per definition a specialized field of application of an AI. Accordingly, there is not necessarily a need for an AGI since &quot;specialization&quot; is exactly what an AGI does not have.
Moreover, you could make an argument that coming up with new explanations related to dark matter is somewhat related to automated theorem proving. Accordingly, we might already have AI which is, in principle, capable of &quot;solving&quot; this task as of today.