Gaia: A Benchmark for General AI Assistants
Paper page - GAIA: a benchmark for General AI Assistants


Abstract
	We introduce GAIA, a benchmark for General AI Assistants that, if solved,
would represent a milestone in AI research. GAIA proposes real-world questions
that require a set of fundamental abilities such as reasoning, multi-modality
handling, web browsing, and generally tool-use proficiency. GAIA questions are
conceptually simple for humans yet challenging for most advanced AIs: we show
that human respondents obtain 92\% vs. 15\% for GPT-4 equipped with plugins.
This notable performance disparity contrasts with the recent trend of LLMs
outperforming humans on tasks requiring professional skills in e.g. law or
chemistry. GAIA's philosophy departs from the current trend in AI benchmarks
suggesting to target tasks that are ever more difficult for humans. We posit
that the advent of Artificial General Intelligence (AGI) hinges on a system's
capability to exhibit similar robustness as the average human does on such
questions. Using GAIA's methodology, we devise 466 questions and their answer.
We release our questions while retaining answers to 300 of them to power a
leader-board available at https://huggingface.co/gaia-benchmark.

View arXiv page
	View PDF
	
			Add to collection
		



Community

	





		

			
				Models citing this paper
				0
			No model linking this paper
				Cite arxiv.org/abs/2311.12983 in a model README.md to link it from this page.
				

			
				Datasets citing this paper
				1
			
				gaia-benchmark/GAIA
				
			
			Viewer
			• 
	
				Updated
					Mar 26
				• 
			932
	
				• 
					1.09k
				
				• 
					153

				
		
				

			
				Spaces citing this paper
				1

			

			
				Collections including this paper
				73
			








					Browse 73 collections that include this paper
						