A3 is a new evaluation platform for testing mobile GUI agents with real-world apps, offering automated evaluation and expanded action space compatibility.

🎯 Original Problem:

Existing mobile GUI agent evaluation platforms focus on static frame tests or limited app scenarios, failing to assess real-world performance effectively.

💡 Solution in this Paper:

→ A3 integrates with 21 popular third-party apps and 201 real-world tasks spanning operations and information queries

→ The platform implements a flexible action space compatible with any dataset annotation style

→ Tasks are categorized into operation, single-frame query, and multi-frame query types

→ A3 introduces both task-specific evaluation functions and an LLM-based evaluation system

→ The automated evaluation process uses GPT-4 and Gemini 1.5 Pro for cross-validation

🔑 Key Insights:

→ Static frame evaluations don't reflect real-world challenges

→ Action history significantly impacts agent performance

→ Self-correction capability is crucial for real-world scenarios

📊 Results:

→ LLM evaluation achieves ~80% accuracy

→ Cross-validation reduces misjudgment to ~3%

→ AppAgent outperforms both finetuned models and GPT-4

→ Success rates: 30.8% (easy tasks), 7% (medium tasks), 2% (hard tasks)