GeoLLM-Engine: A Realistic Environment for Building Geospatial Copilots
Simranjit Singh Michael Fore Dimitrios Stamoulis




[
A New Parameterization for Finding Solutions for Microlensing Exoplanet Light Curves
    Jiyuan Zhang
    April 29, 2024
====================================================================================

type=figure

    < g r a p h i c s >

figure offers a realistic web environment equipped with comprehensive APIs for developing, deploying, and evaluating geospatial Copilots on tasks that authentically reflect the workflows of remote sensing analysts. Unlike existing LLM benchmarks which rely on simplified question-answer pairs,  introduces nuanced, long-horizon natural-language commands over a wide range of tasks - from object detection and land classification to document retrieval and web search.


]




A New Parameterization for Finding Solutions for Microlensing Exoplanet Light Curves
    Jiyuan Zhang
    April 29, 2024
====================================================================================




Geospatial Copilots unlock unprecedented potential for performing Earth Observation (EO) applications through natural language instructions. However, existing agents rely on overly simplified single tasks and template-based prompts, creating a disconnect with real-world scenarios. In this work, we present , an environment for tool-augmented agents with intricate tasks routinely executed by analysts on remote sensing platforms. We enrich our environment with geospatial API tools, dynamic maps/UIs, and external multimodal knowledge bases to properly gauge an agent's proficiency in interpreting realistic high-level natural language commands and its functional correctness in task completions. By alleviating overheads typically associated with human-in-the-loop benchmark curation, we harness our massively parallel engine across 100 GPT-4-Turbo nodes, scaling to over half a million diverse multi-tool tasks and across 1.1 million satellite images. By moving beyond traditional single-task image-caption paradigms, we investigate state-of-the-art agents and prompting techniques against long-horizon prompts.
    


§ INTRODUCTION



With the advent of generative AI, Large Language Models (LLMs) have the potential to significantly enhance Earth Observation (EO) workflows <cit.> across a broad range of tasks, from detection to learning from spatio-temporal data to analyzing aerial, UAV, and satellite images and videos <cit.>. However, existing approaches consider predefined low-level template-based prompts that only capture the textual surface form of the predicted image-caption pairs <cit.>, while often overlooking the functional agent correctness at completing high-level natural language (NL) commands <cit.>. While the need for more representative benchmarks has been underscored across several generative AI domains <cit.>, their significance is even greater in the geospatial domain, as it involves complex multimodal data across diverse spatial and temporal dimensions.

Such disconnect partly stems from prevailing perceptions that benchmark creation, focused on simplistic single-task prompts, is straightforward. However, the overhead extends beyond merely curating benchmarks with image-caption pairs, a task that can be programmatically accomplished, as evidenced by the rapid release of numerous geospatial benchmarks over recent months <cit.>. Instead, the challenge lies in establishing an environment equipped with the requisite tools, dynamic UIs, and real-world APIs to form the “engine” for developing complex tasks. In this work, our key insight is that amidst the abundance of “geospatial benchmarking” works, a subtle refocus is necessary, prioritizing the construction of a robust engine as the foundation for benchmark creation, rather than the benchmarks themselves. 

We draw inspiration from novel work <cit.> that introduces environment-based benchmarking suites for comprehensive agent assessment. While these works highlight potential in their domains, adopting them for geospatial applications necessitates overcoming human-in-the-loop bottlenecks, particularly in manual ground-truth verification and template creation. To mitigate these challenges, we employ formal-language-based verification techniques <cit.>, recently introduced to expedite labor-intensive Reinforcement Learning from Human Feedback (RLHF) workflows. 

In this work, we introduce , a highly realistic environment that captures real-world tasks on EO platforms <cit.>. Our environment comprises various fully operational APIs and dynamic map/web UIs to execute geospatial tasks via high-level NL prompts. More importantly, we employ model-correctness checker techniques <cit.> that allow our “back-end” engine to autonomously verify the accuracy of generated benchmarks, requiring only a one-off initial validation of task templates. By reducing the necessity for human intervention, we can massively parallelize our benchmark suite across 100 GPT-4-Turbo nodes to create large-scale benchmarks with 100,000 prompts that span half a million tasks over 1.1 million images from open-source EO datasets.

Each  prompt exhibits high-level intent that emulates the nuances and abstract language usage patterns typically employed by human operators, as shown in <ref>. Using this benchmark, we follow state-of-the-art evaluation schemes with tool-augmented fine-tuning-free agents in zero-/few-shot in-context learning modality <cit.>, powered by the latest GPT-3.5 and GPT-4  versions. Capturing our key insight, our findings reveal that merely expanding LLM benchmarks with more tasks of uniform complexity (e.g., an excess of visual QA captioning tasks) does not significantly enrich our understanding, as agent performance predictably shows little variation. Conversely, we show that varying levels of complexity better assess agent performance. To this end, we diversify the scope of our benchmark by incorporating various remote sensing (RS) applications over different satellite imagery sources and tasks of escalating intents, ranging from document knowledge retrieval to UI/Web interactions to geospatial data analytics.





§  ENVIRONMENT



Our aim is to establish a realistic environment that clearly advances beyond current benchmarks, featuring a self-contained web UI with a varied array of LLM tools and an integrated benchmarking engine (<ref>). Built upon open-source libraries and APIs,  is designed as a reproducible and scalable platform to support the development and evaluation of geospatial agents.

Environment - “Front-end”:
A key challenge in creating such an environment is the need for reproducibility and comparability across different systems and methodologies. To address this, we leverage a suite of open-source APIs, enabling the seamless integration of a wide array of tools, datasets, and functionalities, while facilitating transparency and accessibility. We intend to release our codebase and benchmark to foster advancements in geospatial Copilots.



Tool Space:
We equip our environment with a comprehensive array of open-source Python packages, catering to various functionalities from data analytics to LLM-specific tasks, such as employing  for FAISS <cit.> embeddings in knowledge retrieval applications. The interface leverages  APIs for interactive mapping, while  and  facilitate advanced manipulation of geospatial data. The complete module inventory, comprising over 175 tools, is presented in <ref>. This diverse toolkit enables us to execute complex tasks ranging from satellite imagery analysis to utilizing vector storage for rapid geographic data querying. 


Engine - “Back-end”:
To facilitate large-scale experimentation,  incorporates a Command-Line Interface (CLI) alongside its UI, providing comprehensive tooling and scripting capabilities. This feature is crucial for conducting extensive investigations, allowing us to efficiently run model-verification checks on over half a million tasks and evaluate baseline agents against our benchmarks. Thanks to this setup, these operations are completed within hours, leveraging hundreds of GPT endpoints. The CLI, in tandem with the UI, provides diverse modalities essential for replicating the intricate demands of geospatial analysis tasks which require the integration of disparate data sources and analytical techniques.


Environment Formulation:
Our environment is represented as ℰ=⟨𝒮, 𝒜, 𝒯⟩, comprising the environment state 𝒮, the action space 𝒜, and the tool space 𝒯 (<ref>). To intuitively understand 𝒮, consider a user query “zoom into the Indo-Pacific region and show me the vessels during November 2021.” Upon query completion, 𝒮 will encapsulate not just the visual or textual response, but also the altered state of the map, such as its zoom level or position, the loaded database, and the temporal window of displayed data.

While detailed notation of transition functions extends beyond our study's scope, previous research <cit.> has established these functions as deterministic. This implies a critical property: given the same starting state 𝒮_0, any two agents executing an identical sequence of actions or tools will invariably arrive at the same final state 𝒮^*. This deterministic nature is vital for our purposes, as it allows for the verifiability of any agent's solution against the benchmark's “ground truth” specifications <cit.>. We exploit this feature in our model checker (elaborated in <ref>) in two principal ways. Firstly, given a “golden” ground truth, we can deterministically assess the functional correctness of any candidate solution that follows the same sequence of actions or tools. Secondly, this principle underpins our approach to benchmark (ground-truth) generation: by evaluating multiple agent solutions, a consensus on the final state among the majority indicates a high likelihood of an accurate ground truth, hence eliminating the need for human inspection.

User Intent Formulation:
Intuitively, each user intent is encapsulated by four parts: the question q that triggers the agent, the executed tool sequence T̂, the agent's textual response r̂ to the user, and the concluding environment state Ŝ. Thus, we can express each task as {q, T̂, r̂, Ŝ}. The sequence T̂ is defined by the set of tool T̂ = {t̂_1, t̂_2, …}, where at each step i the agent invokes tool t̂_i= {t̂ôôl̂_i, âr̂ĝŝ_i}∈𝒯. As shown later in <ref>, by contrasting the task set {q, T̂, r̂, Ŝ} with a gold standard {q, T^*, r^*, S^*}, we can ascertain the functional correctness across our entire benchmark.



§  BENCHMARK SUITE







In this section, we first describe the process for “grounding” high-level natural language instructions into a structured set of task templates and user intents that cover all the  tools. Next, we discuss our GPT-driven, human-out-of-the-loop  ground-truth sampling approach.

Intent Collection: 
The initial phase involves utilizing human annotators (our team members) to craft a small set of user intents, following the qualitative guidelines set forth in <cit.>. These intents are designed to be both nuanced and high-level, requiring the agent to perform more than one or two actions, and should be decomposable into a series of interchangeable templates. While this step is manual, please note that it represents the primary (and essentially the only) offline step necessitating human involvement. Through this procedure, we generate the foundational intents and templates, serving as the modular components from which benchmark queries are constructed.

The annotators are instructed to input detailed prompts, utilizing zero-shot GPT-4-Turbo with Chain-of-Thought prompting <cit.> to propose solutions (as highlighted in our Results, GPT-4 demonstrates notable zero-shot capabilities). Utilizing “User Feedback” UI buttons, annotators identify and confirm correctly executed model responses, hence collecting the “correct” examples along with the corresponding agent actions. By promoting scenarios that necessitate an average of 7-8 tool interactions, we can cover the entire tool space within 250 instantiated queries. Following the granular intent categories as in <cit.>, we classify the various prompts into the categories shown in <ref>:


    
  * Information Seeking: Queries for knowledge retrieval aimed at sourcing information from wikis and documents to support EO investigations.
    
  * UI/Web Navigation: Commands designed for UI interaction, such as opening web search results or displaying images corresponding to detections mapped out.
    
  * Load-Filter-Plot: Operations to load data, apply specific filters, and present geospatial findings in an insightful manner, for example, through change-detection heatmaps or land cover classification (LCC) categories.






Tool Templates:
Given the 250 instantiated queries and GPT's solutions, we programmatically parse all  responses and remove duplicates, obtaining a distinct template for every tool agent-call, i.e., ∀ t_i ∈𝒯 we have the GPT -call {toolname_i, args_i}. These  responses furnish us with the “building blocks” that can be used for benchmark sampling in a straightforward intuition: leveraging the recently introduced  feature in OpenAI's APIs, we can initiate queries to a standalone GPT model. Within this context, we present the model with a specific tool interaction template and pose inquiries akin to: “Given your prior action of zooming into X and the corresponding function call, generate the new function call to instead focus on Vienna.” This allows us to dynamically generate tool-specific commands tailored to new, contextually relevant scenarios.


GPT-Driven Benchmark Creation:
To efficiently scale our benchmark generation, we capitalize on two key attributes of GPT-4-Turbo agents: the extended input token lengths and their ability to parse extensive databases described through SQL-like schemas. We leverage the expanded token capacity and we incorporate directly into the model's context all tool-calling  templates for the 174 tools (manually verified in the previous step). Moreover, we compile the metadata (e.g., coordinates, dates, categories, document titles, etc.) associated with all satellite images and documents into a SQL table. From this, we randomly select 1,000 entries, collecting their categories, coordinates, and dates sets and providing them as SQL schemas to GPT. Last, we append the 250 manually crafted queries as “successfully sampled” examples, and alongside in-context “benchmark creation” instructions, this entire prompt is fed to GPT-4, prompting it to autonomously generate both a suggested task prompt and the corresponding solution, forming a new query task {q, T^*, r^*, S^*}.

To circumvent the need for manual verification of each solution's accuracy, we introduce a novel approach that integrates functional model checking with the principle of LLM self-consistency <cit.>. Drawing from the concept where an agent repeatedly solves the same prompt, typically converging on the correct solution, we apply this by having GPT-4 propose multiple solutions to its self-generated (original) prompt. After 10 iterations, we execute these solutions in our engine (in-parallel via our CLI tools) to ascertain the final state S^* for each. If 9 out of 10 solutions converge on the same end state, we consider this to be a verified ground truth and incorporate it into our dataset. Overall, this novel self-consistency sampling scheme allows us to streamline the benchmark generation process without extensive human intervention.


Model-Checker Formulation:
 incorporates a rigorous set of model checks to determine both the functional correctness and overall success of an agent’s response to a given prompt. As shown in <ref>, correctness checks the agent's tool usage T̂ against the corresponding ground-truth T^*, assessing argument accuracy within function calls (e.g., missing a required tool or calling the right tool with wrong parameters). Success checks the final system state produced by the agent's sequence of actions Ŝ against the expected ground-truth state S^*. We leverage our engine to “run” both the sequences starting from the same initial state to confirm whether the agent's output and ground-truth match. Note the distinction between the two failure cases; they are not mutually inclusive. For example, an agent may erroneously invoke an unnecessary tool (“Function Error”), yet this may not alter the final state, which could still align with the anticipated result.


Agent Evaluation Metrics:
Based on the model checks, we can define the appropriate metrics that we subsequently use to evaluate the performance of different agents: 

    
  * Success rate: the ratio of successfully completed tasks across the entire benchmark as defined by f_success (<ref>). This ratio informs us of the degree to which the agent is able to complete tasks, irrespective of whether it took incorrect or unnecessary intermediate steps. 
    
  * Correctness rate: the ratio of correct function-call operations across the benchmark as defined by the f_correctness error types. Given the total number of errors and ground-truth tools, we compute the correctness ratio R_correct = max(0, 1- N_errors/N_tools) <cit.>, which captures how likely it is for the agent to invoke the correct functions in the expected order.
    
  * ROUGE score: ROUGE-L recall score <cit.> to compare final model replies r̂ with the ground truth r^*.









§ REMOTE SENSING DATASETS



We consider several open-source data sources encompassing tasks like object detection, land cover classification, and visual question answering to build a collective dataset of 1,149,612 images. All data sources have coordinates and time metadata which offer us global spatio-temporal coverage in our benchmark prompts (<ref>): 

    
  * xView1 <cit.>: 846 high-resolution images sourced from WorldView-3 satellites focusing on overhead object detection with 1 million objects and 60 classes. 
    
  * xView2 <cit.>: dataset for building damage assessment with 5,598 images sourced from the Maxar/DigitalGlobe Open Data Program before and after 19 natural disasters with 850,736 building annotations. 
    
  * xView3 <cit.>: a dataset of 23,432 SAR GRD images from the Sentinel-1 mission annotated for (fishing) vessel detections to study illegal fishing practices. 
    
  * SARFish <cit.>: extends the xView3-SAR GRD dataset by providing products from the Sentinel-1 C-band SAR satellite constellation operated by the European Space Agency’s (ESA) Copernicus Program in both real-valued GRD and complex-valued SLC product types. 
    
  * FAIR1M <cit.>: object detection dataset with 24,775 images with over 1 million instances sourced from Gaofen satellites and Google Earth.  
    
  * Functional Map of the World (FMoW) <cit.>: a multi-label LCC dataset for buildings/land use with 727,144 images from over 200 countries. 
    
  * BigEarthNet <cit.>: 344,385 Sentinel-2 LCC images from the CORINE Land Cover database.



§ RESULTS



<ref> summarizes the performance of state-of-the-art GPT-based agents with various prompting schemes. In addition to the LLM metrics, we report the agent's performance with respect to the underlying object detection (F1 score), LCC (Recall), and visual question answering (VQA, Rouge-L) tasks in our benchmark. These metrics provide insights into each agent's efficiency, accuracy, and responsiveness in executing geospatial tasks.

GPT versions:
We observe that there's a marked difference in performance between models based on the GPT-3.5 Turbo and GPT-4 Turbo frameworks, with the latter generally achieving higher scores across all metrics. For instance, both the Chain-of-Thought <cit.> (CoT) and ReAct <cit.> methods, especially in their few-shot configurations, demonstrate a significant leap in correctness and success rates when transitioning from GPT-3.5 to GPT-4. This improvement underscores the advancements in model understanding and task execution capabilities. This finding is consistent with most of the work in language guided agents <cit.>. As anticipated, the few-shot configurations outperform their zero-shot counterparts across all evaluated methods. This trend underscores the value of providing models with a few examples to adapt to specific tasks, significantly enhancing their ability to accurately interpret and respond to complex geospatial queries. 







Prompting Schemes: We observe that ReAct generally outperforms CoT for both GPT versions, as confirmed across several domains deploying ReAct for multimodal tasks <cit.>. Moreover, Chameleon's performance is notably lower than that of CoT and ReAct, especially evident with the GPT-4 model, indicating that while Chameleon is a valuable method, its prompting “decomposition” strategy which is primarily multi-step single-tool prompting makes it less suited for long-horizon (multi/single-step multi-tool) geospatial tasks. Please note that our focus is tool-augmented fine-tuning-free agents, so we leave the exploration of fine-tuned LLMs for future work.

Model Cost: An interesting observation is the lack of a clear correlation between the tokens consumed and the success or correctness rate. Despite Chameleon consuming the highest number of tokens on average, it does not lead in terms of success or correctness rates. This finding suggests token usage does not directly translate to higher performance, challenging the assumption that more extensive responses might yield better results. These insights collectively highlight the nuanced dynamics of model performance within the  benchmark.


Success Rate vs. Task Complexity: 
Delving into the relationship between success rate (SR) and task complexity provides further insights into agent performance. As shown in <ref>, there is an inverse relationship between SR and the number of tool calls required to answer a prompt. Specifically, note how for tasks requiring a single tool call, SR exceeds 95%, while for more complex tasks involving more than eight tool calls SR is below 70% and even dropping below 27% for certain agents. This finding highlights a critical limitation of current geospatial agents: while agents can handle simpler tasks with relative ease, their performance degrades as the task complexity increases. We emphasize the need for benchmarks that measure agents' real-world utility against complex geospatial scenarios.


Success Rate vs. Benchmark size:
Next, we conduct ablations varying dataset size from 500 to 10,000 tasks (<ref>): for all methods, the success rates remain relatively unchanged across this range. In a significant escalation of our testing regime, we scaled our  benchmark to 100,000 queries covering over half a million tool calls (<ref>). For this experiment, which is the largest in this domain to our knowledge, we leverage the massively parallel nature of our engine over 100 GPT endpoints. Despite the tenfold increase, the success rate of GPT-4 CoT showed relative stability, suggesting that merely increasing the benchmark size does not necessarily challenge the agents more or provide a better assessment of their capabilities. Instead, it is the task complexity within the benchmarks that presents a significant factor in evaluating geospatial performance. These findings critique the prevalent approach in recent works where the focus has been on scaling the benchmark size, while our findings highlight that increasing the complexity of tasks is more essential for assessing agent performance.




Success Rate vs. Error tolerance:
Last, we capture 's parameterizable nature that allows us to evaluate agents based on varying error thresholds by properly updating the ϵ in the model-checkers functions. The incremental increase in success rates with the relaxation of error thresholds reflect scenarios allowing for a certain margin of error or applications where absolute precision is often unattainable (e.g., nuanced definition of what a user means when asking “Show me all car detections in Madrid”; whether this assumes to include suburban areas or not might vary per user).







Correctness Rate: 
<ref> shows the error types for CoT and ReAct on GPT-4, in both zero-shot and few-shot scenarios. The most common error type is “Missed Function”, suggesting that the GPT-4 often omits necessary tool calls regardless of the approach used, accounting for more than half of all errors. These confirm recent findings that dynamic/RAG-augmented <cit.> prompting further improves agent performance by addressing such failures. Last, the consistent error distribution across different methods implies that these issues are not method-specific but rather inherent to the current capabilities of the underlying GPT-4.





§ RELATED WORK



Recent advancements in autonomous agents span from reinforcement learning platforms <cit.> to web-based applications and benchmarks for web interactions <cit.>. VisualWebArena <cit.> represents a milestone of LLM benchmarking, offering realistic tasks to evaluate multimodal web agents towards improving performance on complex web pages. The adoption of multimodal models for EO tasks is gaining momentum (<ref>). Innovations like SkyEyeGPT <cit.> and Remote Sensing ChatGPT <cit.> showcase advancements in integrating VQA agents and computer vision models with RS imagery for enhanced multimodal responses. However, existing benchmarks often rely on predefined, single-step text-image prompts.  allows us to assess agents' ability to execute nuanced EO tasks.


§ LIMITATIONS AND FUTURE WORK



We recognize limitations within our framework. First, using GPT-4 for both generating and evaluating ground truths could introduce bias risks, as highlighted by benchmarking work <cit.>. We are currently enhancing sampling diversity by leveraging hybrid strategies that incorporate both GPT-generated outputs and programmatic elements <cit.>. Second, while our platform emulates complex tasks reflective of EO analysts' workflows, we emphasized depth (long-horizon tasks) over breadth in task complexity. We are actively expanding the capabilities of our engine to incorporate a wider variety of tasks (e.g., from maritime traffic analysis <cit.> to illegal fishing <cit.> to damage assessment <cit.>) leveraging 's flexible APIs. Furthermore, we have focused on assessing finetuning-free agents, which is why  doesn't specify train-val-test splits. Following recent work on tuning LLMs on EO tasks <cit.>, we are expanding our benchmark to tool-agents training <cit.>. Last, we have utilized standard LLM APIs without explicitly optimizing for cost (e.g., latency or token usage). We are currently orthogonal optimizations to enhance  efficiency, such as LLM compilers <cit.>, dynamic tooling <cit.>, and token compression <cit.>.


§ CONCLUSION



In this work, we introduce , a novel environment for evaluating geospatial Copilots, designed to bridge the gap between simplistic benchmarks and the complex demands of Earth Observation (EO) applications. By leveraging a rich array of geospatial API tools, dynamic interfaces, and a massive parallel processing framework over 100 GPT-4-Turbo nodes, our environment facilitates the execution of over half a million multifaceted tasks across 1.1 million satellite images. This advancement not only highlights the limitations of existing benchmarks but also sets a new standard for the development and evaluation of AI agents in the geospatial domain. Looking forward,  paves the way for future research to explore sophisticated EO tasks, promising significant strides toward realizing the full potential of geospatial Copilots.

    ieeenat_fullname
    


