Information-Guided Robotic Maximum Seek-and-Sample in Partially Observable Continuous Environments
Genevieve Flaspohler Victoria Preston Anna P. M. Michel Yogesh Girdhar Nicholas Roy




A Survey of Storage Class Memory: 
Principles, Problems, and Possibilities

    Aditya K Kamath,
        Leslie Monis,
        A Tarun Karthik,
        and Basavaraj Talawar
A. Kamath, L. Monis, A. Karthik, and B. Talawar are with the Department of Computer Science and Engineering, National Institute of Technology Karnataka, Surathkal,
Karnataka, 575025.



E-mails: akamath1997@gmail.com, lesliemonis@gmail.com, tarunkarthik999@gmail.com, basavaraj@nitk.edu.in
Manuscript received -; revised -.
    December 30, 2023
=====================================================================================================================================================================================================================================================================================================================================================================================================================================




We present PLUMES, a planner for localizing and collecting samples at the global maximum of an a priori unknown and partially observable continuous environment. This “maximum seek-and-sample” (MSS) problem is pervasive in the environmental and earth sciences. Experts want to collect scientifically valuable samples at an environmental maximum (e.g., an oil-spill source), but do not have prior knowledge about the phenomenon's distribution. We formulate the MSS problem as a partially-observable Markov decision process (POMDP) with continuous state and observation spaces, and a sparse reward signal. To solve the MSS POMDP, PLUMES uses an information-theoretic reward heuristic with continuous-observation Monte Carlo Tree Search to efficiently localize and sample from the global maximum. In simulation and field experiments, PLUMES collects more scientifically valuable samples than state-of-the-art planners in a diverse set of environments, with various platforms, sensors, and challenging real-world conditions.










§ INTRODUCTION


In many environmental and earth science applications, experts want to collect scientifically valuable samples of a maximum (e.g., an oil spill source), but the distribution of the phenomenon is initially unknown. This maximum seek-and-sample (MSS) problem is pervasive. Canonically, samples are collected at predetermined locations by a technician or by a mobile platform following a uniform coverage trajectory. These non-adaptive strategies result in sample sparsity at the maximum and may be infeasible when the geometric structure of the environment is unknown (e.g., boulder fields) or changing (e.g., tidal zones). Increasing the number of valuable samples at the maximum requires adaptive online planning and execution. We present PLUMES — Plume Localization under Uncertainty using Maximum-ValuE information and Search — an adaptive algorithm that enables a mobile robot to efficiently localize and densely sample an environmental maximum, subject to practical challenges including dynamic constraints, unknown geometric map and obstacles, and noisy sensors with limited field-of-view. Fig. <ref> shows a motivating application: coral head localization.



Informative Path Planning: The MSS problem is closely related to informative path planning (IPP) problems. Canonical offline IPP techniques for pure information-gathering that optimize submodular coverage objectives can achieve near-optimal performance <cit.>. However, in the MSS problem, the value of a sample depends on the unknown maximum location, requiring adaptive planning to enable the robot to select actions that explore to localize the maximum and then seamlessly transition to selecting actions that exploitatively collect valuable samples there. Even for adaptive IPP methods, the MSS problem presents considerable challenges. The target environmental phenomenon is partially observable and most directly modeled as a continuous scalar function. Additionally, efficient maximum sampling with a mobile robot requires consideration of vehicle dynamics, travel cost, and a potentially unknown obstacle map. Handling these challenges in combination excludes adaptive IPP algorithms that use discrete state spaces <cit.>, known metric maps <cit.>, or unconstrained sensor placement <cit.>.

The MSS POMDP: Partially-observable Markov decision processes (POMDPs) are general models for decision-making under uncertainty that allow the challenging aspects of the MSS problem to be encoded. We define the MSS POMDP, in which the partially observable state represents the continuous environmental phenomenon and a sparse reward function encodes the MSS scientific objective by giving reward only to samples sufficiently close to the global maximum. Solving a POMDP exactly is generally intractable, and the MSS POMDP is additionally complicated by both continuous state and observation spaces, and the sparse MSS reward function. This presents the two core challenges that PLUMES addresses: performing online search in a belief-space over continuous functions, and overcoming reward function sparsity.



Planning over Continuous Domains: In the MSS problem, the state of the environment can be modeled as a continuous function. PLUMES uses a Gaussian Process (GP) model to represent the belief over this continuous function, and must plan over the uncountable set of possible GP beliefs that arise from future continuous observations. To address planning in continuous spaces, state-of-the-art online POMDP solvers use deterministic discretization <cit.> or a combination of sampling techniques and particle filter belief representations <cit.>. Efficiently discretizing or maintaining a sufficiently rich particle set to represent the underlying continuous function in MSS applications is itself a challenging problem, and can lead to inaccurate inference of the maximum <cit.>. Other approaches have considered using the maximum-likelihood observation to make search tractable <cit.>. However, this assumption can compromise search and has optimality guarantees only in linear-Gaussian systems <cit.>. Instead, PLUMES uses Monte Carlo Tree Search (MCTS) with progressive widening, which we call continuous-observation MCTS, to limit planning tree growth <cit.> and retain optimality <cit.> in continuous environments.



Rewards and Heuristics: In the MSS POMDP, the reward function is sparse and does not explicitly encode the value of exploration. Planning with sparse rewards requires long-horizon information gathering and is an open problem in robotics <cit.>. To alleviate this difficulty, less sparse heuristic reward functions can be optimized in place of the true reward, but these heuristics need to be selected carefully to ensure the planner performs well with respect to the true objective. In IPP, heuristics based on the value of information have been applied successfully <cit.>, primarily using the GP-UCB criteria <cit.>. We demonstrate that within practical mission constraints, using UCB as the heuristic reward function for the MSS POMDP can lead to suboptimal convergence to local maxima due to a mismatch between the UCB heuristic and the true MSS reward. Instead, PLUMES takes advantage of a heuristic function from the Bayesian optimization (BO) community for state-of-the-art black-box optimization <cit.>, which we call maximum-value information (MVI). MVI overcomes sparsity and encourages long-term information gathering, while still converging to the true reward of the MSS POMDP.



The contribution of this paper is the MSS POMDP formalism and the corresponding PLUMES planner, which by virtue of its belief model, information-theoretic reward heuristic, and search framework, enables efficient maximum seek and sample with asymptotic optimality guarantees in continuous environments. PLUMES extends the state-of-the-art in MSS planners by applying a BO heuristic reward function to MSS that alleviates the challenges of the true sparse MSS reward function, and integrating GP belief representations within continuous-observation MCTS. The utility of PLUMES for MSS applications is demonstrated in extensive simulation and field trials,  showing a statistically significant performance improvement over state-of-the-art baselines.




§ MAXIMUM SEEK-AND-SAMPLE POMDP


We formalize the MSS problem by considering a target environmental domain as a d-dimensional compact set 𝕏_w ⊂ℝ^d. We allow 𝕏_w to contain obstacles with arbitrary geometry and let 𝕏⊂𝕏_w be the set of reachable points with respect to the robot's initial pose. We assume there is an unknown underlying continuous function f: 𝕏_w →ℝ representing the value of a continuous phenomenon of interest. The objective is to find the unique global maximizer ^* = _∈𝕏 f() by safely navigating while receiving noisy observations of this function f. Because f is unknown, we cannot access derivative information or any analytic form. 

We model the process of navigating and generating observations as the MSS POMDP: an 8-tuple (𝒮, 𝒜, 𝒵, T, O, R, γ, b_0):

    
  * 𝒮: continuous state space of the robot and environment
    
  * 𝒜: discrete set of action primitives
    
  * 𝒵: continuous space of possible observations
    
  * T: 𝒮×𝒜→𝒫(𝒮), the transition function, i.e., 
 𝐏𝐫(S_t+1 = s' | S_t = s, A_t = a)
    
  * O: 𝒮×𝒜→𝒫(𝒵), the observation model, i.e., 
 𝐏𝐫(Z_t+1 = z | S_t+1 = s, A_t = a)
    
    
  * R: 𝒮×𝒜→ℝ, the reward of taking action a when robot's state is s, i.e., R(s, a)
    
  * γ: discount factor, 0 ≤γ≤ 1
    
  * b_0: initial belief state of the robot, b_0 ∈𝒫(S_0)

where 𝒫(·) denotes the space of probability distributions over the argument.

The Bellman equation is used to recursively quantify the value of belief b_t = 𝒫(S_t) over a finite horizon h under policy π: b_t → a_t as: 

    V^π_h(b_t) =    𝔼[R(s_t, π(b_t))] + 
       γ∫_z ∈𝒵 V^π_h-1(b^π(b_t), z_t+1) Pr(z | b_t, π(b_t)) dz,

where the expectation is taken over the current belief and b^π(b_t), z_t+1 is the updated belief after taking action π(b_t) and observing z ∈𝒵. The optimal policy π_h^* over horizon-h is the maximizer of the value function over the space of possible policies Π: π_h^* = _π∈Π V^π_h(b_t). However, Eq. <ref> is intractable to compute in continuous state and observation spaces; the optimal policy must be approximated. PLUMES uses a receding-horizon, online POMDP planner and heuristic reward function to approximately solve the MSS POMDP in real-time on robotic systems. 






§ THE PLUMES ALGORITHM


PLUMES is an online planning algorithm with a sequential decision-making structure:
 
   
  * Conditioned on b_t, approximate the optimal policy π_h^* for finite horizon h and execute the action a = π̂_h^*(b_t).
   
  * Collect observations z ∈𝒵, according to O.
   
  * Update b_t to incorporate this new observation; repeat.
 
In the following sections, we define the specific choice of belief model, planning algorithm, and heuristic reward function that PLUMES uses to solve the MSS POMDP. 




 §.§ Gaussian Process Belief Model


We assume the robot's pose _t at planning iteration t is fully observable, and the unknown environmental phenomenon f is partially observable. The full belief-state is represented as a tuple b_t of robot state _t and environment belief g_t = 𝒫(f) at time t. Because f is a continuous function, we cannot represent the belief g_t as a distribution over discrete states, as is standard in POMDP literature <cit.>, and must choose an alternate representation. PLUMES uses a Gaussian process (GP) <cit.> to represent g_t conditioned on a history of past observations. This GP is parameterized by mean μ(𝐱) and covariance function κ(𝐱, 𝐱').


As the robot traverses a location 𝐱, it gathers observations z ∈𝒵 of f subject to sensor noise σ_n^2, such that z = f(𝐱) + ϵ with ϵi.i.d.∼𝒩(0, σ_n^2). Given a history 𝒟_t = {𝐱_i, z_i}_i = 0^D of D observations and observation locations at planning iteration t, the posterior belief at a new location 𝐱' ∈𝕏 is computed: 

    g_t (') |𝒟_t∼𝒩(μ_t('), σ_t^2(')), where
       μ_t(') = κ_t(')^⊤(𝐊_t + σ_n^2𝐈)^-1𝐳_t,    
       σ_t^2(') = κ(', ') - κ_t(')^⊤(𝐊_t + σ_n^2𝐈)^-1κ_t('),

where 𝐳_t = [z_0, …, z_D-1]^⊤, 𝐊_t is the positive definite kernel matrix with 𝐊_t[i, j] = κ(_i, _j) for all _i, _j ∈𝒟_t, and κ_t(') = [κ(_0, '), …, κ(_D-1, ')]^⊤. 




 §.§ Planning with Continuous-Observation MCTS


PLUMES selects high-reward actions with receding-horizon search over possible belief states. This search requires a simulator that can sample observations and generate beliefs given a proposed action sequence. For PLUMES, this simulator is the GP model, which represents the belief over the continuous function f, and in turn simulates continuous observations from proposed action sequences by sampling from the Gaussian distribution defined by Eq. <ref> & <ref>.

PLUMES uses continuous-observation MCTS to overcome the challenges of planning in continuous state and observation spaces. Continuous-observation MCTS has three stages: selection, forward simulation, and back-propagation. Each node in the tree can be represented as the tuple of robot pose and GP belief, b_t = {_t, g_t}. Additionally, we will refer to two types of nodes: belief nodes and belief-action nodes. The root of the tree is always a belief node, which represents the entire history of actions and observations up through the current planning iteration. Through selection and simulation, belief and belief-action nodes are alternately added to the tree (Fig. <ref>).

From the root, a rollout begins with the selection stage, in which a belief-action child is selected according to the Polynomial Upper Confidence Tree (PUCT) policy <cit.>. The PUCT value Q̂^*_aug(b_t, a) is the sum of the average heuristic rewards (i.e., MVI) from all previous simulations and a term that favors less-simulated action sequences:

    Q̂^*_aug(b_t, a) = Q̂^*(b_t, a) + √(N(b_t)^e_d/N(b_t, a)),
  
where Q̂^*(b_t, a) is the average heuristic reward of choosing action a with belief b_t in all previous rollouts, N(b_t) is the number of times the node b_t has been simulated, N(b_t, a) is the number of times that particular action from node b_t has been selected, and e_d is a depth-dependent parameter[Refer to Table 1 of Auger et al. <cit.> for parameter settings. ].



Once a child belief-action node is selected, the action associated with the child is forward simulated using the generative observation model O, and a new belief node is generated b_t+1 = {_t+1, g_t+1} as though the action were taken and samples observed. The simulated observations are drawn from the belief-action node's GP model g_t, and the robot's pose is updated deterministically based on the selected action. Since the observations in a GP are continuous, every sampled observation is unique with probability one. Progressive widening, with depth-dependent parameter^* α_d incrementally grows the tree by limiting the number of belief children of each belief-action node. When growing the tree, b_t+1 is either chosen to be the least visited node if ⌊ N(b_t, a)^α_d⌋=⌊ (N(b_t, a) - 1)^α_d⌋, or otherwise is a new child with observations simulated from b_t. By limiting the width of the search tree and incrementally growing the number of explored children, progressive widening avoids search degeneracy in continuous environments.

Once a sequence of actions has been rolled out to a horizon h, the accumulated heuristic reward is propagated upward from the leaves to the tree root. The average accumulated heuristic reward and number of queries are updated for each node visited in the rollout. Rollouts continue until the computation budget is exhausted. The most visited belief-action child of the root node is executed.



Continuous-observation MCTS within PLUMES provides both practical and theoretical benefits. Practically, progressive-widening directly addresses search degeneracy by visiting belief nodes multiple times even in continuous observation spaces, allowing for a more representative estimate of their value. Theoretically, PLUMES can be shown to select asymptotically optimal actions. We briefly describe how analysis in Auger et al. <cit.> for PUCT-MCTS with progressive widening in MDPs can be extended to PLUMES.  

Using standard methods <cit.>, we can reduce the MSS POMDP to an equivalent belief-state MDP. This belief-state MDP has a state space equal to the set of all possible beliefs, and a transition distribution that captures the effect of both the dynamics and the observation model after each action. Planning in this representation is often intractable as the state space is continuous and infinite-dimensional. However, PLUMES plans directly in the belief-state MDP by using its GP belief state to compute the transition function efficiently.

Subsequently, Theorem 1 in Auger et al. <cit.> shows that for an MDP with a continuous state space, like the belief-state MDP representation suggested, the value function estimated by continuous-observation MCTS asymptotically converges to that of the optimal policy:

    |Q̂_h^*(b_t, a)  - Q_h^*(b_t, a)|≤C/N(b_t, a)^γ_d,
 
with high probability <cit.>, for constants C > 0 and γ_d*.



 §.§ Maximum-Value Information Reward


The true state-dependent reward function for the MSS POMDP would place value on collecting sample points 𝐱 within an ϵ-ball of the true global maximum 𝐱^*:

    R(f, 𝐱) = 1_‖𝐱 - 𝐱^* ‖ < ϵ,

where ϵ is determined by the scientific application. Optimizing this sparse reward function directly is challenging, so PLUMES approximates the true MSS reward by using the maximum-value information (MVI) heuristic reward <cit.>. MVI initially encourages exploration behavior, but ultimately rewards exploitative sampling near the inferred maximum.

The belief-dependent MVI heuristic reward R̃(b_t, ) quantifies the expected value of having belief b_t and collecting a sample at location ∈𝕏. MVI reward quantifies the mutual information between the random variable Z, representing the observation at location , and Z^*, the random variable representing the value of the function f at the global maximum:



    R̃(b_t, ) = I({, Z} ; Z^* | b_t),

where Z^* = _' ∈𝕏 f('). To compute the reward of collecting a random observation Z at location  under belief b_t, we approximate the expectation over the unknown Z^* by sampling from the posterior distribution z^*_i ∼ p(Z^* | b_t) and use Monte Carlo integration with M samples <cit.>: 



    R̃(b_t, )  =  H[Pr(Z |, b_t)] - 
       𝔼_z' ∼Pr(Z^* | b_t) [H[Pr(Z |, b_t, Z^* = z')],
    
       ≈ H[Pr(Z |, b_t)] - 1/M∑_i = 0^M H[Pr(Z |, b_t, Z^*=z^*_i)].

Each entropy expression H[·] can be respectively approximated as the entropy of a Gaussian random variable with mean and variance given by the GP equations (Eq. <ref> & <ref>), and the entropy of a truncated Gaussian, with upper limit z^*_i and the same mean and variance.

To draw samples z^*_i from the posterior p(Z^* | b_t), we employ spectral sampling <cit.>. Spectral sampling draws a function f̂, which has analytic form and is differentiable, from the posterior belief of a GP with stationary covariance function <cit.>. To complete the evaluation of Eq. <ref>, z_i^* ∼ p(Z^* | b_t) can be computed by applying standard efficient global optimization techniques (e.g., sequential least squares programming, quasi-Newton methods) to find the global maximum of the sampled f̂. This results in the following expression for MVI reward <cit.>:

    R̃(b_t, ) ≈1/M∑_i = 0^M  γ_z_i^*() ϕ(γ_z_i^*())/2 Φ(γ_z_i^*())-log(Φ(γ_z_i^*()))

where γ_z_i^*() = z_i^* - μ_t()/σ_t(), μ_t(x) and σ_t(x) are given by Eq. <ref> & <ref>, and ϕ and Φ are the standard normal PDF and CDF. For actions that collect samples at more then one location, the reward of an action R̃(b_t, a) is the sum of rewards of the locations sampled by that action.

MVI initially favors collecting observations in areas that have high uncertainty due to sampling maxima from the initial uniform GP belief. As observations are collected and uncertainty diminishes in the GP, the sampled maxima converge to the true maximum and reward concentrates locally at this point, encouraging exploitative behavior. This contrasts with the Upper Confidence Bound (UCB) heuristic, which distributes reward proportional to predictive mean μ_t(𝐱) and weighted variance σ_t(𝐱) of the current GP belief model (Eq. <ref> & <ref>):
R̃_UCB(b_t, 𝐱) = μ_t(𝐱) + √(β_t)σ(𝐱). As the robot explores, UCB reward converges to the underlying phenomenon, f. The difference in convergence characteristics between MVI and UCB can be observed in Fig. <ref>. 



§ EXPERIMENTS AND RESULTS




We analyze the empirical performance of PLUMES in a breadth of MSS scenarios that feature convex and non-convex environments. We compare against three baselines used in environmental surveying: non-adaptive lawnmower-coverage (Boustro., an abbreviation of boustrophedonic <cit.>), greedy myopic planning with UCB reward (UCB-Myopic) <cit.>, and nonmyopic planning with traditional MCTS <cit.> that uses the maximum-likelihood observation and UCB reward (UCB-MCTS) <cit.>. The performance of UCB planners has been shown to be sensitive with respect to β value <cit.>. In order to avoid subjective tuning, we select a time-varying β_t that is known to enable no-regret UCB planning <cit.>. PLUMES uses continuous-observation MCTS with hyperparameters presented in Auger et al. <cit.>.

To evaluate the mission performance of all planners, we report accumulated MSS reward (Eq. <ref>), which directly corresponds to the number of scientifically valuable samples collected within an ϵ-ball of the true maximum. This metric is reported for all trial scenarios in Table <ref>. We additionally report several metrics commonly used in IPP to evaluate posterior model quality: overall environmental posterior root mean-squared error (RMSE) and error in posterior prediction of ^* at the end of a mission (^* error). We use a Mann-Whitney U non-parametric significance test <cit.> to report statistical significance (p = 0.05 level) in performance between PLUMES and baseline algorithms.





 §.§ Bounded Convex Environments





In marine and atmospheric applications, MSS often occurs in a geographically bounded, obstacle-free environment. In 50 simulated trials, we applied PLUMES and our baseline planners to a point robot in a 10 × 10 multimodal environment drawn randomly from a GP prior with a squared-exponential covariance function and zero mean (l = 1.0, σ^2 = 100.0, σ_n^2 = 1.0 [1%]) (see Fig.<ref>). The action set consisted of ten viable trajectories centered at the robot's pose with path length 1.5, and samples were collected every 0.5 of travel. Mission lengths were budgeted to be 200. Nonmyopic planners rolled out to a 5-action horizon and were allowed 250 rollouts per planning iteration. Summary simulation results are presented in Table <ref>.

In these trials, PLUMES accumulated significantly (0.05-level) more reward than baselines. The distribution of accumulated reward (Fig. <ref>) shows that PLUMES has a single dominating mode near reward 200 and few low-performing missions (reward 50). In contrast, both UCB-based methods have distributions which are multimodal, with non-trivial modes in the low-performance region. Boustro. collected consistently few scientifically valuable samples. In addition to collecting many more samples at the maximum, PLUMES achieved statistically indistinguishable levels of posterior RMSE and ^* error compared to baselines (Table <ref>).

The corresponding field trial for convex-world maximum-search was performed in the Bellairs Fringing Reef, Barbados by a custom-built autonomous surface vehicle (ASV) with the objective of localizing the most exposed coral head. Coral head exposure is used to select vantage points for coral imaging <cit.> and in ultraviolet radiation studies on coral organisms <cit.>. Due to time and resource constraints, only one trial of two planners was feasible on the physical reef; we elected to demonstrate PLUMES and Boustro., one of the most canonical surveying strategies in marine sciences.

The ASV (1 × 0.5) had holonomic dynamics and a downward-facing acoustic point altimeter (Tritech Micron Echosounder) with returns at 1. Ten dynamically-feasible 10 straight paths radiating from the location of the ASV were used in the action set. The environment was bounded by a 50 by 50 geofence. Localization and control was provided by a PixHawk Autopilot with GPS and internal IMU; the fused state estimate was empirically suitable for the desired maximum localization accuracy (ϵ = 10). The budget for each mission was 1000, which took approx. 45 minutes to travel. The GP kernel was trained on altimeter data from a dense data collection deployment the day before (parameters l = 2.01, σ^2 = 0.53, σ_n^2 = 0.02 [26%]). Note the high noise in the inferred GP model, as well as the relatively small length-scale in the 2500 field site. The reconstructed bathymetry and vehicle are shown in Fig. <ref>.





PLUMES successfully identified the same coral head to be maximal as that inferred from the GP trained on prior dense data collection, as indicated by accumulated reward in Table <ref>, overcoming the challenges of moving in ocean waves, noisy altimeter measurements, and highly multimodal environment. Additionally, the posterior prediction of ^* had an error of only 1.78 while Boustro. reported 8.75 error due to its non-adaptive sampling strategy.

In the Bellairs Fringing Reef trials, the environment was assumed to be static. However, in many marine domains the impact of sediment transport, waves, and tides could physically change the location of a maximum over the course of a mission. PLUMES can be extended to dynamic environments by employing a spatiotemporal kernel in the GP model, which allows for the predictive mean and variance to change temporally <cit.>. If the dynamics of an environment can be encoded in the kernel function, no other changes to PLUMES are necessary; MVI will be distributed according to the time dynamic. Fig. <ref> demonstrates the properties of PLUMES with a squared-exponential kernel over space (l = 1.5, σ^2 = 100, σ_n^2 = 0.5) and time (l = 100, σ^2 = 100, σ_n^2 = 0.5). In this illustrative scenario, the global maximum moved between planning iteration T = 230 and T = 250. PLUMES with a spatiotemporal kernel maintained multiple hypotheses about the maximum's location given the random-walk dynamic of the environment, resulting in MVI reward being re-distributed between the two maxima over time.



 §.§ Non-Convex Environments


We next consider non-convex environments with potentially unknown obstacles, a situation that occurs frequently in practical MSS applications with geographical no-go zones for rover or ASV missions, and in indoor or urban settings. We evaluated PLUMES, UCB-Myopic, and UCB-MCTS planners in 50 simulated trials with the same environments, vehicle, and actions as described in Section <ref>, with the inclusion of 12 block obstacles placed uniformly around the world in known locations (see Fig.<ref>). Boustro. was not used as a baseline because of non-generality of the offline approach to unknown obstacle maps. 

As indicated in Table <ref>, PLUMES accumulated significantly more MSS reward than UCB-MCTS and UCB-Myopic, at the 0.05-level. The distribution of reward across the trials is visualized in Fig. <ref>. Like in the convex-world, the PLUMES has a primary mode between reward 200-250, while the UCB-based planners have a primary mode in the low-performance region (reward 50). There was no significant difference between planners with respect to RMSE or ^* error. 
The fact that PLUMES maximized the true MSS reward while achieving statistically indistinguishable error highlights the difference in exploitation efficiency between PLUMES and UCB-based methods.





The simulation experiments assume that a geometric map is known a priori. However in practical applications, like indoor gas leak detection, access to a map may be limited or unavailable. We simulate the scenario in which a nonholonomic car equipped with a laser range-finder must build a map online as it seeks the maximum in a cluttered indoor environment (Fig. <ref>). We generate a simulated chemical phenomenon from a GP (l = 0.8, σ^2 = 100.0, σ_n^2 = 2.0 [2%]), and simulate observations at 1. The action set for the vehicle consists of eleven 1.5 Dubins curves projected in front of the vehicle, one straight path behind the vehicle, and a “stay in place” action. Results for five trials are shown in Table <ref> and illustrate that PLUMES accumulates more MSS reward than baselines, indicating robust performance.

These simulation and robot trials demonstrate the utility of PLUMES compared to canonical and state-of-the-art baselines in a diverse set of environments with challenging practical conditions. For high-stakes scientific deployments, the consistent convergence and sampling performance of PLUMES is critical and beneficial.




§ DISCUSSION AND FUTURE WORK


Online planning methods for robotic maximum seek-and-sample are critical in a variety of contexts, including general environmental monitoring (scientific inquiry, reconnaissance) and disaster response (oil spill, gas leak, radiation). For partially observable environments that can be modelled using a GP, PLUMES is a novel approach for global maximum seek-and-sample that provides several key insights.

This work presents MVI as an empirically suitable alternative to the canonical GP-UCB heuristic in MSS solvers, which is both naturally adaptive and avoids a hand-tuned parameter to balance exploration and exploitation. MVI samples potential global maxima from the robot's full belief state to manage exploration and exploitation. In contrast, heuristic functions like UCB place reward on all high-valued or highly uncertain regions, leading to unnecessary exploration and limiting the time available to exploit knowledge of the true maximum. Ultimately, the MVI heuristic allows PLUMES to collect exploitative samples, while still achieving the same overall level of posterior model accuracy (shown by RMSE) as UCB-based planners. Additionally, continuous-observation MCTS allows PLUMES to search over belief-spaces on continuous functions without discretization or maximum-likelihood assumptions.

One important area of future work for PLUMES is online GP kernel hyperparameter learning <cit.>, which is important when only one mission is possible and there is insufficient prior knowledge for hyperparameter selection. Another avenue of future work could be to examine the proprieties of the maxima sampled by MVI, to be used as a heuristic for meta-behavior transitions (e.g., action model switching, dynamic horizon setting) or mission termination. Finally, the performance of PLUMES in non-convex environments is impacted by the chosen discrete action set. Extending PLUMES to continuous actions spaces, in the spirit of, e.g., Morere et al. <cit.>, would allow increased flexibility in these environments.



§ CONCLUSION

This paper formalizes the maximum-seek-and-sample POMDP and presents PLUMES, an adaptive planning algorithm that employs continuous-observation MCTS and maximum-value information reward to perform efficient maximum-seeking in partially observable, continuous environments. PLUMES outperforms canonical coverage and UCB-based state-of-the-art methods with statistical significance in challenging simulated and real-world conditions (e.g. multiple local maxima, unknown obstacles, sensor noise). Maximum seek-and-sample is a critical task in environmental monitoring for which PLUMES, with theoretical convergence guarantees, strong empirical performance, and robustness under real-world conditions, is well-suited.




§ ACKNOWLEDGMENT

We would like to thank our reviewers for their feedback on this manuscript. Additionally, we thank the SLI group, RRG and WARPlab for their insight and support. This project was supported by an NSF-GRFP award (G.F.), NDSEG Fellowship award (V.P.), and NSF NRI Award 1734400.







IEEEtranBST/IEEEtran



