Offline Reinforcement Learning with Behavioral Supervisor Tuning
Padmanaba Srinivasan William Knottenbelt


Event-Triggered Resilient Filtering for 2-D Systems with Asynchronous-Delay: Handling Binary Encoding Decoding with Probabilistic Bit Flips
This paragraph of the first footnote will contain the date on which you submitted your paper for review. This paper was produced by the IEEE Publication Technology Group. This work was supported by the National Natural Science Foundation of China (No. 62250056), the Natural Science Foundation of Shandong Province (No.ZR2021MF069, ZR2021JQ24).

    Yu Chen and Wei Wang
Y. Chen and W. Wang are with the School of Control Science and Engineering, Shandong University, Jinan 250061, P. R. China (Email: ychen_2000@163.com, w.wang@sdu.edu.cn).

    
=====================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================




    Offline reinforcement learning (RL) algorithms are applied to learn performant, well-generalizing policies when provided with a static dataset of interactions. Many recent approaches to offline RL have seen substantial success, but with one key caveat: they demand substantial per-dataset hyperparameter tuning to achieve reported performance, which requires policy rollouts in the environment to evaluate; this can rapidly become cumbersome. Furthermore, substantial tuning requirements can hamper the adoption of these algorithms in practical domains. In this paper, we present TD3 with Behavioral Supervisor Tuning (TD3-BST), an algorithm that trains an uncertainty model and uses it to guide the policy to select actions within the dataset support. TD3-BST can learn more effective policies from offline datasets compared to previous methods and achieves the best performance across challenging benchmarks without requiring per-dataset tuning. 




§ INTRODUCTION


Reinforcement learning (RL) is a method of learning where an agent interacts with an environment to collect experiences and seeks to maximize the reward provided by the environment. This typically follows a repeating cycle of experience collecting and improvement <cit.>. This is termed online RL due to the need for policy rollouts in the environment. Both on-policy and off-policy RL require some schedule of online interaction which, in some domains, can be infeasible due to experimental or environmental limitations <cit.>. With such constraints, a dataset may instead be collected that consists of demonstrations by arbitrary (potentially multiple, unknown) behavior policies <cit.> that may be suboptimal. Offline reinforcement learning algorithms are designed to recover optimal policies from such static datasets. 



The primary challenge in offline RL is the evaluation of out-of-distribution (OOD) actions; offline datasets rarely offer support over the entire state-action space and neural networks overestimate values when extrapolating to OOD actions <cit.>. If trained using standard off-policy methods, a policy will select any actions that maximize reward, which includes OOD actions. The difference between the rewards implied by the value function and the environment results in a distribution shift that can result in failure in real-world policy rollouts. Thus, offline RL algorithms must both maximize the reward and follow the behavioral policy, while having to potentially “stitch” together several suboptimal trajectories. The former requirement is usually satisfied by introducing a constraint on the actor to either penalize deviation from the behavior policy or epistemic uncertainty of the value function, or by regularizing the value function to directly minimize OOD action-values. 

Many recent approaches to offline RL <cit.> demonstrate success in D4RL benchmarks <cit.>, but demand the onerous task of per-dataset hyperparameter tuning <cit.>. Algorithms that require substantial offline fine-tuning can be infeasible in real-world applications <cit.>, hampering their adoption in favor of simpler, older algorithms <cit.>. These older methods <cit.> provide excellent “bang-for-buck” as their hyperparameters work well across a range of D4RL datasets.

Contributions  In this paper, we show how a trained uncertainty model can be incorporated into the regularized policy objective as a behavioral supervisor to yield TD3 with behavioral supervisor tuning (TD3-BST). The key advantage of our method is the dynamic regularization weighting performed by the uncertainty network, which allows the learned policy to maximize Q-values around dataset modes. Evaluation on D4RL datasets demonstrates that TD3-BST achieves SOTA performance, and ablation experiments analyze the performance of the uncertainty model and the sensitivity of the parameters of the BST objective.




§ RELATED WORK


Reinforcement learning is a framework for sequential decision making often formulated as a Markov decision process (MDP), ℳ = {𝒮, 𝒜, R, p, p_0, γ} with state space 𝒮, action space 𝒜, a scalar reward dependent on state and action R(s, a), transition dynamics p, initial state distribution p_0 and discount factor γ∈ [0, 1) <cit.>. RL aims to learn a policy π∈Π that executes action a = π(s) that will maximize the expected discounted reward J(π) = 𝔼_τ∼ P_π(τ)[ ∑_t=0^Tγ_t R(s_t, a_t) ] where P_π(τ) = p_0 (s_0) ∏_t=0^Tπ(a_t | s_t) p(s_t+1| s_t, a_t) is the trajectory under π. Rather than rolling out an entire trajectory, a state-action value function (Q function) is often used: Q_π (s, a) = 𝔼_τ∼ P_π(τ)[ ∑_t=0^Tγ_t r(s_t, a_t) | s_0 = s, a_0 = a ].




 §.§ Offline Reinforcement Learning


Offline RL algorithms are presented with a static dataset 𝒟 that consists of tuples {s, a, r, s'} where r ∼ R(s, a) and s' ∼ p(·| s, a). 

𝒟 has limited coverage over 𝒮×𝒜; hence, offline RL algorithms must constrain the policy to select actions within the dataset support. To this end, algorithms employ one of three approaches: 1) policy constraints; 2) critic regularization; or 3) uncertainty penalization. 

Policy constraint Policy constraints modify the actor's objective only to minimize divergence from the behavior policy. Most simply, this adds a constraint term <cit.> to the policy objective:

    _π𝔼_{s, a}∼𝒟[ Q(s, π(s)) - α D(π, π_β) ],


where α is a scalar controlling the strength of regularization, D(·, ·) is a divergence function between the policy π and the behavior policy π_β. In offline RL, we do not have access to π_β; some prior methods attempt to estimate it empirically <cit.> which is challenging when the dataset is generated by a mixture of policies. Furthermore, selecting the constraint strength can be challenging and difficult to generalize across datasets with similar environments <cit.>. 

Other policy constraint approaches use weighted BC <cit.> or (surrogate) BC constraints <cit.>. The former methods may be too restrictive as they do not allow OOD action selection, which is crucial to improve performance <cit.>. The latter methods may still require substantial tuning and in addition to training if using model-based score methods. Other methods impose architectural constraints <cit.> that parameterize separate BC and reward-maximizing policy models.

Critic Regularization Critic regularization methods directly address the OOD action-value overestimation problem by penalizing large values for adversarially sampled actions <cit.>. 

Ensembles Employing an ensemble of neural network estimators is a commonly used technique for prediction with a measure of epistemic uncertainty <cit.>. A family of offline RL methods
employ large ensembles of value functions <cit.> and make use of the diversity of randomly initialized ensembles to implicitly reduce the selection of OOD actions or directly penalize the variance of the reward in the ensemble <cit.>. 

Model-Based Uncertainty Estimation Learning an uncertainty model of the dataset is often devised analogously to exploration-encouraging methods used in online RL, but, employing these for anti-exploration instead <cit.>. An example is SAC-RND which directly adopts such an approach <cit.>. Other algorithms include DOGE <cit.> which trains a model to estimate uncertainty as a distance to dataset action and DARL <cit.> which uses distance to random projections of state-action pairs as an uncertainty measure. As a whole, these methods optimize a distance d(·, ·) ≥ 0 that represents the uncertainty of an action.




 §.§ Uncertainty Estimation


Neural networks are known to predict confidently even when presented with OOD samples <cit.>. A classical approach to OOD detection is to fit a generative model to the dataset that produces a high probability for in-dataset samples and a low probability for OOD ones. These methods work well for simple, unimodal data but can become computationally demanding for more complex data with multiple modes. Another approach trains classifiers that are leveraged to become finer-grained OOD detectors <cit.>. In this work, we focus on Morse neural networks <cit.>, an approach that trains a generative model to produce an unnormalized density that takes on value 1 at the dataset modes.




§ PRELIMINARIES




A Morse neural network produces an unnormalized density M (x) ∈ [0, 1] on an embedding space ℝ^e <cit.>. A Morse network can produce a density in ℝ^e that attains a value of 1 at mode submanifolds and decreases towards 0 when moving away from the mode. The rate at which the value decreases is controlled by a Morse Kernel.


    
    A Morse Kernel is a positive definite kernel K. When applied in a space Z = ℝ^k, the kernel K(z_1, z_2) takes values in the interval [0, 1] where K(z_1, z_2) = 1 iff z_1 = z_2.


All kernels of the form K(z_1, z_2) = e^-D(z_1, z_2) where D(·,·) is a divergence <cit.> are Morse Kernels. Examples include common kernels such as the Radial Basis Function (RBF) Kernel,  

    K_RBF (z_1, z_2) = e^- λ^2/2|| z_1 - z_2 ||^2.


The RBF kernel and its derivatives decay exponentially, leading learning signals to vanish rapidly. An alternative is the ubiquitous Rational Quadratic (RQ) kernel:


    K_RQ (z_1, z_2) = (1 + λ ^ 2/2 κ|| z_1 - z_2 ||^2 )^-κ


where λ is a scale parameter in each kernel. The RQ kernel is a scaled mixture of RBF kernels controlled by κ and, for small κ, decays much more slowly <cit.>. 

Consider a neural network that maps from a feature space into a latent space f_ϕ: X → Z, with parameters ϕ, X ∈ℝ^d and Z ∈ℝ^k. A Morse Kernel can impose structure on the latent space. 


    
    A Morse neural network is a function f_ϕ: X → Z in combination with a Morse Kernel on K(z, t) where t ⊂ Z is a target, chosen as a hyperparameter of the model. The Morse neural network is defined as M_ϕ (x) = K(f_ϕ (x), t).


Using Definition <ref> we see that M_ϕ(x) ∈ [0, 1], and when M_ϕ(x) = 1, x corresponds to a mode that coincides with the level set of the submanifold of the Morse neural network. Furthermore, M_ϕ(x) corresponds to the certainty of the sample x being from the training dataset, so 1 - M_ϕ(x) is a measure of the epistemic uncertainty of x.

The function -log M_ϕ (x) measures a squared distance, d(·, ·), between f_ϕ (x) and the closest mode in the latent space at m:

    d(z) = min_m ∈ M d(z, m),


where M is the set of all modes. This encodes information about the topology of the submanifold and satisfies the Morse–Bott non-degeneracy condition <cit.>.

The Morse neural network offers the following properties:


    
    
  * M_ϕ(x) ∈ [0, 1].
    
  * M_ϕ(x) = 1 at its mode submanifolds.
    
  * -log M_ϕ (x) ≥ 0 is a squared distance that satisfies the Morse–Bott non-degeneracy condition on the mode submanifolds. 
    
  * As M_ϕ(x) is an exponentiated squared distance, the function is also distance aware in the sense that as f_ϕ (x) → t, M_ϕ(x) → 1.


Proof of each property is provided in the appendix.




§ POLICY CONSTRAINT WITH A BEHAVIORAL SUPERVISOR


We now describe the constituent components of our algorithm, building on the Morse network and showing how it can be incorporated into a policy-regularized objective. 



 §.§ Morse Networks for Offline RL


The target t is a hyperparameter that must be chosen. Experiments in <cit.> use simple, toy datasets with classification problems that perform well for categorical t. We find that using a static label for the Morse network yields poor performance; rather than a labeling model, we treat f_ϕ as a perturbation model that produces an action f_ϕ (s, a) = â such that â = a if and only if s, a ∼𝒟.

An offline RL dataset 𝒟 consists of tuples {s, a, r, s'} where we assume {s, a} pairs are i.i.d. sampled from an unknown distribution. The Morse network must be fitted on N state-action pairs [{s_1, a_1,}, ..., {s_N, a_N}] such that M_ϕ (s_i, a_j) = 1, ∀ i, j ∈ 1,...,N ] only when i = j. 

We fit a Morse neural network to minimize the KL divergence between unnormalized measures <cit.> following <cit.>, D_KL (𝒟(s, a) || M_ϕ (s, a)):

    min_ϕ𝔼_s, a ∼𝒟[ log𝒟 (s, a)/M_ϕ (s, a)] + ∫ M_ϕ (s, a) - 𝒟 (s, a)  da.


With respect to ϕ, this amounts to minimizing the empirical loss: 

    L(ϕ) = -   1/N∑_s, a ∼𝒟log K( f_ϕ (s, a), a) 
     +    1/N∑_s ∼𝒟
     a_𝒟̅∼𝒟_uni K( f_ϕ (s, a_u), a_u),

where a_u is an action sampled from a uniform distribution over the action space 𝒟_uni. 

A learned Morse density is well suited to modeling ensemble policies <cit.>, more flexibly <cit.> and without down-weighting good, in-support actions that have low density under the behavior policy <cit.> as all modes have unnormalized density value 1.

A Morse neural network can be expressed as an energy-based model (EBM) <cit.>:

    A Morse neural network can be expressed as an energy-based model: E_ϕ (x) = e^-log M_ϕ (x) where M_ϕ: ℝ^d→ℝ. 


Note that the EBM E_ϕ is itself unnormalized. Representing the Morse network as an EBM allows analysis analogous to <cit.>. 


    
    For a set-valued function F(x): x ∈ℝ^m→ℝ^n\{∅}, there exists a continuous function g: ℝ^m + n→ℝ that is approximated by a continuous function approximator g_ϕ with arbitrarily small bounded error ϵ. This ensures that any point on the graph F_ϕ (x) = _y g_ϕ (x, y) is within distance ϵ of F. 


We refer the reader to <cit.> for a detailed proof.

The theorem assumes that F(x) is an implicit function and states that the error at the level-set (i.e. the modes) of F(x) is small.




 §.§ TD3-BST


We can use the Morse network to design a regularized policy objective. Recall that policy regularization consists of Q-value maximization and minimization of a distance to the behavior policy (Equation <ref>). We reconsider the policy regularization term and train a policy that minimizes uncertainty while selecting actions close to the behavior policy. Let C^π(s, a) denote a measure of uncertainty of the policy action. We solve the following optimization problem:

    π^i+1 = _π∈Π𝔼_a ∼π (·| s)[ C^π(s, a) ]
        
    s.t.   D_KL( π(·| s) ||π_β (·| s) ) ≤ϵ.


This optimization problem requires an explicit behavior model, which is difficult to estimate and using an estimated model has historically returned mixed results <cit.>. Furthermore, this requires direct optimization through C^π which may be subject to exploitation. Instead, we enforce this implicitly by deriving the solution to the constrained optimization to obtain a closed-form solution for the actor <cit.>. Enforcing the KKT conditions we obtain the Lagrangian:

    L(π, μ) = 𝔼_a ∼π(·| s)[ C^π (s, a) ] + μ (ϵ - D_KL (π||π_β)).


Computing ∂ L/∂π and solving for π yields the uncertainty minimizing solution π^C^*(a | s) ∝π_β(a | s) e^1/μ C^π(s, a). When learning the parametric policy π_ψ, we project the non-parametric solution into the policy space as a (reverse) KL divergence minimization of π_ψ under the data distribution 𝒟: 

    _ψ𝔼_s ∼𝒟[ D_KL( π^C^*(·| s) ||π_ψ(·| s) ) ]
        
    
        = _ψ𝔼_s ∼𝒟[ D_KL( π_β(a | s) e^1/μ C^π(s, a)||π_ψ(·| s) ) ]
        
    
        = _ψ𝔼_s, a ∼𝒟[ -logπ_ψ (a | s) e^1/μ C^π(s, a)],


which is a weighted maximum likelihood update where the supervised target is sampled from the dataset 𝒟 and C^π(s, a) = 1 - M_ϕ (s, π_ψ (s)). This avoids explicitly modeling the behavior policy and uses the Morse network uncertainty as a behavior supervisor to dynamically adjust the strength of behavioral cloning. We provide a more detailed derivation in the appendix.

Interpretation  Our regularization method shares similarities with other weighted regression algorithms <cit.> which weight the advantage of an action compared to the dataset/replay buffer action. Our weighting can be thought of as a measure of disadvantage of a policy action in the sense of how OOD it is.

We make modifications to the behavioral cloning objective. From Morse network property 1 we know M_ϕ∈ [0, 1], hence 1 ≤ e^1/μ C^π≤ e^1/μ, i.e. the lowest possible disadvantage coefficient is 1. To minimize the coefficient in the mode, we require it to approach 0 when near a mode. We adjust the weighted behavioral cloning term and add Q-value maximization to yield the regularized policy update:

    π^i+1←_π𝔼_s, a ∼𝒟, 
     a_π∼π^i (s) [ 1/Z_Q Q^i+1 (s, a_π) 
     - (e^1/μC^π(s, a) - 1) ( a_π - a )^2 ],


where μ is the Lagrangian multiplier that controls the magnitude of the disadvantage weight and Z_Q = 1/N∑_n = 1^N| Q(s, a_π) | is a scaling term detached from the gradient update process <cit.>,  necessary as Q(s, a) can be arbitrarily large and the BC-coefficient is upper-bounded at e^1/μ. 

The value function update is given by:

    Q^i+1←_Q𝔼_s, a, s' ∼𝒟 [ (y - Q^i (s, a))^2 ],


with y = r(s,a) + γ𝔼_s' ∼π̅(s')Q̅ (s', a') where Q̅ and π̅ are target value and policy functions, respectively. 




 §.§ Controlling the Tradeoff Constraint


Tuning TD3-BST is straightforward; the primary hyperparameters of the Morse network consist of the choice and scale of the kernel, and the temperature μ. Increasing λ for higher dimensional actions ensures that the high certainty region around modes remains tight. Prior empirical work has demonstrated the importance of allowing some degree of OOD actions <cit.>; in the TD3-BST framework, this is dependent on λ. In Figure <ref> we provide a didactic example of the effect of λ. We construct a dataset consisting of 2-dimensional actions in [-1, 1] with means at the four locations {[0.0, 0.8], [0.0, -0.8], [0.8, 0.0], [-0.8, 0.0]} and each with standard deviation 0.05. We sample M = 128 points, train a Morse network and plot the density produced by the Morse network for λ = {1/10, 1/2, 1.0, 2.0}. A behavioral cloning policy learned using vanilla MLE where all targets are weighted equally results in an OOD action being selected. Training using Morse-weighted BC downweights the behavioral cloning loss for far away modes enabling the policy to select and minimize error to a single mode.






 §.§ Algorithm Summary


Fitting the Morse Network  The TD3-BST training procedure is described in Algorithm <ref>. The first phase fits the Morse network for T_M gradient steps.

Actor–Critic Training  In the second phase of training, a modified TD3-BC procedure is used for T_AC iterations with alterations highlighted in red. 

We provide full hyperparameter details in the appendix.






§ EXPERIMENTS


In this section, we conduct experiments that aim to answer the following questions:

    
  * How does TD3-BST compare to other baselines, with a focus on comparing to newer baselines that use per-dataset tuning?
    
  * Can the BST objective improve performance when used with one-step methods (IQL) that perform in-sample policy evaluation?
    
  * How well does the Morse network learn to discriminate between in-dataset and OOD actions?
    
  * How does changing the kernel scale parameter λ affect performance? 
    
  * Does using independent ensembles, a second method of uncertainty estimation, improve performance?


We evaluate our algorithm on the D4RL benchmark <cit.>, including the Gym Locomotion and challenging Antmaze navigation tasks.




 §.§ Comparison with SOTA Methods


We evaluate TD3-BST against the older, well known baselines of TD3-BC <cit.>, CQL <cit.>, and IQL <cit.>. There are more recent methods that consistently outperform these baselines; of these, we include SQL <cit.>, SAC-RND <cit.>, DOGE <cit.>, VMG <cit.>, ReBRAC <cit.>, CFPI <cit.> and MSG <cit.> (to our knowledge, the best-performing ensemble-based method). It is interesting to note that most of these baselines implement policy constraints, except for VMG (graph-based planning) and MSG (policy constraint using a large, independent ensemble). We note that all the aforementioned SOTA methods (except SQL) report scores with per-dataset tuned parameters in stark contrast with the older TD3-BC, CQL, and IQL algorithms, which use the same set of hyperparameters in each D4RL domain. All scores are reported with 10 evaluations in Locomotion and 100 in Antmaze across five seeds.

We present scores for D4RL Gym Locomotion in Table <ref>. TD3-BST achieves best or near-best results compared to all previous methods and recovers expert performance on five of nine datasets. The best performing prior methods include SAC-RND and ReBRAC, both of which require per-dataset tuning of BRAC-variant algorithms <cit.>. 

We evaluate TD3-BST on the more challenging Antmaze tasks which contain a high degree of suboptimal trajectories and follow a sparse reward scheme that requires algorithms to stitch together several trajectories to perform well. TD3-BST achieves the best scores overall in Table <ref>, especially as the maze becomes more complex. VMG and MSG are the best-performing prior baselines and TD3-BST is far simpler and more efficient in its design as a variant of TD3-BC. The authors of VMG report the best scores from checkpoints rather than from the final policy. MSG report scores from ensembles with both 4 and 64 critics of which the best scores included here are from the 64-critic variant.

We pay close attention to SAC-RND, which, among all baselines, is most similar in its inception to TD3-BST. SAC-RND uses a random and trained network pair to produce a dataset-constraining penalty. SAC-RND achieves consistent SOTA scores on locomotion datasets, but fails to deliver commensurate performance on Antmaze tasks. TD3-BST performs similarly to SAC-RND in locomotion and achieves SOTA scores in Antmaze. 








 §.§ Improving One-Step Methods


One-step algorithms learn a policy from an offline dataset, thus remaining on-policy <cit.>, and using weighted behavioral cloning <cit.>. Empirical evaluation by <cit.> suggests that advantage-weighted BC is too restrictive and relaxing the policy objective to Equation <ref> can lead to performance improvement. We use the BST objective as a drop-in replacement for the policy improvement step in IQL <cit.> to learn an optimal policy while retaining in-sample policy evaluation.

We reproduce IQL results and report scores for IQL-BST, both times using a deterministic policy <cit.> and identical hyperparameters to the original work in Table <ref>. Reproduced IQL closely matches the original results, with slight performance reductions on the  datasets. Relaxing weighted-BC with a BST objective leads to improvements in performance, especially on the more difficult  and  datasets. To isolate the effect of the BST objective, we do not perform any additional tuning.






 §.§ Ablation Experiments


Morse Network Analysis  We analyze how well the Morse network can distinguish between dataset tuples and samples from 𝒟_perm, permutations of dataset actions, and 𝒟_uni. We plot both certainty (M_ϕ) density and t-SNEs <cit.> in Figure <ref> which show that the unsupervised Morse network is effective in distinguishing between 𝒟_perm and 𝒟_uni and assigning high certainty to dataset tuples.




Ablating kernel scale  We examine sensitivity to the kernel scale λ. Recall that k = (𝒜). We see in Figure <ref> that the scale λ = k/2 is a performance sweet-spot on the challenging Antmaze tasks. We further illustrate this by plotting policy deviations from dataset actions in Figure <ref>. The scale λ = 1.0 is potentially too lax a behavioral constraint, while λ = k is too strong, resulting in performance reduction. However, performance on all scales remains strong and compares well with most prior algorithms. Performance may be further improved by tuning λ, possibly with separate scales for each input dimension.






Independent or Shared Targets?  Standard TD3 employs Clipped Double Q-learning (CDQ) <cit.> to prevent value overestimation. On tasks with sparse rewards, this may be too conservative <cit.>. MSG <cit.> uses large ensembles of fully independent Q functions to learn offline. We examine how independent double Q functions perform compared to the standard CDQ setup in Antmaze with 2 and 10 critics. The results in Figure <ref> show that disabling CDQ with 2 critics is consistently detrimental to performance. Using a larger 10-critic ensemble leads to moderate improvements. This suggests that combining policy regularization with an efficient, independent ensemble could bring further performance benefits with minimal changes to the algorithm. 






§ DISCUSSION


Morse Network  In <cit.>, deeper architectures are required even when training on simple datasets. This rings true for our application of Morse networks in this work, with low-capacity networks performing poorly. Training the Morse network for each locomotion and Antmaze dataset typically takes 10 minutes for 100 000 gradient steps using a batch size of 1 024. When training the policy, using the Morse network increases training time by approximately 15%.

Optimal Datasets  On Gym Locomotion tasks TD3-BST performance is comparable to newer methods, all of which rarely outperform older baselines. This can be attributed to a significant proportion of high-return-yielding trajectories that are easier to improve.




§ CONCLUSION


In this paper, we introduce TD3-BST, an algorithm that uses an uncertainty model to dynamically adjust the strength of regularization. Dynamic weighting allows the policy to maximize reward around individual dataset modes. Our algorithm compares well against prior methods on Gym Locomotion tasks and achieves the best scores on the more challenging Antmaze tasks, demonstrating strong performance when learning from suboptimal data. 

In addition, our experiments show that combining our policy regularization with an ensemble-based source of uncertainty can improve performance. Future work can explore other methods of estimating uncertainty, alternative uncertainty measures, and how best to combine multiple sources of uncertainty.




named


