Closed-loop Parameter Identification of Linear Dynamical Systems through the Lens of Feedback Channel Coding Theory
Ali Reza Pedram Takashi Tanaka
eess.SY cs.LG cs.SY


Heisenberg scaling precision in multi-mode distributed quantum metrology
    Vincenzo Tamma
    December 30, 2023
========================================================================

empty
empty




This paper considers the problem of closed-loop identification of linear scalar systems with Gaussian process noise, where the system input is determined by a deterministic state feedback policy. The regularized least-square estimate (LSE) algorithm is adopted, seeking to find the best estimate of unknown model parameters based on  noiseless measurements of the state.  We are interested in the fundamental limitation of the rate at which unknown parameters can be learned, in the sense of the D-optimality scalarization criterion subject to a quadratic control cost.  We first establish a novel connection between a closed-loop identification problem of interest and a channel coding problem involving an additive white Gaussian noise (AWGN) channel  with feedback and a certain structural constraint. 

Based on this connection, we show that the learning rate is fundamentally upper bounded by the capacity of the corresponding AWGN channel.  Although the optimal design of the feedback policy remains challenging, we derive conditions under which the upper bound is achieved. Finally, we show that the obtained upper bound implies that super-linear convergence is unattainable for any choice of the policy.   






§ INTRODUCTION

System identification (ID)

has been a subject of research interest broadly in control and computer science communities <cit.>. 


Research efforts on the ID methods have generated a number of key results, such as the guaranteed consistency (convergence of estimates to true parameters) of the open-loop methods for uniform asymptotically stable systems under the persistently exciting inputs <cit.>. 
 

Besides consistency, the rate at which the true values are recovered is another important aspect of system ID settings. In <cit.>, an upper bound for the rate of convergence is provided for a specific class of ID schemes. It is also shown that for LTI systems an adaptive observer can achieve exponential convergence <cit.>.








 

In this paper, we consider the problem of closed-loop ID of a linear stochastic system with quadratic cost. We seek to characterize feedback policies achieving high convergence rates with low input costs.  

The problem we study is generally known as the optimal input design (see, e.g., <cit.> and the references therein) which has been extensively studied in the literature.

A large portion of input design methods for linear systems is in frequency domain and is based on the asymptotic theory. This theory states that the  quadratic predictive estimator, under some mild assumption, converges to the true parameters and   the covariance of estimation error decays linearly in number of samples T as 1/T P in the limit of T→∞, where P is referred to as asymptotic covariance matrix (see <cit.> for more details).

 It is shown for the open-loop ID of linear systems in <cit.> that the inverse of asymptotic covariance matrix is an affine function of the input power spectrum. Based on this affine relation, it is shown in <cit.> that by parametrizing the input power spectrum and imposing  the power constraint via Parseval's theorem, the input design problem can be formulated as an LMI. A deeper variance analysis for both open-loop and closed-loop ID of Box-Jenkins models is provided in <cit.>.
In the time domain approach, it is proposed in <cit.> to maximize, in some sense, the Fisher information matrix based on the recognition that the inverse of the Fisher information is a lower bound on the achievable covariance matrix of an unbiased estimator <cit.>. 

 



Recently, non-asymptotic results are derived in <cit.>  and <cit.> for ID of linear systems  based on non-stationary statistical analysis. In this approach, ID is performed via Gaussian excitation through independant rollouts which are long sequences of inputs.  The upper bound for the estimation error of the parameters is then computed, using only the last measurement of each rollout.


 In the reinforcement learning literature, it is common to analyze the rate at which the optimal policy is reconstructed in terms of regret. Regret is defined to be the performance deviation from the controller designed based on the true value of the parameters. For instance, <cit.> proposed to use the optimism in the face of uncertainty (OFU) principle, by propagating the confidence ellipsoid on the true parameters. 
The exponential dependence of regret on the order of the system was later reduced to linear dependency under further sparsity constraints on the dynamics in <cit.>. A Thompson sampling-based learning algorithm with dynamic episodes was proposed in <cit.>, showing that the regret up to time T is bounded by O(√(T)) . 

To gain further insights on the best possible learning rate, in this paper we introduce a new perspective on the closed-loop ID problems through the lens of feedback channel coding theory. To relate the system ID problem with information theory, we adopt the D-optimality scalarization metric instead of regret as the learning performance. We first observe that the system ID problems with input cost and the communication over noisy channels with input power constraint  share the same spirit of information gain maximization.

Inspired by this observation, our analysis builds a bridge between system ID and channel coding theories.







 




 §.§ Contributions

We uncover a close kinship between  closed-loop ID schemes and communications schemes over the channels with feedback. This connection enables us to use the well established information-theoretic tools to provide non-stationary results, such as theoretical upper-bounds for the rate of convergence, and to specify the characteristics of optimal ID polices. More precisely, this paper makes the following contributions:

    
  * We show the equivalence between the system ID problem with the D-optimality criterion and a channel coding problem  under the existence of noiseless feedback. To the best of our knowledge, this equivalence has not been established in the existing literature. 
    
  * A correspondence between the rate of convergence and the feedback channel capacity (the temporal average of directed information from channel input to its output) is made. Based on this analogy, a relation between the excitation cost and the upper-bound of  the achievable reduction of estimation error within a finite number of ID steps, which is a non-stationary analysis, is provided. This upper bound shows that the convergence is at most linear. 
    
  * Although the existence of an ID scheme that achieves the provided upper-bound is not known currently, the conditions under which the derived upper bound is achieved are discussed. More precisely,  we show that the upper bound is tight if and only if the state sequence is temporally independent and the input distribution of the corresponding channel matches the capacity achieving distribution.  




 §.§ Notation and Convention
 
Random variables and their realizations are denoted by upper-case and lower-case symbols respectively. The notation X^t = (X_0,  X_1, … X_t) is used to denote the history of state X. h(Y):=-∫ p(y)log p(y)dy is used to denote the differential entropy of random variable Y. For a Gaussian random variable Y∼𝒩(y,Σ_y), h(Y)=1/2log(2π eΣ_y). Mutual information between random variables X and Y is denoted by I(X;Y):=h(X)-h(X|Y) = h(Y)-h(Y|X). 



§ PROBLEM FORMULATION


Consider the discrete-time linear  system

    X_t+1=A X_t+B U_t+W_t    X_0=0,
 
where X_t is a scalar-valued state and W_t∼𝒩(0, W) are i.i.d Gaussian random variables with variance W. We assume that the realization  of the parameter  Θ=[A   B]^⊺ is not directly observable but the prior belief about Θ is known to be Θ∼𝒩(θ̂_0, Π_0), where Π_0 ≻ 0.

We formulate the optimal ID for the system defined above as finding the deterministic excitation policy U_t=g_t(X^t) optimizing a performance metric for a learning rate we introduce shortly. By defining z_t :=[x_t   u_t]^⊺, the state transition can be written as x_t+1=z_t^⊺θ+ w_t for t=0, …, T-1. We recursively compute the minimum variance unbiased estimator of the system's parameter θ, by exploiting the l_2-regularized LSE algorithm based on the full observation of state history X^T. Regularized estimation is selected to enforce the numerical stability of the algorithm and to exploit the initial knowledge of θ. The best estimate can be characterized as <cit.>:

    θ̂_t=_θ ||θ-θ̂_0||_Π_0^-1^2+||𝒳_t-𝒵_tθ||_𝒲^-1^2,
 
 where 𝒵_t:=[z_0^⊺; …; z_t-1^⊺], 𝒳_t:=[x_1; …; x_t], and 𝒲:=WI. We define Π_t^-1:=∑_i=1^t z_i-1 W^-1z_i-1^⊺+Π_0^-1, which yields for all t that Π_t^-1=Π_t-1^-1+ z_t-1 W^-1z_t-1^⊺, starting from t=1. By the matrix inversion lemma, the optimal estimate and its error covariance can be recursively computed as:

 

    Π_t=    Π_t-1-Π_t-1 z_t-1 z_t-1^⊺Π_t-1/W+z_t-1^⊺Π_t-1z_t-1
    θ̂_t=    θ̂_t-1+ Π_t-1 z_t-1/W+z_t-1^⊺Π_t-1z_t-1 (x_t- z_t-1^⊺θ̂_t-1).



Note that this iteration corresponds to Kalman filter (KF) associated with the following system.


 
    Θ_t=Θ_t-1
        X_t=Z_t-1^⊺Θ_t +W_t-1,

 
 where θ̂_t = 𝔼[Θ|X^t=x^t] is the estimator of Θ_t and Π_t=𝔼[(Θ-θ̂_t)(Θ-θ̂_t)^⊺| X^t=x^t] is the associated error covariance. 
 
Nonsingularity of 𝔼[Z_t Z_t^⊺] guarantees the consistency (convergence to the true θ) <cit.>. However, it is worth noting that linear control policies U_t=kX_t do not satisfy this nonsingularity condition and cannot be deployed for deterministic closed-loop ID of linear systems. We measure the performance of  the closed-loop ID process based on the following excitation cost and information utility. 


 §.§ Excitation Cost

Inspired by the standard linear quadratic regulator (LQR), we adopt the following quadratic function of states and inputs:

    J_T:= 1/T( ∑_t=0^T-1𝔼[ q X^2_t+ r U^2_t]+q𝔼[X^2_T])

as the excitation cost, where q  and r are positive scalars.
 
 

 §.§ Information Utility

The information utility can be defined based on the so-called alphabetical design criteria <cit.>, which includes several scalarizations of error covariance such as A-optimality, D-optimality, E-optimality, and T-optimality for which the trace, the log-determinant, the largest eigenvalue, and the trace of the inverse of error covariance is adopted, respectively.

In this work, we select the D-optimality scalarization of the estimation error covariance with multiplicative factor 1/2 as a metric of information utility. This metric is equivalent to the  information-theoretic quantity of entropy. Specifically, the information utility obtained at time t is defined as the entropy reduction:

    F_t(X^t=x^t)=   1/2log(Π_t-1)-1/2log(Π_t) 
    
    =   h(Θ|X^t-1=x^t-1)-h(Θ|X^t=x^t).

The accumulated utility up to time T will be

    L_T(X^T=x^T):=∑_t=1^T F_t= 1/2log(Π_0)-1/2log(Π_T).

We stress that the information utilities F_t(X^t) and L_T(X^T) are in fact random variables, since Π_t is computed based on the realizations of  Z^t (or equivalently X^t) in (<ref>). If we consider the expectation of information utility as F̅_̅t̅:=𝔼[F_t], we have

    F̅_̅t̅   = ∫_Ω F_t(X^t=x^t)   dP_x^t
       = ∫_Ω(h(Θ|X^t-1=x^t-1)-h(Θ|X^t=x^t))   dP_x^t
       = h(Θ|X^t)-h(Θ|X^t-1).

The expectation of the accumulated information utility L̅_T:=𝔼[L_T] is equal to the mutual information I(Θ, X^T):

    L̅_T    = ∑_t=1 ^T F̅_t = ∑_t=1 ^T ( h(Θ|X^t)-h(Θ|X^t-1) ) 
       = h(Θ)-h(Θ|X^T)=I(Θ;X^T).


Mutual information L̅_T=I(Θ;X^T) provides a metric quantifying how much the uncertainty of Θ i.e., h(Θ) decreases in expectation after observing the realizations of X^T. Based on these interpretations, we respectively refer to F̅_t, L̅_T, L̅_T/T as information gain, cumulative information gain, and the rate of convergence in the following sections. 



 §.§ Characterization of Optimal Policy

Using the aforementioned measures of convergence and control performance, the main optimization problem considered in this paper can be formulated as:

    max_{U_t=g_t (X^t)}_t=0^T-1   I(Θ; X^T)
    s.t.     1/T(∑_t=0^T-1𝔼[q X^2_t+r U^2_t]+q𝔼[X^2_T]) ≤γ
       X_t+1=A X_t+B U_t+W_t.

We are also interested in the optimal time invariant policy U_t=g(X^t) for infinite horizon counterpart of the problem (<ref>) as 

    max_U_t=g_t (X^t)   lim inf_T→∞1/TI(Θ; X^T)
    s.t.     lim sup_T→∞1/T (∑_t=0^T-1𝔼[q X^2_t+r U^2_t]+q𝔼[X^2_T]) ≤γ
       X_t+1=A X_t+B U_t+W_t.




§ PROPOSED APPROACH 

The problem of maximizing mutual information with power constraints similar to  problems (<ref>) and (<ref>) has a long history in the study of communication over noisy channels. This similarity motivates us to bridge system ID and information theory by finding  channel coding problems analogous to (<ref>) and (<ref>). 

In this paper, in lieu of direct analysis, we model the system ID algorithm as a communication scheme over a Gaussian channel with feedback. To the best of our knowledge, this is the first time that such a connection is shown explicitly and the equivalence between the rate of convergence of the system ID and directed information between channel input and output is demonstrated.


We extend the existing results in the context of channel coding and investigate their implications such as theoretical bounds for the rate of convergence and the required conditions to achieve such bounds.



 §.§ Communication over a Noisy Channel with Feedback

In this section, we review basic results for the problem of communication over noisy channels with feedback, which has been studied extensively in the network information theory literature <cit.>.
Consider the problem of transmitting a message M over an additive white Gaussian noise (AWGN) channel with noiseless feedback depicted in Figure <ref>. 
The noise W_t is drawn i.i.d from a Gaussian distribution W_t∼𝒩(0, W) and it is independent of input signal Y^t. The output of the channel is X_t=Y_t+W_t.



The capacity of the AWGN channel with feedback, denoted by C_FB(P), is defined as the largest R∈ℝ_+ for which there exists an encoder-decoder pair such that the associated sequence of inputs {y_t(M,X^t-1)}_t=1^T for every message M ∈{1, 2, …, 2^TR} satisfies the power input constraint

    ∑_t=1^T𝔼_W[y^2_t(M,X^t-1)]≤ TP,

 and the message can be decoded by the decoder after T channel uses with diminishing probability of decoding error, i.e., lim sup_T→∞ P(M̂≠ M)=0. The reader is referred to <cit.> for further discussion. The next theorem exhibits a connection between the feedback channel capacity C_FB(P) and the directed information I(Y^T→ X^T)=∑_t=1^T I(Y^t;X_t|X^t-1) for Markov channels <cit.> with feedback.
 
 

For Markov channels with feedback, channel capacity is equal to the limsup of the temporal average of directed information from the channel input Y^T to its output X^T maximized over all causally conditioned
distributions P(Y^T||X^T-1):=Π_t=1^T P(Y_t|X^t-1) satisfying the power constraint i.e.,   
 
    C_FB(P)= lim sup_T→∞max_P(Y^T||X^T-1)
    s.t. 1/T∑_t=1^T𝔼[Y^2_t]≤ P1/T I(Y^T→ X^T).

 

 
    See <cit.> for proof. Similar results are derived for the finite state channels without inter-symbol interference in <cit.> and for  stationary nonanticipatory channels in <cit.>.


It is well known <cit.> that feedback does not increase the capacity of memory-less channels (including AWGN channels). Consequently, we have C_FB(P)=C(P), where C(P):=1/2log(1+P/W) is the capacity of AWGN channel without feedback <cit.>. Nevertheless, feedback helps to simplify the coding scheme and increases the achievable error exponent <cit.>.



§ MAIN RESULTS


In this section, we construct a channel coding problem which is equivalent to the online parameter ID problems (<ref>) and (<ref>). We then demonstrate that cumulative information gain  and the directed information from the input to the output of the equivalent communication scheme  are equal. Based on this equivalence and Theorem <ref>, it is shown that the asymptotic rate of convergence i.e. the solution to (<ref>) is upper-bounded by the capacity  of the corresponding channel C(P). Even for case (<ref>) with the finite number of ID steps where Theorem <ref> is not directly applicable, we show that the step-wise information gain is upper bounded by  C(P) and that the value of (<ref>) is upper bounded by TC(P).

We provide necessary and sufficient conditions to achieve these upper limits. More precisely, an excitation policy U_t=g_t(X^t) is shown to be optimal if and only if it is matched with the capacity achieving distribution for the corresponding channel and X_t is an independent random process. [It is currently not known if there exists a deterministic feedback policy U_t=g_t(X^t) that meets this optimality condition.] Finally, we prove that input power of the equivalent channel P is bounded and finite for every policy with finite control cost J. Therefore, cumulative information gain L̅_T is upper bounded by the linear function of TC(P) meaning that L̅_T/T=𝒪(1) and super-linear convergence is impossible.



 §.§ Equivalent Channel Formulation

A system ID scheme with a feedback policy U_t=g_t(X^t) can be modeled  as communication system over a Gaussian channel with noiseless, one-step delayed feedback as depicted in Figure <ref>. The sensor model (<ref>) is interpreted as an AWGN channel over which the message θ is communicated. The decoder tries to estimate θ based on the sequence X^T of the channel outputs using (<ref>).



The following theorem states that the aforementioned measure of convergence is equivalent to directed information between the channel input and output. This theorem also indicates a close connection between the parameter ID problems (<ref>) and (<ref>) and the feedback channel coding problem (<ref>).


 
 
For any deterministic control policy U_t=g_t(X^t), we have

    I(Θ; X^T)= I(Y^T → X^T),

and 𝔼[(Π_T)]≥(Π_0)2^-2I(Y^T→ X^T).
 

By the chain rule, we have


    I(Θ;X^T)   =∑_t=1^T I(Θ; X_t| X^t-1)
       =∑_t=1^T( h( X_t| X^t-1)-h(X_t|Θ, X^t-1))
       =∑_t=1^T( h(X_t| X^t-1)-h(X_t|Y^t, X^t-1))
       =∑_t=1^T I(Y^t;X_t|X^t-1)= I(Y^T→ X^T),


where from (<ref>) to (<ref>) we used the fact that the coding scheme is a 1-1 causal mapping from Y^t to (Θ,X^t-1). Additionally, 1/2log(𝔼[(Π_T)]) ≥1/2𝔼[log(Π_T)]=1/2log(Π_0)-I(Y^T→ X^T), which yields 𝔼[(Π_T)]≥(Π_0)2^-2I(Y^T→ X^T).




 §.§ Upper Bound Analysis


The equivalent channel representation of the system ID problem (Figure <ref>) is a specific example of general communication schemes over the additive Gaussian channel depicted in Figure <ref>. Therefore, the maximum directed information achievable by general feedback scheme is an upper bound to the directed information achievable by system ID scheme. In particular, C_FB(P) characterized by Theorem <ref> provides an upper bound to the asymptotic learning rate i.e., lim_T→∞L̅_T/T or the solution to (<ref>).

These asymptotic results may not be sufficient to analyze the non-stationary performance when only a finite number of ID steps are performed. In the next lemma, we derive an upper-bound for obtainable information gain in each step, i.e., an upper bound for (<ref>). Note that

    F̅_t=   I(Θ; X^t)-I(Θ; X^t-1)
    
    =   I(Y^t→ X^t)-I(Y^t-1→ X^t-1)=I(Y^t;X_t|X^t-1).

Assuming a step-wise energy constraint, 𝔼[Y^2_t]≤ P_t, we intend to solve

    max_P(Y_t|X^t)     I(Y^t;X_t|X^t-1)
    s.t.     𝔼[Y^2_t]≤ P_t.



For any coding-decoding scheme that  satisfies 𝔼[Y^2_t]≤ P_t, the increase in the directed information i.e.,  I(Y^t;X_t|X^t-1) is bounded from above by C(P_t). Consequently,  
    L̅_T=I(Θ;X^T)=I(Y^T→ X^T) ≤∑_t=1^T C(P_t).




    I(Y^t;X_t|X^t-1)
       =h(X_t| X^t-1)-h(X_t|Y^t, X^t-1) 
       ≤ h(X_t)-h(W_t-1)
        = h(Y_t+W_t-1)-h(W_t-1)
       ≤1/2log (1+P_t/W)=C(P_t),


where from (<ref>) to (<ref>) we  used the fact that conditioning won't increase the entropy i.e. h(X_t|X^t-1)≤ h(X_t). The inequality in (<ref>) comes from the non-Gaussianity of Y_t and the power constraint in (<ref>). Notice that Y_t and W_t-1 are independent. 


 


For the problem with power constraint

    max_P(Y^T||X^T)     I(Y^T→ X^T)
    s.t.     1/T∑_t=1^T𝔼[Y^2_t]≤ P,

we have L̅_T= ∑_t=1^T F̅_t ≤∑_t=1^T C(P_t) ≤ T C(P), where P_t denotes channel input power at time t. The final inequality comes from the fact that C(.) is concave and 1/T∑_t=1^T P_t=P. This corollary extends the asymptotic result mentioned in Theorem <ref> to cases with finite T.
 

Corollary <ref> states that the solution to (<ref>) is bounded from above by TC(P). In the following sections, we discuss the conditions under which this upper bound is achievable and build the connection between the excitation cost and the input power of the equivalent channel.  



 §.§ Tightness of the Upper Bound

If we assume Y_t is Gaussian and we make full use of power budget i.e. 𝔼[Y^2_t]= P_t, the inequality (<ref>) holds with equality. The inequality in (<ref>) is also tight if X_t and X^t-1 are independent. This tightness, presuming the (<ref>) holds with equality, 
proves that a coding scheme achieves the maximum possible increase of the information gain C(P_t) if and only if X_t and X^t-1 are independent. This supports the idea that the excitation strategy is optimal (most informative) if it results in statistically independent observations X_t.

In the field of information theory, the fact that a coding-decoding scheme which achieves the channel capacity will produce a sequence of temporally independent channel output is not a new result. For example, it is shown in <cit.> that the well-known Schalkwijk-Kailath (SK) coding scheme induces independent  outputs. 

In our ID setting, the upper bound can be achieved if and only if X_t is an independent process and the input distribution in the equivalent channel matches with one of the capacity achieving distributions like the SK scheme. Unfortunately, our equivalent coding scheme has a restricted structure and no choice of excitation policy U_t=g_t(X^t) recovers SK scheme. Therefore, the SK scheme cannot be directly implemented in our ID setting.



 §.§ Infeasibility of Super-linear Convergence

In previous sections, the provided bounds are accounting for the channel input power P and not the excitation cost J. Establishing an explicit relation between the excitation cost and the channel input power for general excitation policy U_t=g_t(X^t) is nontrivial. However, it is shown in the next theorem that P is finite under any excitation scheme that yields a finite J.


For any excitation policy 

    P_T≤2/q J_T+2W.


    


    P_T    =1/T∑_t=1^T 𝔼[Y_t^2]=1/T∑_t=1^T 𝔼[(X_t-W_t-1)^2]
       ≤1/T∑_t=1^T 𝔼[2 X^2_t+2 W^2_t-1] 
    ≤2/q J_T+2W.


From (<ref>) to (<ref>), we have used Cauchy-Schwarz inequality. Last inequality (<ref>) is trivial result of  (<ref>).


Theorem <ref> implies that for any deterministic closed-loop system ID scheme with bounded excitation cost, C(P) is also bounded and thus L̅_T=𝒪(T), meaning that the scheme can at most achieve a linear convergence. To the best of our knowledge, the infeasibility of super-linear convergence was not shown previously for finite number of ID steps T. Theorem <ref> completes our analysis and establishes an explicit relation between the excitation cost and the information gain as L̅_T≤ TC(2/q J_T+2W).



§ NUMERICAL DEMONSTRATION

In this section, we consider an example of a linear system with initial belief as Θ∼𝒩([0 0]^⊺, I_2) and W=0.1 in infinite horizon limit, where the true parameters are θ=[a   b]^⊺=[0.9   1]^⊺. The excitation policy is adopted as U_t=kX_t(1+sin(X_t)) for different stable values of linear feedback gain k∈(-1.4,-0.9), where the stability can be demonstrated by Popov Criterion <cit.>. Based on the data from Monte Carlo simulation of the system averaged for 1000 realizations, an empirical comparison between the upper bound for the asymptotic convergence rate and actual convergence rate as functions of channel input power P is provided in Fig. <ref>. 
 
As demonstrated in Fig. <ref>, the  naive selection of excitation policy will result in poor performance for our choice of U_t, there is a noticeable gap between the provided upper bound and the actual rate of convergence.




§ CONCLUSION AND FUTURE WORK

We studied a fundamental limitation of system ID schemes of linear systems under quadratic control costs via deterministic excitation policies. We show that under specific choice of information utility, the closed-loop ID problem can be modeled as Gaussian channel coding problem with feedback. With this connection, we showed that the rate of convergence (information utility obtained per ID step) is bounded from above by capacity of the equivalent channel. This implies that the convergence is always sublinear or linear at best. Finally, it was shown that a feedback policy achieves the provided upper-bound if and only if the input distribution to the corresponding channel matches the capacity achieving distribution and the characteristics of the capacity achieving schemes like independence of output process were discussed.


Our future research will focus on discovering, or examining the existence of, policies that satisfy this optimality condition. If such optimal policy does not exist, the main question will be how to narrow the gap between the achievable convergence rate and its upper bound. Generalization of the modeling and the results over the space of non-deterministic (stochastic) policies is another interesting future direction. We will also investigate the relationship between information utility metric considered in this paper and regret.     

-12cm   
                                  
                                  
                                  
                                  
                                  

















IEEEtran



