On Convergence of some Gradient-based Temporal-Differences Algorithms for Off-Policy Learning
Huizhen Yu
cs.LG math.OC 90C40, 62L20, 68W40
 

Convergence of some Gradient-based TD Algorithms





On Convergence of some Gradient-based Temporal-Differences Algorithms for Off-Policy LearningThis research was supported by a grant from Alberta Innovates—Technology Futures.
    Huizhen YuRLAI Lab, Department of Computing Science, University of Alberta, Canada ()
    
==============================================================================================================================================================================




We consider off-policy temporal-difference (TD) learning methods for policy evaluation in Markov decision processes with finite spaces and discounted reward criteria, and we present a collection of convergence results for several gradient-based TD algorithms with linear function approximation. 
The algorithms we analyze include: (i) two basic forms of two-time-scale gradient-based TD algorithms, which we call GTD and which minimize the mean squared projected Bellman error using stochastic gradient-descent; (ii) their “robustified” biased variants; (iii) their mirror-descent versions which combine the mirror-descent idea with TD learning; and (iv) a single-time-scale version of GTD that solves minimax problems formulated for approximate policy evaluation. We consider primarily constrained algorithms which confine their iterates in bounded sets; for the single-time-scale GTD algorithm, we also analyze its unconstrained version. 

We derive convergence results for three types of stepsizes: constant stepsize, slowly diminishing stepsize, as well as the standard type of diminishing stepsize with a square-summable condition. For the first two types of stepsizes, we apply the weak convergence method from stochastic approximation theory to characterize the asymptotic behavior of the algorithms, and for the standard type of stepsize, we analyze the algorithmic behavior with respect to a stronger mode of convergence, almost sure convergence. 
Our convergence results are for the aforementioned TD algorithms with three general ways of setting their λ-parameters: (i) state-dependent λ; (ii) a recently proposed scheme of using history-dependent λ to keep the eligibility traces of the algorithms bounded while allowing for relatively large values of λ; and (iii) a composite scheme of setting the λ-parameters that combines the preceding two schemes and allows a broader class of generalized Bellman operators to be used for approximate policy evaluation with TD methods.


Keywords:
Markov decision processes; approximate policy evaluation; reinforcement learning; temporal-difference methods; importance sampling;  stochastic approximation; optimization; convergence








§ INTRODUCTION


We consider off-policy temporal-difference (TD) learning methods for policy evaluation in Markov decision processes (MDPs) with finite spaces and discounted reward criteria. Off-policy TD learning extends on-policy model-free TD learning <cit.> (see also the books <cit.>) to cases where stationary policies of interest are evaluated using data collected without executing the policies.
It is more flexible than on-policy learning and can be useful not only as a computational tool to solve MDPs, but also as an aid to building experience-based knowledge representations for autonomous agents in AI applications <cit.>.
The specific class of algorithms that we consider in this technical report is the class of gradient-based off-policy TD algorithms with linear function approximation. Our purpose is to analyze several such algorithms proposed in the literature, and present a collection of convergence results for a broad range of choices of stepsizes and other important algorithmic parameters.

The algorithms that we will analyze include the following:



  * Two two-time-scale gradient-based TD algorithms proposed and studied by Sutton et al. <cit.> and Maei <cit.>. These two algorithms use stochastic gradient-descent to minimize the mean squared projected Bellman error, a convex quadratic objective function, for approximate policy evaluation, thereby overcoming the divergence issue in off-policy TD learning. They have been called GTD2, TDC, as well as GTD(λ) in the early works just mentioned. Here we shall refer to them as GTDa and GTDb, respectively, and refer to both algorithms as GTD algorithms.

  * A single-time-scale version of GTDa that solves minimax problems formulated for approximate policy evaluation. 
This algorithm was also considered in those early works on GTD just mentioned. However, that it is trying to solve a minimax problem equivalent to the projected-Bellman-error minimization was pointed out later by Liu et al. <cit.> (see also Mahadevan et al. <cit.>). The latter viewpoint facilitates convergence analysis of the algorithm, by placing it in a more general class of stochastic approximation algorithms for solving minimax problems. 

  * The mirror-descent versions of GTD and TD. Combining the mirror-descent idea of Nemirovsky and Yudin <cit.> with TD learning was proposed by Mahadevan and Liu <cit.> (see also <cit.>). 

  * “Robustified” biased variants of the preceding algorithms. These algorithms use a “robustification” procedure to mitigate the high-variance issue in off-policy learning, at the price of introducing biases in this procedure. They are similar to the biased variant algorithms considered by the author <cit.> for the emphatic TD (ETD) algorithm proposed by Sutton et al. <cit.>. In the present context, as we will show, for two-time-scale GTD algorithms, the variant algorithms can be viewed as approximate gradient algorithms, and for the single-time-scale GTDa, its variant tries to solve minimax problems that approximate the ones GTDa tries to solve.


We will analyze primarily constrained algorithms which confine their iterates in bounded sets. Only for the single-time-scale GTDa algorithm, we will also analyze its unconstrained version under certain conditions.

We will present convergence results for three types of stepsizes: constant stepsize, slowly diminishing stepsize,  as well as the standard type of diminishing stepsize with a square-summable condition. For the first two types of stepsizes, we apply the weak convergence method from stochastic approximation theory <cit.> to characterize the asymptotic behavior of the algorithms. For the third, standard type of stepsize, we analyze the algorithmic behavior with respect to a stronger mode of convergence, almost sure convergence, by using general results on stochastic approximation <cit.>.

Our convergence results are for the aforementioned TD algorithms with three general ways of setting the λ-parameters in TD learning: 



  * state-dependent λ <cit.>; 

  * a case of history-dependent λ as proposed recently by Yu et al. <cit.>,  which can keep the eligibility traces in the off-policy TD algorithms bounded while allowing for relatively large values of λ; 

  * a composite scheme of setting the λ-parameters <cit.>, which combines the preceding two schemes and allows a broader class of generalized Bellman operators to be used for approximate policy evaluation with TD methods.



To our knowledge, for off-policy gradient-based TD algorithms with linear function approximation, there are few prior convergence results that address the case of general nonzero λ-parameters. Although such algorithms with constant or state-dependent λ have been proposed and investigated (see e.g., <cit.>), the analyses given earlier <cit.> have only proved convergence for the case where λ = 0 and the data consist of i.i.d. state transitions. But the assumption on i.i.d. data is unrealistic for reinforcement learning even in the case λ = 0. Moreover, these analyses cannot be extended to the case of positive λ, where the algorithms need to use non-i.i.d. off-policy data in order to gather information about the multistep Bellman operator with respect to which the mean squared projected Bellman error is defined. To our knowledge, the only prior convergence result that applies to off-policy data is given by Karmakar and Bhatnagar <cit.>. It is a convergence result for the two-time-scale TDC algorithm (GTDb as we call it) with the standard type of diminishing stepsize, as an application of the theoretical results developed in their work <cit.> for two-time-scale differential inclusions. The result is for λ = 0, although its arguments can be applied in the case where λ is small enough so that the eligibility traces produced in the algorithm are bounded.

Our results on gradient-based TD algorithms differ from that of <cit.> not only in the range of algorithms and parameter settings they cover, but also in the proof approaches by which they are derived. Specifically, we combine the ordinary-differential-equation (ODE) based proof methods in stochastic approximation theory <cit.> with special properties of the eligibility traces and the ergodicity of the joint state and eligibility trace process under various settings of the λ-parameters mentioned above. Those properties were derived in the author's early works <cit.> for state-dependent λ and in the recent work <cit.> for the special case of history-dependent λ mentioned above. The properties of the joint state and eligibility trace process are not considered in <cit.> since it treats the case λ = 0 where the eligibility traces are simply functions of states, but these properties are important for convergence analysis of those TD algorithms that use general nonzero λ-parameters. The ODE-based line of analysis we use is less general than the differential inclusion-based method studied in <cit.>, however. As future work, it can be worthwhile to use the latter approach to handle even more flexible ways of choosing history-dependent λ than the one we consider in this work.

Another difference between our work and <cit.> is that we analyze primarily constrained algorithms, as mentioned earlier. The result given in <cit.> is for the unconstrained two-time-scale TDC under the assumption that the iterates are almost surely bounded. With constraints, we do not need such assumptions, and instead we could simply require that the constraint sets are large enough so that the algorithms can estimate the gradients correctly. The presence of constraint sets helps us avoid some theoretical difficulties in convergence analysis. However, extra work is also needed to ensure that the constraint sets do not interfere with the algorithms to prevent them from achieving the original goals that they are designed for. We take care of such issues in our analyses, especially for mirror-descent GTD/TD algorithms and for the single- and two-time-scale GTDa algorithms, which are not as straightforward as the two-time-scale GTDb algorithm.


This technical report is organized as follows. 
Section <ref> is about the preliminaries: We first describe the off-policy policy evaluation problem and the two two-time-scale GTD algorithms. We then explain the role of the λ-parameters in TD learning, and discuss the properties of the eligibility traces and the properties of the joint state and eligibility trace process, in order to prepare the stage for convergence analyses.
In Sections <ref>-<ref>, we present convergence results for slowly diminishing stepsize and constant stepsize, which are derived with the weak convergence method. Section <ref> is for the two two-time-scale GTD algorithms and their biased variants. Section <ref> is for the mirror-descent GTD/TD algorithms, as well as a single-time-scale GTDa algorithm and its biased variant, both of which solve minimax problems for policy evaluation. In this section we also use the minimax problem formulation to strengthen the results of Section <ref> for the two-time-scale GTDa algorithm and its biased variant.
In Section <ref>, we consider standard stepsize conditions and present almost sure convergence results for 
both the two-time-scale and single-time-scale algorithms, including a result for the unconstrained single-time-scale GTDa for certain choices of the λ-parameters. We then conclude the paper in Section <ref> with a brief discussion about these results and open questions.
For quick access to our convergence results in Sections <ref>-<ref>, the convergence theorems will be listed at the beginning of each of those sections.




§ PRELIMINARIES
 

In this section we first introduce the off-policy policy evaluation problem, and describe two basic forms of the gradient-based TD algorithms which were proposed and studied in <cit.>. 
We then explain generalized Bellman operators, how they relate to the λ-parameters of the TD algorithms and to the objectives of these algorithms.
We also specify two ways of choosing these λ-parameters, for the algorithms that we will analyze in the paper.
These background materials will be given in Section <ref>. 
In Section <ref>, the second half of this section, we present materials to prepare the stage for analyzing the gradient-based TD algorithms as stochastic approximation algorithms in the rest of this paper. These materials are about the state-trace process, the random process that underlies and drives the TD algorithms, and its important properties for convergence analysis (including, among others, ergodicity and uniform integrability properties). 



 §.§ Problem Setup and Two Basic Forms of Algorithms
 

The gradient-based TD algorithms we consider belong to the class of model-free, temporal-differences based learning algorithms for evaluating a stationary policy in a Markov decision process (MDP). We shall consider MDP with finite spaces. For the purpose of this paper, however, we do not need the full MDP framework.
[For references on MDP, see the excellent textbook <cit.>.]
It is adequate to consider two Markov chains on a finite state space .
[The states in these Markov chains need not correspond to the states of the MDP; they can correspond to state-action pairs. Depending on whether one evaluates the value function for states or state-actions pairs in the MDP, the Markov chains here correspond to slightly different processes in the MDP. (For the details of these correspondences, see <cit.>.) But the analysis is the same, so for notational simplicity, we have chosen not to introduce action variables in the paper.]

The first Markov chain has transition matrix , and the second ^o.
Whatever physical mechanisms that induce the two chains shall be denoted by π and π^o, and referred to as the target policy and behavior policy, respectively. The second Markov chain we can observe; however, what we want is to evaluate the system performance with respect to (w.r.t.) the first Markov chain that we do not observe—the “off-policy” learning case.

The performance of the target policy π is defined w.r.t. a discounted total reward criterion as follows.
A one-stage reward function r_π : → specifies the expected reward r_π(s) at each state s ∈. 
Each state is also associated with a state-dependent discount factor γ(s) ∈ [0,1], s ∈. 
The expected discounted total rewards for each initial state s ∈ is defined by

    v_π(s) : = ^π_s [   r_π(S_0) +  ∑_n=1^∞γ(S_1) γ(S_2)  ⋯ γ(S_n)  · r_π(S_n) ].
 
Here the notation ^π_s indicates that the expectation is taken w.r.t. the Markov chain {S_n}_n ≥ 0 induced by π and with the initial state S_0 =s. The function v_π in (<ref>) is called the value function of π, and it is well-defined under Condition <ref>(i) given below, which we shall assume throughout the paper. 

Denote by  the || × || diagonal matrix with the discount factors γ(s) as its diagonal entries. 
[Conditions on the target and behavior policies]  



  (i)  is such that the inverse (I - )^-1 exists, and 

  (ii) ^o is such that for all s,s' ∈, ^o_ss' = 0 ⇒_ss'=0,  and moreover, ^o is irreducible.




The second part of this condition is for the behavior policy π^o that generates the observed Markov chain. It will be needed when we describe the off-policy learning algorithms. 
 
By standard MDP theory (see e.g., <cit.>), under Condition <ref>(i), v_π satisfies uniquely the linear equation (expressed in matrix/vector notation with v_π, r_π viewed as ||-dimensional vectors): 

    v_π = r_π +   v_π    (i.e.,  v_π = (I - )^-1 r_π).

It is known as the Bellman equation (or dynamic programming equation) for π. Besides this equation,
v_π also satisfies a broad family of generalized Bellman equations, which have v_π as their unique solution and like (<ref>), express v_π as the sum of two rewards terms, with the first term representing the expected rewards received prior to a certain (randomized stopping) time and the second term those received afterwards. (We shall discuss these equations further in the subsequent Section <ref>.)

TD algorithms compute v_π by solving such a Bellman equation for π. Which Bellman equation to solve is determined by certain parameters, which we call the λ-parameters, used by the algorithms in their iterative computation of eligibility traces (which are iterates that carry information about the past states).
We shall give more details about this correspondence between λ and the generalized Bellman equations in Section <ref>, after we describe two basic forms of the GTD algorithms. For now, we will focus on the overall structure of the computation problem tackled by the gradient-based TD algorithms. Think of v_π =  v_π as one generalized Bellman equation that an algorithm chooses to solve. The operator  is an affine operator on ^|S| and similar to (<ref>), can be expressed as

    v = r_π^(λ) +  v,     ∀  v ∈^|S|,
  
for a vector r_π^(λ)∈^|S| and a substochastic matrix . We shall refer to  as a generalized Bellman operator for π.
The TD algorithms we consider try to find an approximate solution to the linear equation 

    v =  v,
 
by solving an optimization problem on a lower dimensional space using linear function approximation. 
Let us describe first the approximation architecture and then the formulation of the optimization problem.

Let ϕ : →^d be a given function that maps each state to a d-dimensional feature vector (it will be taken for granted that ϕ is non-trivial; i.e., ϕ(s) ≠ 0 for at least one state s).
Write ϕ=(ϕ_1, …, ϕ_d), and denote the subspace spanned by these component functions ϕ_i by Ł_ϕ.
To approximate v_π, the TD algorithms look for some function v ∈Ł_ϕ that satisfies the generalized Bellman equation approximately: v ≈ v. The functions in the approximation subspace Ł_ϕ are parameterized as v(s) = ϕ(s)^θ, s ∈, for parameters θ∈^d.
(We treat ϕ(s) and θ as column vectors; the symbol ^ stands for transpose.) 
We do not require the functions ϕ_i to be linearly independent; for this reason, another subspace will be useful later. This is the subspace in ^d spanned by the feature vectors, {ϕ(s) | s ∈}; below we shall write it as {ϕ()} for short. In matrix notation, Ł_ϕ is the column space of the matrix Φ that has the feature vectors ϕ(s) as its rows; i.e.,

    Φ  = [ [     ⋮; ϕ(s)^;     ⋮ ]]     or    Φ^  = [ [    ⋯ ϕ(s)    ⋯ ]].

Any approximate value function v ∈Ł_ϕ can be written as v = Φθ for some θ∈^d—note that θ is uniquely determined by v if θ∈{ϕ()}.

Let us now describe the optimization problem that the gradient-based TD algorithms try to solve in order to find an approximation of v_π in Ł_ϕ.



  §.§.§ The objective function
 
To find a function v ∈Ł_ϕ with v ≈ v, the two original gradient-based TD algorithms, GTDa and GTDb, to be described shortly, try to minimize an objective function of the form

    J(θ) = 12Π_ξ ( v_θ - v_θ) ^2_ξ,     where  v_θ = Φθ,  θ∈^d.
  
Here Π_ξ denotes projection onto the approximation subspace Ł_ϕ w.r.t. the weighted Euclidean norm ·_ξ given by v_ξ^2 = ∑_s ∈ξ_s v(s)^2, for a positive ||-dimensional vector ξ with components ξ_s. 
The objective J(θ) measures the magnitude of the “Bellman error” v_θ - v_θ on the subspace Ł_ϕ. If the projected Bellman equation
v_θ = Π_ξ v_θ has a unique solution, the relation between this solution and v_π can be characterized using the oblique projection viewpoint and related approximation error bounds (see the early work <cit.> and a summary in more general terms given in the recent work <cit.>). In this paper, since our focus is on convergence properties of the algorithms and since the problem inf_θ J(θ) always has an optimal solution, we do not require the projected Bellman equation v_θ = Π_ξ v_θ to have a unique solution or any solution. (When it has no solution or multiple solutions, the quality of the approximate value function from the minimization of J(θ) could be a concern, though.)  

In this paper, we shall take ξ to be the invariant probability distribution of the Markov chain with transition matrix ^o induced by the behavior policy π^o (i.e., ξ^^o = ξ^); such a distribution exists and is unique under Condition <ref>(ii). 
This choice of ξ is mostly for notational simplicity: our analyses extend to cases where ξ does not coincide with the invariant distribution of ^o, but the algorithms in those cases have additional weighting terms and are notationally more cumbersome. 

Later we will also discuss objective functions of the form J(θ) + p(θ), where p(θ) is some smooth convex function that serves as a regularizer. It will be seen that to handle this additional p(θ) term, little extra effort is needed in the convergence analysis. So, for notational simplicity, we will take J(θ) to be the objective function in the first half of the paper, and discuss the regularized objective function after we have presented the main convergence proof arguments. One can also consider a mixed objective function by combing the projected Bellman errors for multiple Bellman operators, for instance, J_1(θ) + J_2(θ) with J_1, J_2 defined like J but for two different . The convergence analysis of the gradient-based TD algorithms for such mixed objectives is essentially the same as that for J, so we will focus on the latter for simplicity.

Let us now work out two expressions of ∇ J(θ), which are used respectively by the two GTD algorithms, before we describe these algorithms.
Let ⟨·, ·⟩_ξ denote the inner product on the Euclidean space (^||, ·_ξ); i.e., ⟨ v, v'⟩_ξ = ∑_s ∈ξ_s v(s) v'(s). (The notation ⟨·, ·⟩ will be used for the usual inner product in Euclidean spaces.) 
For i ≤ d, let Φ_i denote the i-th column of Φ, and θ_i the i-th component of θ. Since v_θ = Φθ = ∑_i=1^d θ_i Φ_i, the partial derivative of J w.r.t. each θ_i is 

    ∇_θ_i J(θ) = ⟨Π_ξ( v_θ - v_θ),  Π_ξ( - I) Φ_i ⟩_ξ.

(Recall that  is the substochastic matrix in the affine operator ; cf. (<ref>).) Observe two facts. First, for any v, v',

    ⟨Π_ξ v, Π_ξ v' ⟩_ξ = ⟨ v, Π_ξ v' ⟩_ξ = ⟨Π_ξ v, v' ⟩_ξ.

Second, for any v, there is a unique x ∈{ϕ()} with Φ x = Π_ξ v, and therefore, given θ, there is a unique solution x_θ to the linear equation (in x),

    Φ x = Π_ξ( v_θ - v_θ),    x ∈{ϕ()},
 
which is also the unique solution to the equivalent linear equation
[To see (<ref>) is equivalent to (<ref>), note that for any v, Φ x = Π_ξ v if and only if w.r.t. ⟨·, ·⟩_ξ, 
v - Φ x is perpendicular to the approximation subspace Ł_ϕ, which is true if and only if ⟨Φ_i, v - Φ x ⟩_ξ = 0 for all Φ_i, i ≤ d (since Ł_ϕ is the column space of Φ). The latter system of linear equations is the same as the first equation in (<ref>) when we set v =  v_θ - v_θ.]


    Φ^Ξ Φ x = Φ^Ξ ( v_θ - v_θ),    x ∈{ϕ()},

where Ξ denotes the || × || diagonal matrix with the components of ξ on its diagonal. 

Using the above facts, we can write

    ∇_θ_i J(θ) = ⟨Φ x_θ,  ( - I) Φ_i ⟩_ξ = x_θ^·Φ^Ξ( - I) Φ_i,
 
which gives the expression of the gradient as

    ∇ J(θ) = ( Φ^Ξ( - I) Φ)^ x_θ.

Alternatively, we can write 

    ∇_θ_i J(θ)     = - ⟨Π_ξ( v_θ - v_θ),  Π_ξΦ_i ⟩_ξ  + ⟨Π_ξ( v_θ - v_θ),  Π_ξΦ_i ⟩_ξ
        =  - ⟨ v_θ - v_θ,  Φ_i ⟩_ξ + ⟨Φ x_θ,  Φ_i ⟩_ξ
        = - Φ_i^Ξ( v_θ - v_θ) + x_θ^·Φ^Ξ Φ_i
 
(where in the second equality we used Π_ξΦ_i = Φ_i). This gives another expression of the gradient:

    ∇ J(θ) = - Φ^Ξ( v_θ - v_θ) + ( Φ^Ξ Φ)^ x_θ.

In principle one can derive other gradient expressions and formulate corresponding gradient-based algorithms; we shall, however, focus on the expressions (<ref>) and (<ref>) only.




  §.§.§ GTDa and GTDb
 

We now describe two basic forms of GTD algorithms. As mentioned earlier, they can only observe the Markov chain {S_n}_n ≥ 0 
induced by the behavior policy π^o, instead of the target policy π. Upon each state transition (S_n, S_n+1), they receive a random reward R_n+1 that is a function of the transition, r(S_t, S_t+1), plus a zero-mean finite-variance noise term whose distribution is determined by the state transition. The reward function r(·) relates to the target policy's one-stage reward r_π as r_π(s) = ^π_s [ r(s, S_1) ] for all s ∈.
The rewards {R_n} and the states {S_n} are all that the algorithms can observe.

Define ρ(s, s') = _ss'/^o_ss' for s, s' ∈. They are the importance sampling ratios that can be used to compensate for the differences in the dynamics of the two Markov chains. We assume that the algorithms know these ratios (this is the case for standard value function or state-action value function estimation, as well as for the simulation context where both  and ^o are known). 
To simplify notation, for n ≥ 0, we write 

    ρ_n = ρ(S_n, S_n+1),     γ_n=γ(S_n).

For any given approximate value function v on , we write δ_n(v) for the (scalar) temporal-difference term calculated based on the observed random transition (S_n, S_n+1) and reward R_n+1:

    δ_n(v) = ρ_n ( R_n+1 + _n+1 v(S_n+1) - v(S_n) ).

The conditional expectation of δ_n(v) given the history {S_0, …, S_n} measures the difference between the two sides of the Bellman equation (<ref>) for the state S_n when v_π in (<ref>) is replaced by v.

Using the states {S_n}, a sequence of eligibility trace vectors _n ∈^d is calculated iteratively by both GTD algorithms according to this formula:
given an initial _0 ∈^d, for n ≥ 1,

    _n = λ_n _n ρ_n-1 e_n-1 + ϕ(S_n).

Here λ_n ∈ [0,1], n ≥ 1, are the λ-parameters we referred to earlier. They are important parameters in TD learning. Not only do they affect the behavior of the algorithms, but they also determine the Bellman operator  appearing in the objective function J(θ). In the next subsection we shall describe the choices of these parameters that we consider in this paper, and explain what are the associated Bellman operators .

The eligibility traces (or traces, for short) are combined with temporal-differences terms by the algorithms to generate a sequence of iterates (θ_n, x_n) ∈^d ×^d, starting from some initial (θ_0, x_0).
In particular, let v_θ = Φθ as before. The first algorithm, GTDa, calculates the sequence iteratively according to

    θ_n+1    = θ_n + α_n  ρ_n (ϕ(S_n) - _n+1ϕ(S_n+1) ) ·_n^ x_n  ,  
    
       x_n+1    = x_n + β_n (_n δ_n(v_θ_n) - ϕ(S_n) ϕ(S_n)^ x_n).
 
The second algorithm, GTDb, has the same formula for {x_n}, but calculates {θ_n} according to

    θ_n+1 = θ_n + α_n (_n δ_n(v_θ_n) -   ρ_n (1 - λ_n+1)  _n+1ϕ(S_n+1) ·_n^ x_n ).

In the above {α_n} and {β_n} are stepsizes, with α_n < < β_n. (We will consider a broad range of stepsizes and we defer the precise stepsize conditions till later sections where we analyze the algorithms.) Although it can be hard, for readers unfamiliar with TD algorithms, to see how the preceding formulae relate to the gradient ∇ J(θ), the GTD algorithms do correspond to applying gradient-descent to minimize J(θ) for the two gradient expressions (<ref>) and (<ref>), respectively.
The idea of the two algorithms, roughly speaking, is to let the θ-iterates evolve at a slow time-scale and the x-iterates at a fast time-scale. 
As θ_n varies slowly, the fast-evolving x-iterates aim to track the solution x_θ of (<ref>) for θ = θ_n, the “current” θ-iterate. The information about x_θ is then used to perform stochastic gradient-descent in the θ-space to minimize J(θ).

To gain more intuition and insights about the algorithms, we suggest the reader consult the original derivations given in e.g., <cit.>.
[GTDa and GTDb here are called  GTD2 and TDC, respectively, in <cit.>, for the case λ = 0. For the case of nonzero λ, GTDb here is called GTD(λ) for value function estimation and GQ(λ) for state-action value function estimation in <cit.>. More precisely, the GQ(λ) algorithm does not coincide exactly with GTDb for estimating state-action values; it differs from the latter in a term with a conditional mean of zero, which does not make any difference in convergence analysis, however.]

We need to also point out, however, that these derivations have issues. For example, they involve various expectations such as [ _n δ_n(v) ] that are taken for granted to be independent of the time n. But before we know the properties of the process {(S_n, _n)}, it is not clear w.r.t. which probability distribution one can define [ _n δ_n(v) ] so that it is independent of n. Indeed, even with constant λ_n = λ for all n and a stationary state process {S_n}, it does not immediately follow just from these that {(S_n, _n)} has to have a stationary distribution, let alone a unique one. 

We shall discuss the properties of the state-trace process {(S_n, _n)} in Section <ref>. As can be seen from (<ref>), this process depends on the λ-parameters in the algorithms. So let us first explain the relation between the λ-parameters and the generalized Bellman operators , since this will at least let us complete the definition of the objective function J(θ). We will then focus the discussion on the process {(S_n, _n)} and its many properties that will be needed—actually, for some choices of the λ's that we consider,  also depends on the property of the process {(S_n, _n)}. As to the connection between the above algorithms and the gradient ∇ J(θ), it will be seen first in Prop. <ref>, Section <ref>, after we explain  and the ergodicity property of {(S_n, _n)}.




  §.§.§ Choices of λ-parameters and associated Bellman operators
 

We will consider in this paper three ways of setting the λ-parameters in (<ref>) for the trace iterates {_n}. We discuss the first two in this subsection (the third one builds upon them and will be discussed in Section <ref>.) 
These two choices are state-dependent λ and a case of history-dependent λ with special properties: 




  (i) State-dependent λ <cit.>, where λ_n = λ(S_n) for a given function λ : → [0,1].

  (ii) History-dependent λ as introduced in <cit.>, where we choose λ_n based on the previous trace _n-1 directly, in order to make {_n} bounded. In particular, we introduce additional memory states y_n, n ≥ 0, to summarize the history of past states up to time n. We let y_n evolve in a Markovian way and choose λ_n based on the current y_n and the previous trace _n-1 as follows: 

    y_n = f(y_n-1, S_n),     λ_n=λ(y_n, _n-1)
  
where f and λ are some given functions, whose properties will be given shortly.


Although state-dependent λ is a special case of history-dependent λ (e.g., take y_n = S_n in (ii)), generality is not our purpose here. The primary purpose of choosing λ based on (ii), as explained in <cit.>, is to exploit the flexibility of history-dependent λ to bound the traces {_n} easily, while allowing for a large range of λ values. The latter is important because the choice of λ affects the choice of the generalized Bellman operator  appearing in the objective function J(θ), and in turn, this choice of  affects approximation error. Bounding the traces is also important, as it facilitates convergence of the algorithms. Thus, instead of the most general history-dependent λ, we shall focus on the special case (ii) under additional conditions studied in <cit.>. These conditions concern the memory states and the function λ(·), and they will be needed in the next subsection to ensure certain desired properties of the state-trace process:

[Evolution of memory states in (<ref>)]  
The memory states y_n take values in a finite space , and under the behavior policy π^o, the Markov chain {(S_n, y_n)} on × has a single recurrent class.
 

[Condition for λ(·) in (<ref>)]  
The function λ : ×^d → [0,1] in (<ref>) satisfies the following. For some norm · on  and for each memory state y  ∈:



  (i) For any , ' ∈^d, λ(y, )   - λ(y, ')  ' ≤ - '.

  (ii) For some constant _y, γ(s') ρ(s, s') ·λ(y, )   ≤_y for all ∈^d and all possible state transitions (s, s') that can lead to the memory state y.



Several existing off-policy algorithms, Tree-Backup <cit.>, Retrace <cit.> and ABQ <cit.>, choose state or state transition-dependent λ accordingly to keep the trace iterates bounded (in fact, these works motivated the history-dependent λ described above). Such choices of λ satisfy the above conditions, since one can simply let y be a state or state transition and let λ(y,) be a function of y only. One disadvantage of these choices, however, is that they are too conservative and often result in small values of λ. A few examples of memory states and function λ(y,) that satisfy the above conditions are given in <cit.>.

In the rest of this subsection, let us explain, at a level of detail adequate for the purpose of this paper, what are the generalized Bellman operators  for the target policy associated with the preceding two ways of setting λ.
As mentioned above, corresponding to different choices of λ are different Bellman operators  in the objective function J(θ). With different , the solutions of the minimization problem inf_θ J(θ) are also different and can have different approximation biases (see e.g., <cit.>). 
For the purpose of this paper, however, the details of  do not matter, because our focus is on the stochastic approximation aspects of the algorithms and what we care about is whether the average dynamics of the algorithms can be characterized by mean ODEs that are related to the minimization of J(θ) for the associated . The two cases of choosing λ mentioned above share many properties in common. Once their common properties are made clear, as will be done here and in the next subsection, the two cases can be treated together in most of our convergence analysis of the algorithms.
For this reason, regarding the Bellman operators , we shall recount only the facts that we will need in this paper (for a detailed study, see the paper <cit.>).

As mentioned earlier, different choices of λ induce different Bellman operators  for the target policy.
They are members of a broad family of generalized Bellman operators T_τ associated with randomized stopping times τ <cit.>, which are all contractive operators that have v_π as their unique fixed point (see <cit.>). Such an operator takes the general form of

    (T_τ v)(s) = ^π[ R^τ + γ_1^τ v(S_τ) | S_0 = s ],      s ∈,  ∀  v ∈^||,
  
where ^π denotes expectation over the randomized stopping time τ and the states {S_t} generated according to the target policy, R^τ is the total discounted rewards received prior to the time τ of stopping, S_τ is the state at time τ, and γ_1^τ is a shorthand for the product of discount factors γ_1 ⋯γ_τ. These generalized Bellman equations and operators are a consequence of the strong Markov property of Markov chains <cit.>. We refer the reader to <cit.> for a fuller account of the framework and the mathematical notions and derivations involved.

For state-dependent λ,  corresponds to a randomized stopping time τ for the Markov chain {S_n} under the target policy, where τ is such that 

    τ≥ 1,      (τ = n |τ > n-1, S_0, …, S_n) = 1 - λ(S_n)    for n ≥ 1.

(I.e., the probability of stopping at time n given that the system has not stopped yet is 1 - λ(S_n).) The associated operator  can be expressed in several equivalent ways. Besides the general form (<ref>) above, we can write  as
[This follows from (<ref>) by taking conditional expectation over τ.] 


    ( v)(s) = ^π[  ∑_n=0^∞λ_1^n ·γ_1^n   r_π(S_n) + ∑_n=1^∞λ_1^n-1 (1 - λ_n) ·γ_1^n   v(S_n)  |   S_0 = s ],    s ∈,  ∀  v ∈^||,
  
where we used the shorthand notation λ_1^n = λ_1 ⋯λ_n with λ_1^0=1. 
We can also write  explicitly in terms of λ and the model parameters as
[This follows from (<ref>) by a direct calculation using the definition λ_n = λ(S_n).] 


    v = (I - P ΓΛ)^-1 r_π + (I - P ΓΛ)^-1 P Γ (I - Λ)   v,

where Λ denotes the ||× || diagonal matrix with diagonal entries λ(s), s ∈. (Thus, the substochastic matrix in (<ref>) has the explicit expression = (I - P ΓΛ)^-1 P Γ (I - Λ).)

In the case of history-dependent λ, it is shown in <cit.> under Conditions <ref>-<ref> that  is also a generalized Bellman operator (for the target policy) corresponding to a certain randomized stopping time τ. But this random time τ now depends on the behavior policy in a much more complex way than in the case of state-dependent λ. Among others, it depends on the dynamics of the traces under the behavior policy. As such, we generally cannot write  explicitly in terms of the model parameters P, P^o and the function λ. We express  in other ways, in order to relate it to the algorithms that employ history-dependent λ. In particular, an expression of  similar to (<ref>) will be useful in our subsequent analysis:

    ( v)(s) = ^π_ζ[  ∑_n=0^∞λ_1^n ·γ_1^n   r_π(S_n) + ∑_n=1^∞λ_1^n-1 (1 - λ_n) ·γ_1^n   v(S_n)  |   S_0 = s ],    s ∈,  ∀  v ∈^||.

In the above ^π_ζ denotes expectation with respect to the probability measure of the following process:



  * The states {S_n} are generated under the target policy π.

  * For n ≥ 1, the memory state y_n, the parameter λ_n, and the trace _n are calculated according to (<ref>) and (<ref>), respectively.
[The randomized stopping time τ is generated according to the following rule: τ≥ 1 and for n ≥ 1, (τ = n |τ > n -1, y_0, _0, S_0, S_1, …, S_n) = 1 - λ_n (i.e., the probability of stopping at time n given that the system has not stopped yet is 1 - λ_n, which is similar to the case of state-dependent λ). The random time τ does not appear in the expression (<ref>) of , because (<ref>) is an equivalent form of (<ref>) after taking conditional expectation over τ.]


  * The initial state, memory state and trace (S_0, y_0, _0) are distributed according to ζ, the unique invariant probability measure of the process {(S_n, y_n, _n)} under the behavior policy π^o.


The existence and uniqueness of the invariant probability measure ζ just mentioned is ensured under Condition <ref> on the two policies and Conditions <ref>-<ref> on the memory states and the function λ(·). Further details will be explained in the next subsection (see <cit.> for the derivations of the preceding results).

Another expression that will be useful later is an expression of the “Bellman error” v - v in terms of temporal-differences terms (this expression follows from (<ref>) by rearranging terms):

    ( v - v)(s) = ^π_ζ[  ∑_n=0^∞λ_1^n ·γ_1^n  ( r_π(S_n) + γ_n+1 v(S_n+1) - v(S_n) )  |    S_0 = s ],    s ∈,  ∀  v ∈^||.

The same expression (ignoring the subscript ζ of ) also holds for the case of state-dependent λ.





 §.§ Properties of State-Trace Process
 

The purpose of this subsection is to prepare the stage for analyzing the asymptotic behavior of the gradient-based TD algorithms using stochastic approximation theory.  We collect here important properties of the state-trace process that our subsequent analysis will rely on. 
Specifically, in Section <ref>, we first discuss ergodicity properties of the state-trace process. We then consider several functions on the state-trace space that appear in the TD algorithms, and we derive their expectations w.r.t. the stationary state-trace process, which can be related to expressions of the gradient ∇ J(θ). In Section <ref>, we include more properties of the trace iterates, which will be needed later in analyzing the average dynamics of the algorithms and proving their convergence.

Most of the results in this subsection were proved earlier by the author <cit.>. There are some small differences in the setup of the problem considered in those earlier works; these differences are nonessential and do not affect the conclusions obtained. For clarity, however, we will give additional details to bridge the gap, when this can be done quickly without repeating long proofs.



  §.§.§ Some notations and definitions


Let us first introduce some notations and definitions that we will need below and throughout the paper. In most of our analysis, we will treat the two cases of λ together. For brevity, let us collect the conditions given earlier for each case of λ in a single assumption.

  
Condition <ref> holds. In the case of history-dependent λ given in (<ref>), Conditions <ref>-<ref> also hold. 


By the state-trace process, we mean {(S_n, _n)} for the case of state-dependent λ, and {(S_n, y_n, _n)} (including the memory states y_n) for the case of history-dependent λ, generated under the behavior policy π^o. The state-trace process is a Markov chain with the weak Feller property—this means that, with X_n = (S_n, _n) or (S_n, y_n, _n) (depending on the case of λ), [ f(X_1) | X_0 = x] is a continuous function of x for any bounded continuous function f <cit.>. (Using the definitions of the traces and memory states, one can verify that this is the case.) Weak Feller Markov chains have nice ergodicity properties <cit.>, which helped us in obtaining some of the ergodicity properties of the state-trace process that will be discussed shortly.

Let (·) denote the indicator function. For each initial condition x of (S_0, _0) or (S_0, y_0, _0), define random probability measures μ_x, n, n ≥ 0, on the state-trace space by

    μ_x, n(D) = 1/n+1∑_i = 0^n( (S_i, _i) ∈ D )   or   μ_x, n(D) = 1/n+1∑_i = 0^n( (S_i, y_i, _i) ∈ D )
 
for all Borel subsets D of the state-trace space.
[We take the topology on the state-trace space to be the product topology, with the discrete topology on the space of states/memory states and with the usual topology on ^d, the trace space. The Borel sigma-algebra on the state-trace space is generated by this topology.]

We refer to them as the occupation probability measures of the state-trace process. 
Their convergence to the unique invariant probability measure of the state-trace process is crucial for our convergence analysis of the TD algorithms. 
Here the sense of convergence for these probability measures is weak convergence, and it is defined as follows: if μ and {μ_n} are probability measures on the state-trace space and as n →∞, ∫ f dμ_n→∫ f dμ for all bounded continuous functions f, then the sequence {μ_n} is said to converge weakly to μ. 

Let Z_n=(S_n, _n, S_n+1) in the case of state-dependent λ, and Z_n= (S_n,  y_n, _n, S_n+1) in the case of history-dependent λ. Denote the space of Z_n in each case by the same notation . (This is to prepare for handling temporal-differences terms, which involve state transitions.)
Among (vector-valued) functions on , a set of them will be important in our analysis: these are functions f(z) that are Lipschitz continuous in the trace variable, uniformly w.r.t. the other components of z. We have z = (s, , s') or z = (s, y, , s') (depending on the case of λ), and since the state space  and the memory state space  are finite, such a function f(z) is just one that is Lipschitz continuous in  for each (s, s') or (s, y, s'). So in what follows, when referring to such a function f, we will simply say f is Lipschitz continuous in the trace variable .

Regarding other notations and terminologies, the abbreviation “a.s.” stands for “almost surely,” and “_x-a.s.” for “almost surely w.r.t. the probability measure _x,” where the subscript x indicates that the process under consideration starts from the initial condition x. 
We shall use · to denote the sup-norm and ·_2 the standard Euclidean norm. For a sequence of random variables X_n, we say X_n converges in mean to a random variable X, if [  X_n - X  ] → 0 as n →∞. 




  §.§.§ Ergodicity properties
 

The ergodicity results given in Theorem <ref> below are proved essentially in <cit.> for the case of state-dependent λ, and in <cit.> for the case of history-dependent λ we consider.
[The paper <cit.> analyzed the state-trace process in the case of constant λ. Its proof arguments and conclusions, however, extend to the case of state-dependent λ, under Condition <ref>. In fact, this extension was incorporated in the convergence analysis of the more complex ETD algorithm <cit.>, and that is why we do not repeat here the proof arguments for this extension. The paper <cit.> analyzed the state-trace process in the case of history-dependent λ under Assumption <ref>.]

The first part of the theorem concerns the existence and uniqueness of an invariant probability measure of the state-trace process, and the convergence of the occupation probability measures. Without these ergodicity properties, the behavior of the TD algorithms will be quite different, indeed much more complex, and will have to be analyzed using more advanced stochastic approximation theory for differential inclusions (which are beyond the scope of the present paper).


The second part of the theorem will be used, among others, to characterize the average dynamics of the algorithms. We need it in the case of state-dependent λ. In that case, the functions appearing in the algorithms can be unbounded, but they have the Lipschitz continuity property required in the theorem.  Note that for bounded continuous functions f(z), by the weak convergence of occupation probability measures given in Theorem <ref>(i),  the conclusions in the part (ii) automatically hold. (However, even in the case of history-dependent λ, we will still need the function f(z) to be Lipschitz continuous in the trace variable, in order to show that it satisfies a certain “averaging condition” that is stronger than the convergence-in-mean ensured by Theorem <ref> and is needed in the subsequent convergence analysis. See Prop. <ref>(i) and Remark <ref> in Section <ref>.)

 



 
Under Assumption <ref>, the following hold: 



  (i) The state-trace process is a weak Feller Markov chain and has a unique invariant probability measure ζ. For each initial condition x of the process, the occupation probability measures {μ_x,n} converge weakly to ζ, _x-a.s.

  (ii) Let _ζ denote expectation w.r.t. the stationary state-trace process with initial distribution ζ. Then _ζ[ f(Z_0)] < ∞ for any vector-valued function f(z) that is Lipschitz continuous in the trace variable . Furthermore, for such function f, given each initial condition of Z_0, as n →∞, 1/n∑_i=0^n-1 f(Z_i) converges to _ζ [f(Z_0) ] in mean and almost surely.






Next, w.r.t. the stationary state-trace process with initial distribution ζ, we derive expressions of the expectation _ζ [ f(Z_0) ] for several functions f involved in the GTD algorithms.
These expressions are related to the expressions (<ref>)-(<ref>) of the gradient ∇ J(θ) and will appear in the mean ODEs associated with the algorithms. 
To state the results concisely, let 

    δ̅(s, s', v) = ρ(s) ( r(s, s') +  (s') v(s') - v(s) ),     δ̅_0(v) = δ̅(S_0, S_1, v).

(Recall r(s,s') is the mean reward associated with the transition (s,s'); cf. Section <ref>. The above are temporal-difference terms without noises in the rewards.)
Recall that  is the substochastic matrix in the generalized Bellman operator  (cf. (<ref>)), ϕ_i is the i-th component of the function ϕ, and Φ_i the i-th column of the matrix Φ.



 
Under Assumption <ref>, we have

    _ζ[ ϕ(S_0) ϕ(S_0)^]     = Φ^Ξ Φ, 
    _ζ[ _0  δ̅_0(v) ]     = Φ^Ξ  ( v - v),      ∀  v ∈^||, 
    _ζ[ _0 ·ρ_0 (ϕ_i(S_0) - _1ϕ_i(S_1) ) ]     = Φ^Ξ  (I - ) Φ_i,       1 ≤ i ≤ d, 
    _ζ[ _0 ·ρ_0 (1 - λ_1)  _1ϕ_i(S_1) ]     = Φ^Ξ Φ_i,      1 ≤ i ≤ d.




To prove this proposition, it is convenient to extend the stationary state-trace process whose time is indexed by n ≥ 0, to a double-ended stationary state-trace process {(S_n, _n)} or {(S_n, y_n, _n)} with - ∞ < n < ∞. Let _ζ denote the probability measure of the latter process. We shall keep using _ζ to denote expectation with respect to _ζ.
The following lemma gives an expression of _0 in this stationary process. It will facilitate our calculation of _ζ [ f(Z_0) ] for various functions f in the proposition.

Regarding notation in the lemma and in what follows, for k ≤ m, let ρ_k^m = ∏_i=k^m ρ_i, λ_k^m = ∏_i=k^m λ_i, γ_k^m = ∏_i=k^m  γ_i, and in addition, adopt the convention that λ_k^m=γ_k^m=ρ_k^m=1 if k > m. 
Let  denote the d-dimensional vector of all 1's.

   
Let Assumption <ref> hold. Then _ζ-almost surely, ∑_n=1^∞λ_1-n^0  γ_1-n^0 ρ_-n^-1(S_-n) is well-defined and finite, and

    _0 =  (S_0) + ∑_n=1^∞λ_1-n^0  γ_1-n^0 ρ_-n^-1 (S_-n).




For the case of history-dependent λ we consider, this is proved in <cit.>.
We give the proof for the case of state-dependent λ. The beginning part of the proof is same as that of <cit.> and similar to that of <cit.> for the case of constant λ.
Under Condition <ref>(i), (I - Γ)^-1 = ∑_n=0^∞ (Γ)^n and therefore,

    _ζ[ ∑_n=1^∞γ_1-n^0 ρ_-n^-1] =  ∑_n=1^∞_ζ[ γ_1-n^0 ρ_-n^-1] 
       =  ∑_n=1^∞ξ^ (Γ)^n  < ∞

where the first equality follows from the monotone convergence theorem, and the second equality follows from a direct calculation together with the fact that the marginal of ζ on  coincides with ξ, the unique invariant probability measure of {S_n} under Condition <ref>(ii).
Since λ_1-n^0 ≤ 1 for all n, the above relation implies that 

    _ζ[ ∑_n=1^∞λ_1-n^0  γ_1-n^0 ρ_-n^-1 (S_-n) ]  ≤max_s ∈(s) ·_ζ[ ∑_n=1^∞γ_1-n^0 ρ_-n^-1] < ∞.

It then follows from a theorem on integration <cit.> that _ζ-almost surely, the infinite series ∑_n=1^∞λ_1-n^0  γ_1-n^0 ρ_-n^-1(S_-n) converges to a finite limit. 

We now prove (<ref>). Since under Condition <ref>(i), ∑_n=m^∞ (Γ)^n converges to the zero matrix as m →∞, it follows from an argument very similar to the above that

    _ζ[ ∑_n=m^∞λ_1-n^0  γ_1-n^0 ρ_-n^-1 (S_-n) ]→ 0,     as m →∞.
 
Unfolding the iteration (<ref>) for _0 backwards in time, we have that for all m ≥ 1,

    _0 = (S_0) + ∑_n=1^m-1λ_1-n^0  γ_1-n^0 ρ_-n^-1 (S_-n)  +  λ_1-m^0  γ_1-m^0 ρ_-m^-1 _-m.
 
Let _0 denote the expression on the right-hand side (r.h.s.) of (<ref>). Then, for all m ≥ 1,

    _ζ[ _0 - _0 ] ≤_ζ[ ∑_n=m^∞λ_1-n^0  γ_1-n^0 ρ_-n^-1 (S_-n) ] + _ζ[ λ_1-m^0  γ_1-m^0 ρ_-m^-1 _-m].
 
Let m →∞ in the r.h.s. of (<ref>). The first term converges to 0 by (<ref>). For the second term, since λ_1-m^0 ≤ 1, it is bounded above by

    _ζ[ γ_1-m^0 ρ_-m^-1 _-m]     = _ζ[ _-m·_ζ [  γ_1-m^0 ρ_-m^-1| S_-m, _-m ]  ] 
       ≤_ζ[  _-m·^ (P Γ)^m ] = _ζ[ _0 ] ·^ (P Γ)^m  m →∞⟶ 0,

where the last equality uses the stationarity of the process, and the convergence to zero follows from the fact _ζ[ _0 ] < ∞ (Theorem <ref>(ii)) and the fact that under Condition <ref>(i), (P Γ)^m converges to a zero matrix. Hence, letting m →∞ in the r.h.s. of (<ref>), we obtain 
_ζ[ _0 - _0 ] = 0 and consequently, _0 = _0, _ζ-a.s., which proves (<ref>).




We now use the preceding lemma to calculate the expectations in Prop. <ref>.


Since ξ is the invariant distribution of {_n}, the marginal of ζ on  coincides with ξ. Then (<ref>) obviously holds. 
Using the expression of _0 given in Lemma <ref>, the verifications of (<ref>)-(<ref>) are similar and follow from direct calculations. 
We do this first for the case of state-dependent λ. Consider the double-ended stationary state-trace process.
First, let us calculate _ζ[ _0 ·ρ_0 f(S_0, S_1) ] for an arbitrary bounded function f on ^2. 
We have

    _ζ[ _0 ·ρ_0 f(S_0, S_1) ]     = ∑_n=0^∞_ζ[ λ_1-n^0  γ_1-n^0 ρ_-n^-1 (S_-n) ·ρ_0 f(S_0, S_1) ]
        = ∑_n=0^∞_ζ[ λ_1^n  γ_1^n ρ_0^n-1(S_0) ·ρ_n  f(S_n, S_n+1) ]
        =  ∑_n=0^∞_ζ[ (S_0) ·_ζ[ λ_1^n  γ_1^n ρ_0^n   f(S_n, S_n+1) | S_0 ] ] ,
 
where the first equality follows from applying Lemma <ref> and then changing the order of expectation and summation (which, in view of (<ref>), is justified by the dominated convergence theorem), and the second equality uses the stationarity of the double-ended state-trace process. From the change of measure performed through ρ_0^n, we have

    _ζ[ λ_1^n  γ_1^n ρ_0^n   f(S_n, S_n+1) | S_0 ] = ^π[ λ_1^n  γ_1^n f(S_n, S_n+1) | S_0 ]

and therefore,

    _ζ[ _0 ·ρ_0 f(S_0, S_1) ]     = ∑_s ∈ξ_s (s) ·∑_n=0^∞^π[ λ_1^n  γ_1^n f(S_n, S_n+1) | S_0 = s]
        = ∑_s ∈ξ_s (s) ·^π[  ∑_n=0^∞λ_1^n  γ_1^n f(S_n, S_n+1) | S_0 = s]
  
where the change of the order of summation and expectation in (<ref>) is justified by the dominated convergence theorem, since by Condition <ref>(i), we have ^π[  ∑_n=0^∞λ_1^n  γ_1^n  f(S_n, S_n+1) | S_0 = s] ≤ f·∑_n=0^∞^ (P Γ)^n  < ∞.

Equations (<ref>)-(<ref>) then follow from (<ref>) by specializing the function f to the corresponding function in each case. 
In particular, for a given v ∈^||, corresponding to δ̅_0(v) in (<ref>), we let f(s,s') = r(s, s') +  (s') v(s') - v(s). Then by (<ref>) and the expression of v - v given in (<ref>), we have _ζ[ _0 ·ρ_0 f(S_0, S_1) ] = Φ^Ξ  ( v - v), verifying (<ref>). We omit the similar calculation for (<ref>). 
Equation (<ref>) is implied by (<ref>), as can be seen by letting r(s,s') ≡ 0 and v=Φ_i. 
This proves the proposition for the case of state-dependent λ.

In the case of history-dependent λ we consider, (<ref>) is proved in <cit.>. As just mentioned, (<ref>) is implied by (<ref>). Let us verify (<ref>). In this case the same proof of <cit.> (which works with the double-ended stationary state-trace process {(S_n, y_n, _n)}_- ∞ < n < ∞ and is similar to the preceding proof leading to (<ref>)) shows that for any bounded function f on , 

    _ζ[ _0 ·ρ_0 (1 - λ_1) f(S_1) ] = ^π_ζ[ (S_0) ·^π_ζ[ ∑_n=0^∞λ_1^nγ_1^n· (1 - λ_n+1) f(S_n+1) | S_0 ] ],

where ^π_ζ is expectation w.r.t. the process described immediately after (<ref>), the same process w.r.t. which  is defined.
Letting f(s) = γ(s) ϕ_i(s) for each i ≤ d, we then have

    _ζ[ _0 ·ρ_0 (1 - λ_1)  _1 ϕ_i(S_1) ] 
          = ^π_ζ[ (S_0) ·^π_ζ[ ∑_n=0^∞λ_1^nγ_1^n+1· (1 - λ_n+1)  _i(S_n+1) | S_0 ] ].

On the other hand, from the expression of  in (<ref>), we see that the substochastic matrix  in the affine operator  is given by

    ( v)(s)  =  ^π_ζ[ ∑_n=0^∞λ_1^n (1 - λ_n+1)  ·γ_1^n+1  v(S_n+1)  | S_0 = s],    s ∈,  ∀  v ∈^||.

It then follows that 
_ζ[ _0 ·ρ_0 (1 - λ_1)  _1ϕ_i(S_1) ] = ∑_s ∈ξ_s (s) · (ϕ_i)(s) = Φ^Ξ Φ_i. This proves the proposition for the case of history-dependent λ.




  §.§.§ More properties of traces
 
In the rest of this subsection, we state several more properties of the trace iterates, which will be needed in the subsequent convergence analysis.
Some of the properties given in the part (i) of the following proposition were in fact used to prove the ergodicity of the state-trace process given in Theorem <ref>.

One important property of the trace iterates {_n} is its uniform integrability—a set 𝒳 of ^m-valued random variables is said to be uniformly integrable if lim_a →∞sup_X ∈𝒳 [  X(  X > a) ] = 0. This property is much stronger than and entails that sup_ n ≥ 0 [ _n  ] < + ∞ <cit.>. In the case of history-dependent λ, since {_n} lies in a pre-determined bounded set by the choice of {λ_n} (cf. Condition <ref>(ii)), trivially {_n} is uniformly integrable. So the part (ii) of the proposition below concerns only the case of state-dependent λ. 


 
Under Assumption <ref>, the traces {_n} have the following properties:



  (i) Let {_n} and {_n} be generated by the iteration (<ref>), using the same trajectory of states {S_n} and also the same initial memory state y_0 in the case of history-dependent λ, but with different initial _0 and _0, respectively. Then _n - _n  0. Moreover, for any initial _0, _0 in a compact set E, [ _n - _n  ] ≤ c_n for some constant c_n that depends on the set E and satisfies c_n → 0 as n →∞.

  (ii) In the case of state-dependent λ, for any given initial _0, {_n } is uniformly integrable.




In (i), that _n - _n  0 is proved in <cit.> for the case of history-dependent λ, using the continuity condition in Condition <ref>(i), the supermartingale convergence theorem, and the fact that (P Γ)^n converges to a zero matrix under Condition <ref>(i). For state-dependent λ, it follows from the same argument (which is similar to the proof of <cit.> for constant λ).
The proof, for both cases of λ, also shows that for all n,

    [ _n - _n  ] ≤_0 - _0 ·^ (P Γ)^n .

Since lim_n →∞^ (P Γ)^n  = 0 and sup_, ∈ E -  < ∞ if E is compact, the constants c_n in (i) can be chosen as c_n = sup_, ∈ E - ·^ (P Γ)^n for all n.
 
We now show (ii) is true. We can apply the same proof of a similar result <cit.>, which establishes the uniform integrability of the trace iterates produced by the ETD algorithm, provided that we can show the following approximation property of what we call truncated traces. (The proof of <cit.> uses only the approximation property just mentioned and does not care about how {_n} is generated. This is why we can apply that proof exactly to the case considered here.) Specifically, for each integer K ≥ 1, define truncated traces {_n,K} as follows:

    _n = _n    if n ≤ K;      _n,K = ∑_i = 0^K λ_n-i+1^n γ_n-i+1^n ρ_n-i^n-1ϕ(S_n-i)   if n > K.

By definition {_n,K} lies in a bounded set depending on the initial _0 and K.
[To see this, notice that for n > K, the truncated traces _n,K do not depend on the initial _0, and being the sum of K+1 functions of the states, they lie in a bounded set determined by K (since the state space is finite). For all n ≤ K, _n,K also lies in a bounded set, which is determined by K and the initial _0.]

To use these bounded truncated traces for establishing the uniform integrability of the original traces {_n}, the approximation property we need is that given an initial _0, for each K, there is a constant L_K that depends on K and _0 such that

    sup_n ≥ 0[ _n - _n,K] ≤ L_K,   and  L_K ↓ 0   as K → + ∞
   
(i.e., L_K decreases to 0 as K →∞). 
For n > K, with c = max{_0 ,  max_s ∈ϕ(s)} < ∞, we have

    [ _n - _n,K]    ≤∑_i = K+1^n-1[  λ_n-i+1^n γ_n-i+1^n ρ_n-i^n-1ϕ(S_n-i) ] + 
    [ λ_1^n γ_1^n ρ_0^n-1_0] 
       ≤ c ·∑_i = K+1^n[ γ_n-i+1^n ρ_n-i^n-1] ≤ c ·∑_i = K+1^n ^ (P Γ)^i ≤ c ·∑_i = K+1^∞^ (P Γ)^i .

Now let L_K be the constant in the last expression above. Then the bound in (<ref>) holds and moreover, since under Condition <ref>(i) the matrix ∑_i = K+1^∞ (P Γ)^i converges to a zero matrix as K →∞, we have L_K ↓ 0 as K →∞ as required in (<ref>). Feeding this approximation property of truncated traces to the proof of <cit.>, we obtain the uniform integrability of trace variables stated in (ii).


The next proposition, particularly, its part (i), will be important for convergence analysis. It will be used to show that the algorithms satisfy certain “averaging conditions” that will allow us to relate their average dynamics to respective mean ODEs. Part (i) is similar to <cit.> for the ETD algorithm, and its proof is also similar, albeit simpler. Part (ii) will be used in the analysis of the two-time-scale algorithms at their fast time-scale. In the proposition, the notation lim_m, n →∞ or lim_m, n, j →∞ means that the limit is taken as m, n  or m, n, j go to infinity in any possible way.

 
Let Assumption <ref> hold. Let f(z) be a vector-valued function that is Lipschitz continuous in the trace variable , and let f̅ = _ζ [ f(Z_0) ]. 
Then the following hold:



  (i) For each compact set D ⊂,

    lim_m, n →∞1/m∑_i=n^n+m-1( f(Z_i) - f̅ ) (Z_n ∈ D) = 0   in mean.


  (ii) For a given sequence {c_i} of positive numbers with c_i → 0 as i →∞, 

    lim_m, n →∞1/m∑_i=n^n+m-1[ c_i · f(Z_i) ] = 0,     lim_m, n, j →∞1/m∑_i=n^n+m-1[c_j · f(Z_i) ] = 0.
 




For z to be in a given compact set D ⊂, there is only a finite number of values that the state and memory state components of z can take (since the spaces of states and memory states are finite). 
This means that to prove (i) for each compact set D, it is sufficient to prove it for each D of the form {(s, , s') |∈  E } or {(s, y, , s') |∈  E }, where E is an arbitrary compact set in ^d, and (s,s') or (s,y,s') (depending on the case of λ) is an arbitrary, possible state transition or an arbitrary, possible triple of memory state and state transition. To simplify notation, consider such a pair (s,s') or triple (s,y,s'), and denote it by z̅^o. 
Denote Z^o_i = (S_i, S_i+1) for the case of state-dependent λ and Z^o_i = (S_i, y_i, S_i+1) for the case of history-dependent λ. 
Then, as just discussed, to prove (i), it is sufficient to show that for an arbitrary compact set E ⊂^d and two arbitrary sequences of integers m_j, n_j →∞, 

    lim_j →∞ 1/m_j∑_i=n_j^n_j+m_j-1( f(Z_i) - f̅ ) ·(_n_j∈ E, Z^o_n_j = z̅^o ) = 0     in mean.
 

To prove (<ref>), we first define, for each j, auxiliary trace variables _i^j, i ≥ n_j, to decompose the difference term f(Z_i) - f̅ in (<ref>). Specifically, we define _i^j as follows:



  (a) Fix a point e̅∈ E. 

  (b) For each j ≥ 1, define a sequence of traces, _i^j, i ≥ n_j, by using the same recursion that defines the traces {_i}, based on the same trajectory of {S_i} or {(S_i, y_i)} (for each case of λ, respectively), but starting at the iteration n_j with the initial _n_j^j =.
  

Let Z_i^j =(S_i, e_i^j, S_i+1) or Z_i^j =(S_i, y_i, e_i^j, S_i+1) (for each case of λ, respectively). We write the summation in (<ref>) as

    ∑_i=n_j^n_j+m_j-1( f( Z_i) - f̅) = ∑_i=n_j^n_j+m_j-1( f(Z^j_i) - f̅ ) + ∑_i=n_j^n_j+m_j-1( f(Z_i) - f(Z^j_i) ).

Equation (<ref>) will follow immediately if we show that

    lim_j →∞ 1/m_j∑_i=n_j^n_j+m_j-1( f(Z^j_i) - f̅ ) ·(_n_j∈ E, Z^o_n_j = z̅^o) = 0     in mean, 
       lim_j →∞ 1/m_j∑_i=n_j^n_j+m_j-1( f(Z_i) - f(Z^j_i) ) ·(_n_j∈ E, Z^o_n_j = z̅^o) = 0     in mean.


To prove (<ref>), note that by construction, for all j, conditioned on the event Z^o_n_j = z̅^o, the process {Z^j_n_j + i}_ i ≥ 0 has the same probability distribution: it is just the distribution of the process Ẑ_i = (Ŝ_i, _i, Ŝ_i+1), i ≥ 0, or the process Ẑ_i = (Ŝ_i, ŷ_i, _i, Ŝ_i+1), i ≥ 0 (depending on the case of λ), for a state-trace process {(Ŝ_i, _i)} or {(Ŝ_i, ŷ_i, _i)} that starts from _0 = e̅ and (Ŝ_0, Ŝ_1) = z̅^o or (Ŝ_0, ŷ_0, Ŝ_1) = z̅^o (depending on the case of λ). Therefore, the convergence-in-mean stated in (<ref>) is equivalent to 

    lim_m →∞ 1/m∑_i=0^m-1( f(Ẑ_i) - f̅ ) = 0   in mean.
  
Since the function f(z) is Lipschitz continuous in the trace variable , (<ref>) is true by Theorem <ref>(ii). Therefore, (<ref>) holds. 

We now prove (<ref>).
Again, by the Lipschitz continuity of f(z) in the trace variable , it is sufficient to show 

    lim_j →∞1/m_j∑_i=n_j^n_j+m_j-1_i - ^j_i ·(_n_j∈ E,   Z^o_n_j = z̅^o) = 0     in mean.
 
By Prop. <ref>(i), for the given compact set E, starting from time n_j and initial _n_j, ^j_n_j∈ E, we have _n_j + i - ^j_n_j + i≤ c_i for some constant c_i that satisfies c_i → 0 as i →∞. This implies (<ref>), and hence (<ref>) also holds. We have thus proved (i).

To prove (ii), note that since f(z) is Lipschitz continuous in the trace variable , there is some constant ℓ such that f(z) ≤ℓ (1 + ) for all z. Hence, [  f(Z_i) ] ≤ℓ + ℓ [ _i ] for all i ≥ 0. 
We also have sup_i ≥ 0[ _i] < ∞ by Prop. <ref>(ii), and lim_i →∞ c_i = 0 by assumption. 
These facts together clearly imply (ii). 



 
Note that for a bounded measurable function f, Prop. <ref>(ii) holds trivially. For Prop. <ref>(i), however, our proof (in particular, the proof of (<ref>)) needs the function f to be Lipschitz continuous in the trace variable, even if f is bounded and continuous, and it is not clear to us if this Lipschitz continuity condition can be relaxed. 





§ CONVERGENCE ANALYSES I
 

This section starts our convergence analysis of several gradient-based TD algorithms. 
The first two subsections are for the two-time-scale GTDa and GTDb, respectively. In the subsequent Section <ref>, we will analyze several other algorithms, including single-time-scale algorithms.
The purposes of this section are: (i) to present specific convergence results for GTD (Sections <ref>, <ref>); (ii) to discuss a way to “robustify” the algorithms against the high-variance issue in off-policy learning (Section <ref>), and to discuss a composite scheme of setting the λ-parameters that extends the ones given in Section <ref> (Section <ref>), both of which can also be applied to other algorithms later; and (iii) to show the overall proof structure and the main proof arguments, so that when we analyze other algorithms, we can focus on their differences from GTD, without repeating similar proofs. 

In this and next sections, we use the weak convergence method from stochastic approximation theory <cit.> to analyze the algorithms for a broad range of stepsizes. The advantage of the weak convergence method is that the conditions involved are much weaker than those for almost sure convergence. As a result, one can characterize the asymptotic behavior of the algorithms for slowly diminishing stepsizes and constant stepsizes, which are often used in practice. In comparison, almost sure convergence, which we will address in Section <ref>, requires more stringent conditions, and the range of stepsizes for convergence is narrower—especially narrow for the case of state-dependent λ where the trace iterates can be unbounded and have unbounded variances.

In most part of this paper, we will analyze constrained versions of the algorithms, where the iterates are confined in some bounded sets. This is not a serious limit in practice if the constraint sets are large enough; in addition, constraints are certainly needed if constant stepsizes are used.

We find it better to state the convergence theorems close to where they are finally proved. 
For quick access to the convergence results of this section, we list them here:




  * GTDa: Theorem <ref> (for diminishing stepsize), Theorem <ref> (for constant stepsize), and  Theorem <ref> (for averaged iterates in the constant-stepsize case);

  * GTDb: Theorem <ref>;

  * “robustified” biased variants of GTD: Theorem <ref>;

  * GTD with a composite scheme of setting λ: Theorem <ref>.

We remark that for the (constrained) GTDa and GTDb algorithms, the convergence theorems we present in this section have the same conclusions and are proved essentially with the same arguments. However, in the case of GTDa, these theorems can be improved further by using a minimax problem formulation, and we will address that in Section <ref> (see Theorem <ref>) when we have all the tools needed.



 §.§ GTDa
 

In this subsection we analyze a constrained version of GTDa under Assumption <ref>:

    θ_n+1    = Π_B_θ( θ_n + α_n  ρ_n (ϕ(S_n) - _n+1ϕ(S_n+1) ) ·_n^ x_n ),  
    
       x_n+1    = Π_B_x( x_n + β_n (_n δ_n(v_θ_n) - ϕ(S_n) ϕ(S_n)^ x_n) ),
 
where we take B_θ and B_x to simply be closed balls in ^d centered at the origin and with large radii, and Π_B_θ and Π_B_x denote the projection on the constraint sets B_θ and B_x, respectively, w.r.t. ·_2 (the usual Euclidean norm). We will later impose a condition on B_x (cf. (<ref>)) to ensure that it is large enough so that the algorithm can carry out gradient-descent to minimize J(θ) over B_θ. A much weaker condition on B_x also works for GTDa (under which GTDa computes gradients of a function different from but related to J) and will be discussed in Section <ref>.

We choose the constraint sets this way because of a nice property: the projection operations will not take the iterates outside the subspace {ϕ()} if the iterates are initially in this subspace. This is due to the nature of TD algorithms, as can be see from the above formulae and the formula (<ref>) for calculating traces:

    _n = λ_n _n ρ_n-1 e_n-1 + ϕ(S_n).

It is especially convenient to have the x-iterates lying in {ϕ()}: It avoids all the nuisances due to the possible linear dependence between the component functions of the feature map ϕ, so we can focus the convergence analysis on issues that are important.

The GTDa algorithm (<ref>)-(<ref>) is a two-time-scale algorithm. Its θ-iterates vary at a slow time-scale determined by the stepsizes {α_n}, while its x-iterates vary at a fast time-scale determined by the stepsizes {β_n}. We will consider two types of stepsizes: diminishing stepsize and constant stepsize. The latter refers to the case where α_n = α and β_n = β for all n, with α < < β. For diminishing stepsizes, we impose the following condition:

[Condition on diminishing stepsizes for two-time-scale algorithms]  

The (deterministic) nonnegative sequences {α_n} and {β_n} satisfy that 

    ∑_n ≥ 0α_n = ∑_n ≥ 0β_n =∞,   α_n, β_n → 0   and  α_n/β_n → 0   as n →∞.
 
Moreover, for some sequences of integers m_n →∞ and m'_n →∞,

    lim_n →∞ sup_0 ≤ j ≤ m_n| α_n+j/α_n - 1 | = 0,     lim_n →∞ sup_0 ≤ j ≤ m'_n| β_n+j/β_n - 1 | = 0.




The condition α_n/β_n → 0 above shows the separation between the time-scale of the θ-iterates and that of the x-iterates. The condition (<ref>) above allows large stepsizes that diminish very slowly, such as n^-c, c ∈ (0,1]. This stepsize condition is used in the book <cit.> to analyze stochastic approximation algorithms with the weak convergence method; the results from that book will be used in this section.

The structure of the proof is the same for the two-time-scale GTDa, GTDb, and other two-time-scale algorithms that we will consider later. We first use the ergodicity of the state-trace process to identify the desired mean ODEs associated with the algorithms. We then use the ergodicity property and other properties of the trace iterates given in Section <ref> to show that the conditions of the analyses in <cit.> (for diminishing as well as constant stepsizes) are satisfied by each algorithm. The theorems in <cit.> ensure that the asymptotic behavior of the algorithm is characterized by the asymptotic behavior of the solutions of its associated mean ODE. 
There are actually two mean ODEs associated with a two-time-scale algorithm, one for each time-scale. 
We apply the theorems in <cit.> for single-time-scale algorithms twice to analyze the behavior of the algorithm at each time-scale, feeding the result on the fast time-scale to the analysis of the slow time-scale.
For constrained algorithms, almost no conditions are imposed on the mean ODE for the slow time-scale. This allows us to establish the connection between the behavior of an algorithm and the solutions of its slow-time-scale ODE, without worrying about what those ODE solutions actually are. We can study the ODE separately to get its solution properties and translate them into the convergence properties of the algorithm. This last proof step is simpler and quicker to carry out, so when presenting our proofs, we will first tackle this step by discussing the desired mean ODEs and deriving their solution properties.

This subsection is relatively long. After we explain the main proof arguments for GTDa in this subsection, we will apply the same arguments to other algorithms in later subsections, so they will be much shorter than this one.




  §.§.§ Associated mean ODEs and their solution properties
 

We first rewrite the GTDa algorithm in a form that is more concise and convenient for analysis.
Recall we defined the random variable Z_n = (S_n, _n, S_n+1) for state-dependent λ and Z_n = (S_n, y_n, _n, S_n+1) for history-dependent λ. From now on, let us write Z_n = (S_n, y_n, _n, S_n+1) in both cases, treating the memory states y_n as dummy variables in the case of state-dependent λ.
Let  denote the space of z as before. 
Let ω_n+1 = ρ_n (R_n+1 - r(S_n, S_n+1) ), which is a noise term in the observed one-stage reward and which, given the history (Z_0, …, Z_n) and (R_1, …, R_n), has zero-mean.
We rewrite the GTDa algorithm (<ref>)-(<ref>) as

    θ_n+1    = Π_B_θ( θ_n + α_n   g(θ_n, x_n, Z_n) ), 
    
          x_n+1    = Π_B_x( x_n + β_n (k(θ_n, x_n, Z_n) + _n ω_n+1 ) ),

where with z = (s, y, , s'), the functions g(θ, x, z) and k(θ, x, z) are given by

    g(θ, x, z) =  ρ(s, s') (ϕ(s) - (s') ϕ(s') ) ·^ x,      k(θ, x, z) =  δ̅(s, s', v_θ) - ϕ(s) ϕ(s)^ x.
 
Recall that δ̅(s, s', v_θ) above is the temporal-difference term with the noise in the reward subtracted: 
δ̅(s, s', v_θ) =  ρ(s) ( r(s, s') +  (s') v_θ(s') - v_θ(s) ), 
and we use the shorthand δ̅_0(v_θ) for δ̅(S_0, S_1, v_θ).

As mentioned earlier, there are two ODEs associated with the algorithm, one for each time-scale. The slow-time-scale one depends on the solution to the fast-time-scale one. Let us work out what these ODEs are—at this stage they are only desired ODEs that we wish to associate with the algorithm, for their real connection with the algorithm can only be established after we carry out the analysis using stochastic approximation theory in the following subsections.

For the fast time-scale, by Prop. <ref>, for each fixed (θ, x), 

    k̅(θ, x) : = _ζ[ k(θ, x, Z_0) ]     = _ζ[ _0  δ̅_0(v_θ) ] - _ζ[ ϕ(S_0) ϕ(S_0)^] · x 
        = Φ^Ξ  ( v_θ - v_θ) - Φ^Ξ Φ  x.

For the function g in the slow-time-scale iteration, by (<ref>) in Prop. <ref>, for fixed (θ, x),

    g̅(θ, x): = _ζ[ g(θ, x, Z_0) ]     = _ζ[  ρ_0 (ϕ(S_0) - _1ϕ(S_1) ) ·_0^ x ] 
        = (Φ^Ξ  (I - )  Φ)^ x.

Recall from (<ref>)-(<ref>) that for any given θ, there is a unique solution x_θ to the linear equation

    k̅(θ, x) = 0,    x ∈{ϕ()}.
  
Let us define a function x̅(θ) = x_θ. 
By (<ref>), 

    g̅(θ, x̅(θ))  = _ζ[ g(θ, x̅(θ), Z_0) ] = (Φ^Ξ  (I - )  Φ)^ x_θ =: g̅(θ),

where we have let g̅(θ) stand for g̅(θ, x̅(θ)) in order to simplify notation for the subsequent analysis. 
From the expression (<ref>) of the gradient ∇ J(θ), we see that

    g̅(θ) = - ∇ J(θ).
 

At the fast time-scale determined by the stepsizes {β_n}, we can view the algorithm (<ref>)-(<ref>) as the single-time-scale algorithm,

    θ_n+1    = Π_B_θ( θ_n + β_n · (α_n/β_n)   g(θ_n, x_n, Z_n) ), 
    
          x_n+1    = Π_B_x( x_n + β_n (k(θ_n, x_n, Z_n) + _n ω_n+1 ) ).

Since α_n < < β_n, we wish the terms (α_n/β_n)   g(θ_n, x_n, Z_n) are negligible, and we want to relate the average dynamics of the algorithm to the mean ODE,

    ( [ θ̇(t);  ẋ(t) ]) 
     =  ( [                      0; k̅(θ(t), x(t) ) + z(t) ]),     θ(0) ∈ B_θ,   z(t) ∈ - _B_x(x(t)),
 
where _B_x(x(t)) is the normal cone of B_x at x(t), and z(t) is the boundary reflection term. 
Specifically, if x(t) is in the interior of B_x, the reflection term z(t) is zero. 
If x(t) is on the boundary of B_x, z(t) cancels out the projection of k̅(θ(t), x(t) ) on _B_x(x(t)); namely, 

    z(t) = - Π_D ( k̅ (θ(t), x(t)) )     with D = _B_x(x(t))

(the projection is w.r.t. ·_2).
This boundary reflection term is the “minimal force” needed to keep the solution x(·) in the constraint set B_x (cf. <cit.>).

At the slow time-scale determined by the stepsize {α_n}, we can view the θ-iteration (<ref>) as the single-time-scale iteration:

    θ_n+1  = Π_B_θ( θ_n + α_n (g(θ_n, x̅ (θ_n), Z_n)  + Δ_n ) ),    for  Δ_n  = g(θ_n, x_n, Z_n) - g(θ_n, x̅ (θ_n), Z_n).

We wish that x_n “tracks” x̅(θ_n) so that the Δ_n terms have negligible effects on the average dynamics of the θ-iterates, thereby allowing us to associate the following mean ODE with the θ-iteration above: 

    θ̇(t) = g̅(θ(t)) + z(t),     z(t) ∈ - _B_θ(θ(t)),

where _B_θ(θ(t)) is the normal cone of B_θ at θ(t), and z(t) is the boundary reflection term; i.e., z(t) equals minus the projection of g̅(θ(t)) on _B_θ(θ(t)). 

As mentioned earlier, if we can establish the connection between the algorithm and these two ODEs, then the asymptotic properties of the solutions to these ODEs will tell us the asymptotic behavior of the algorithm. So let us examine first the solution properties of these two ODEs.

For the fast-time-scale ODE (<ref>), with θ(·) ≡θ∈ B_θ, note that the solution x(·) stays in the subspace {ϕ()} if x(0) ∈{ϕ()},
[We have k̅(θ, x) ∈{ϕ()} (cf. the expression (<ref>) of k̅). Therefore, its projection on the ball B_x lies in {ϕ()} and so does its boundary reflection term. This shows x(·) stays in {ϕ()} if x(0) ∈{ϕ()}.] 

and we wish x(t) to converge to x̅(θ)=x_θ as t →∞ (so that there is hope for the iterates x_n to track x̅ (θ_n) as desired for the slow-time-scale θ-iteration).
For this we need B_x to be large enough so that
[A sufficient condition for (<ref>) is that 
radius(B_x) ≥sup_θ∈ B_θΦ^Ξ  ( v_θ - v_θ) _2 /c, where c is the smallest nonzero eigenvalue of the matrix Φ^Ξ Φ. 
]


    B_x ⊃{x_θ|θ∈ B_θ}.

Now notice that for each θ, k̅(θ, x) = - ∇ f(x) for the convex quadratic function f(x) =  12 x^ C x -  x^ b, where C =  Φ^Ξ Φ and b = Φ^Ξ  ( v_θ - v_θ), and f(x) attains its minimum at x_θ. So the fast-time-scale ODE is simply solving the problem inf_x ∈ B_x f(x) with gradient descent, and it is straightforward to derive its solution properties stated in the following lemma. (We defer the proof details until after we have discussed the slow-time-scale ODE, which has a similar analysis.)

Let D denote the closure of a set D. The limit set of the ODE (<ref>) is by definition the set

    ⋂_t̅≥ 0  {(θ(t), x(t))  |  θ(0) ∈ B_θ,   x(0) ∈ B_x,   t ≥t̅ }

where each set in the intersection is the closure of the set of points that can be reached after time t̅ by a solution (θ(·), x(·)) of the ODE from some initial condition in B_θ× B_x. Since we are interested only in those solutions with x(0), x(t) ∈{ϕ()} for all t, we focus on a smaller limit set corresponding to those solutions, which is  the set ∩_t̅≥ 0  {(θ(t), x(t))  |  θ(0) ∈ B_θ,   x(0) ∈ B_x ∩{ϕ()},   t ≥t̅ }. 

 
Let the constraint set B_x satisfy (<ref>). Then for all initial x(0) ∈ B_x ∩{ϕ()} and θ(0) ∈ B_θ,
the limit set of the ODE (<ref>) is {(θ, x̅(θ)) |θ∈ B_θ}.


Now for the slow time-scale, consider the desired mean ODE (<ref>).
Let Θ_opt denote the compact set of optimal solutions of inf_θ∈ B_θ J(θ):

    Θ_opt = _θ∈ B_θ J(θ).
  

 
The limit set of the ODE (<ref>) is
⋂_ t̅≥ 0 {θ(t)   |  θ(0) ∈ B_θ,   t ≥t̅ }  = Θ_ opt.


To prove the lemma, we need the following facts concerning convex sets, which will also be useful in later sections:

 
Let D be a closed convex set in ^d and let θ∈ D. For any h ∈^d, let y be the projection of h on _D(θ), the normal cone of D at θ. Then y- h is the point in the convex set _D(θ)- h that has the minimal ·_2-norm. Moreover, h - y lies in the tangent cone of D at θ, and ⟨ y, h - y ⟩ = 0.


The first statement of Lemma <ref> is obvious: Since y is the closest point in _D(θ) to h, by translation, y - h is the closest point in _D(θ)- h to the origin. The second statement about h - y comes from a property of projections on a polar pair of cones <cit.>.
[This theorem asserts that given a closed convex cone K, h can be decomposed uniquely as the sum of two orthogonal vectors, one in K and the other in the polar of K, and these two vectors are given by the projections of h on the two cones, respectively.]


Here are the implications of Lemma <ref> for a solution θ(·) of the ODE (<ref>). Take D = B_θ, θ=θ(t), h = g̅(θ(t)), and y = -z(t); recall that our boundary reflection term z(t) corresponds precisely to - Π__D(θ) h. Therefore, 

    ⟨ z(t),  g̅(θ(t) + z(t) ⟩ = 0,

and the vector  - g̅(θ(t)) - z(t) =  ∇ J(θ(t)) - z(t) is the point in ∇ J(θ(t)) + _B_θ(θ(t)) that has the minimal ·_2-norm.
Notice that for θ∈ B_θ, the set ∇ J(θ) + _B_θ(θ) consists of the subgradients ∂ f(θ) of the convex function f(θ) = J(θ) + δ_B_θ(θ) at the point θ (here δ_B_θ(θ) denotes the indicator function that takes the value 0 on B_θ and + ∞ outside B_θ). For the problem inf_θ f(θ) (which is equivalent to inf_θ∈ B_θ J(θ)), at a non-optimal θ, 0 ∉∂ f(θ), but unlike an arbitrary subgradient, the minimal-norm subgradient has the property that its negation always points in a descent direction of f. At an optimal θ, 0 ∈∂ f(θ) by the optimality condition, and then, of course, the minimal-norm subgradient must be the zero vector. Thus we see that the path of a solution θ(·) of (<ref>) follows descent directions of the objective function f that we want to minimize.

Let us now prove Lemma <ref> for the desired mean ODE at the slow time-scale. 


Since g̅(θ) = - ∇ J(θ), the function V(θ)= J(θ) serves as a Lyapunov function for the ODE (<ref>). In particular, we have

    V̇(θ(t))     = ⟨∇ J(θ(t)), - ∇ J(θ(t)) + z(t) ⟩
        = - ∇ J(θ(t)) - z(t) ^2_2 +  ⟨ z(t), - ∇ J(θ(t)) + z(t) ⟩
        = - ∇ J(θ(t)) - z(t) ^2_2,
  
where the last equality follows from (<ref>). As discussed before the proof, ∇ J(θ(t)) - z(t) is the minimal-norm subgradient of f at θ(t). So, if θ(t) ∈Θ_opt, V̇(θ(t)) = 0 by the optimality condition; if θ(t)  ∉Θ_opt, then ∇ J(θ(t)) - z(t) ≠0 and hence V̇(θ(t)) < 0. 
Moreover, for any 0 < ϵ≤sup_θ∈ B_θ J(θ) - inf_θ∈ B_θ J(θ), there exists η_ϵ > 0 such that
[To derive (<ref>), consider the convex function f(θ) = J(θ) + δ_B_θ(θ) mentioned before the proof. As a set-valued mapping, the subgradient mapping ∂ f is outer semicontinuous and therefore its graph, gph ∂ f = {(θ, y) | y ∈∂ f(θ)}, is closed <cit.>. Then the set gph ∂ f ∩( {θ∈ B_θ| J(θ) ≥inf_θ∈ B_θ J(θ) + ϵ}×^d ) is also closed, and on this set, the continuous function (θ, y) ↦ y^2_2 attains the minimal value, which must be nonzero (since no optimal θ is involved in this set). We can take this value to be η_ϵ in (<ref>).]


    V̇(θ(t)) ≤ - η_ϵ,   if  J(θ(t)) ≥inf_θ∈ B_θ J(θ) +ϵ.
 
Since V is bounded on B_θ, this implies that there exists a time t_ϵ such that all solutions of (<ref>) satisfy that 

    J(θ(t)) ≤inf_θ∈ B_θ J(θ)  + ϵ,     ∀  t ≥ t_ϵ.
 
Thus the limit set of the ODE must be ∩_ϵ > 0{θ∈ B_θ| J(θ) ≤inf_θ∈ B_θ J(θ)  + ϵ} = Θ_opt.


The proof of Lemma <ref> is similar:
 
 
For each θ∈ B_θ, with θ(·) ≡θ, consider a solution x(·) of the ODE (<ref>) with x(0) ∈{ϕ()}. 
We can write k̅(θ, x) = - C(x - x_θ) for C = Φ^Ξ Φ. 
Let V_θ(x) = 12 x - x_θ_2^2. Then with a calculation similar to (<ref>), we have 

    V̇_θ(x(t)) = ⟨ x(t) - x_θ,   - C(x(t) - x_θ) + z(t) ⟩ = - ⟨ x(t) - x_θ, C(x(t) - x_θ) ⟩ + ⟨ x(t) - x_θ,   z(t) ⟩.
   
Since x(t) ∈{ϕ()}, the first term on the r.h.s. is bounded above by - c   x(t) - x_θ_2^2, where c > 0 is the smallest nonzero eigenvalue of the symmetric positive semidefinite matrix C. The second term on the r.h.s. is nonpositive, because x_θ∈ B_x under our assumption and by the definition of the boundary reflection term, - z(t) ∈_B_x(x(t)), which implies ⟨ x - x(t), - z(t) ⟩≤ 0 for all x ∈ B_x.
Thus, we have V̇_θ(x(t)) ≤ - c  x(t) - x_θ_2^2. Since on B_x, {V_θ|θ∈ B_θ} are uniformly bounded, this implies that for any ϵ > 0, there is a time t_ϵ independent of θ such that x(t) - x_θ_2^2  ≤ϵ for all t ≥ t_ϵ. The conclusion of the lemma then follows. 


We have now obtained solution properties of the desired mean ODEs. Next, we need to show that everything we “wished” in the above actually holds for the GTDa algorithm, so that the θ_n iterates produced by the algorithm will behave like the solutions θ(·) of the slow-time-scale ODE and converge, in a sense to be made precise, to the optimal solution set Θ_opt.



  §.§.§ Conditions to verify
 

We use the weak convergence method from stochastic approximation theory to analyze the convergence properties of the algorithm for constant and slowly diminishing stepsizes. Specifically, in the terminology of <cit.>, our algorithm is of the type that involves “exogenous noises,” meaning that the evolution of {Z_n} is not influenced by the iterates {(θ_n, x_n)} themselves. We will apply the theorems in <cit.> for this type of algorithms, and in this subsection, we first use the results of Section <ref> to verify several major conditions that will be needed when applying those theorems. 

We list here only the conditions for the case of diminishing stepsize. The conditions for the constant-stepsize case differ only in notation and will be addressed in Section <ref> when we analyze that case. 
We are going to show that the algorithm fulfills the following seven conditions. They are closely related to the conditions in <cit.>. (We will apply <cit.> twice, once for each time-scale, in our convergence proof.)



  (i) The following sets of random variables are uniformly integrable:
[In this condition, the random variables in the first (third) set correspond to the terms that appear after the stepsizes α_n (β_n) in the θ-iteration (x-iteration). For different algorithms, these two sets in the condition (i) can have different expressions; for example, for GTDb, the first set will also involve the noise terms _n ω_n+1.] 


    {g(θ_n, x_n, Z_n) }_n≥0,   {g(θ, x, Z_n) | n ≥ 0, (θ, x) ∈ B_θ× B_x },
 

    {k(θ_n,x_n,Z_n)+_nω_n+1}_n≥0,   {k(θ, x, Z_n) | n ≥ 0, (θ, x) ∈ B_θ× B_x }.


  (ii) The set of random variables {Z_n}_n ≥ 0 is tight.
[This means that for any ϵ > 0, there exists a compact set D such that inf_n(Z_n ∈ D) ≥ 1 - ϵ.]


  (iii) The functions g(θ, x, z) and k(θ, x, z) are continuous in (θ, x) uniformly in z ∈ D, for any compact set D ⊂.

  (iv) There exists a continuous function k̅(θ, x) such that for each (θ, x) ∈ B_θ× B_x, the convergence-in-mean stated below in Lemma <ref>(i) holds for {k(θ, x, Z_n)}. Also, the convergence-in-mean stated in the first part of Lemma <ref>(iii) holds for {g(θ, x, Z_n)} and each (θ, x) ∈ B_θ× B_x.

  (v) 
There exists a continuous function x̅(θ) such that { (θ, x̅(θ)) |θ∈ B_θ} is the limit set of the projected ODE θ̇(t) = 0, ẋ(t) = k̅(θ(t), x(t)) + z(t), where z(t) ∈ - _B_x(x(t)) is the boundary reflection term, for all initial conditions θ(0) ∈ B_θ and x(0) ∈ B_x ∩{ϕ()}.
[Normally, this condition is required to hold for all initial x(0) ∈ B_x, but here we are interested only in those solutions from x(0) ∈{ϕ()} and they all stay in the subspace {ϕ()} for the ODE under our consideration, as discussed earlier.]



More conditions for the algorithm at its slow time-scale:



  (vi) 
The following sets of random variables are uniformly integrable:

    {g(θ_n, x̅(θ_n), Z_n)}_n ≥ 0,     { g(θ, x̅(θ), Z_n) | n ≥ 0, (θ, x) ∈ B_θ× B_x }.


  (vii) For {g(θ, x̅(θ), Z_n)} and some continuous function g̅(θ),  the convergence-in-mean stated below in Lemma <ref>(ii) holds for each θ∈ B_θ.



Among the above conditions, the continuity condition (iii) and the condition (v) on the fast-time-scale ODE are obviously satisfied: (iii) follows simply from the definitions of the functions g and k (cf. (<ref>)), and (v) follows from Lemma <ref> and the continuity of the function x̅(θ) = x_θ. 
In the case of history-dependent λ, since the trace iterates lie in a given bounded set and hence {Z_n} lie in a compact set,  the conditions (i)-(ii) also hold trivially. 
In the case of state-dependent λ, the tightness condition (ii) follows from the fact sup_n ≥ 0 [ _n ] < ∞ (which is implied by Prop. <ref>(ii)), together with the Markov inequality.  The uniform integrability required by the condition (i) is ensured by the uniform integrability of the traces {_n} established in Prop. <ref>(ii). The proof of this uses the uniform integrability of {_n} together with the following simple facts: the iterates θ_n and x_n are confined in bounded sets, the number of states is finite, the functions g, k are Lipschitz continuous in the trace variable  uniformly w.r.t. (θ, x) in a bounded set and w.r.t. other components of z, and the noise terms {ω_n+1} have finite variances. We omit the proof because it is straightforward and the details are the same as those given for a similar result <cit.> in the analysis of the ETD algorithm with the weak convergence method.

We now address the two remaining convergence-in-mean conditions (iv) and (vii), which are stated in the lemma below. The functions k̅(θ, x), g̅(θ), and x̅(θ) involved here are as defined in Section <ref>. The main result we need is the first two parts of the lemma. We will use them to relate the average dynamics of the algorithm to the desired mean ODEs at the fast and slow time-scales, respectively. The third part of the lemma will also be needed in the analysis of the fast time-scale, to show that the (α_n/β_n)   g(θ_n, x_n, Z_n) terms in (<ref>) are negligible as wished (the first and second relations in this part will be used respectively in the diminishing-stepsize and constant-stepsize cases).
In the lemma, _n denotes conditional expectation given the history (Z_0, …, Z_n) and (R_1, …, R_n). 
Recall that the notation “lim_m, n →∞” means that the limit is taken as m and n go to infinity in any possible way, and the notation “lim_m, n, j →∞” is defined likewise.

 
Under Assumption <ref>, the following hold for each θ∈ B_θ, x ∈ B_x:



  (i) For each compact set D ⊂,

    lim_m, n →∞1/m∑_i=n^n+m-1_n [ k(θ, x, Z_i) - k̅(θ, x) ] (Z_n ∈ D) = 0   in mean.


  (ii) For each compact set D ⊂,

    lim_m, n →∞1/m∑_i=n^n+m-1_n [ g(θ, x̅(θ), Z_i) - g̅(θ) ] (Z_n ∈ D) = 0   in mean.


  (iii)If {α_n} and {β_n} satisfy Assumption <ref>, then

    lim_m, n →∞1/m∑_i=n^n+m-1_n [ (α_i/β_i) · g(θ, x, Z_i) ] = 0   in mean.

For any positive sequence c_j → 0 as j →∞,

    lim_m, n, j →∞1/m∑_i=n^n+m-1_n [ c_j · g(θ, x, Z_i) ] = 0   in mean.





Note first that by Jensen's inequality, it is sufficient to prove the convergence-in-mean statements (i)-(iii) with the conditional expectation _n removed. (E.g., for (i), it is sufficient to show that lim_m, n →∞1/m∑_i=n^n+m-1( k(θ, x, Z_i) - k̅(θ, x) ) (Z_n ∈ D) = 0 in mean.)
Now for each fixed θ and x, as functions of z, k(θ, x, z), g(θ, x, z), and g(θ, x̅(θ), z) are Lipschitz continuous in the trace variable  (cf. (<ref>)), and moreover, by 
(<ref>)-(<ref>), 
_ζ[ k(θ, x, Z_0) ] = k̅(θ, x) and _ζ[ g(θ, x̅(θ), Z_0) ] = g̅(θ). 
So (i) and (ii) follow from Prop. <ref>(i), and (iii) follows from Prop. <ref>(ii). In particular, the first statement in (iii) follows from the first equation in Prop. <ref>(ii) with c_i = α_i/β_i, since lim_i →∞α_i/β_i = 0 by Assumption <ref>.



 
Before we proceed to state and prove two convergence theorems for  GTDa with diminishing and constant stepsizes, we remark that besides the seven conditions listed above, when analyzing the slow time-scale (in particular, to show that the noise terms Δ_n in (<ref>) have negligible effects), our proofs will also use a Lipschitz continuity property of the function g: for (θ, x) ∈ B_θ× B_x and z in a compact set, g(θ, x, z) is Lipschitz continuous in x uniformly w.r.t. (θ, z) (cf. the definition (<ref>) of g). This condition is different from the continuity condition (iii) above, and we use it to make the proofs a little shorter—in fact, the condition (iii) is sufficient.
Besides this Lipschitz continuity property, the characterization of the limit set of the slow-time-scale mean ODE given in Lemma <ref> will also be used in the proofs.
The analysis applies to two-time-scale constrained TD algorithms other than GTDa, as long as they satisfy these same conditions. Lemma <ref> is used only at the end of the proofs; if an algorithm satisfies all the other conditions but the limit set of its slow-time-scale mean ODE is not Θ_ opt, then with that limit set in place of Θ_ opt, the conclusions for GTDa also hold for that algorithm. 





  §.§.§ Convergence proof for the diminishing-stepsize case
 

Recall Θ_ opt is the set of optimal solutions of inf_θ∈ B_θ J(θ).
For a set D ⊂^d, let N_ϵ(D) denote the ϵ-neighborhood of D. For any T > 0 and n ≥ 0, define an integer m(n,T) that points to the last iteration before the sum of the stepsizes α_i between it and the n-th iteration exceeds T:

    m(n,T) = min{ m ≥ n |∑_i=n^m+1α_i > T }.
   
To refer to an iteration i with n ≤ i ≤ m(n,T), we will simply write i ∈ [n, m(n, T) ].

We are now ready to prove our first convergence result for the constrained GTDa algorithm (<ref>)-(<ref>) with diminishing stepsizes.
Denote by dist(θ, Θ_opt) the distance between θ and the optimal solution set Θ_opt.
The convergence is, first of all, in the sense that dist( θ_n,  Θ_ opt) converges to 0 in mean, and it implies the convergence of θ_n to Θ_ opt in probability. But more strongly than this kind of convergence of θ_n, the powerful weak convergence method allows one to assert that as n →∞, increasingly more consecutive θ-iterates must all lie in an arbitrarily small neighborhood of Θ_ opt with an arbitrarily high probability.
This is what the following theorem says with the relation (<ref>).


 
Let Assumption <ref> hold. Let {(θ_n, x_n)} be generated by the two-time-scale constrained GTDa algorithm (<ref>)-(<ref>), with diminishing stepsizes that satisfy Assumption <ref>, with the constraint set B_x satisfying the condition (<ref>), and with initial x_0, _0 ∈{ϕ()}.
Then for each given initial condition, lim_n →∞[  dist( θ_n,  Θ_ opt )] = 0. Moreover,
there exists a sequence of positive numbers T_n →∞ such that for any ϵ > 0,

    lim sup_n →∞( θ_i ∉N_ϵ(Θ_ opt),  some i ∈ [n, m(n, T_n) ] )  = 0.
  



 
First, we consider (θ_n, x_n) together at the fast time-scale determined by {β_n}. Our goal is to show 

    lim sup_n →∞[  x_n - x̅ (θ_n) ] = 0.

We will then use (<ref>) in analyzing the θ-iterates at the slow time-scale determined by {α_n}.

At the fast time-scale, let g_n(θ_n, x_n, Z_n) = α_n/β_n g(θ_n, x_n, Z_n), and rewrite the algorithm in the form of a single-time-scale algorithm as

    ( [ θ_n+1; x_n+1 ]) = Π_B_θ× B_x( [           θ_n + β_n   g_n(θ_n, x_n, Z_n); x_n + β_n ( k(θ_n, x_n, Z_n) + _n ω_n+1) ]).

To analyze its asymptotic behavior, we apply <cit.> for single-time-scale algorithms with diminishing stepsizes and exogenous noises.
The desired mean ODE is (<ref>), namely,

    ( [ θ̇(t);  ẋ(t) ]) 
     =  ( [                      0; k̅(θ(t), x(t) ) + z(t) ]),
 
where θ(t) ∈ B_θ and z(t) ∈ - _B_x(x(t)) is the boundary reflection term. Lemma <ref>(i) shows that the function k̅(θ, x) in the ODE indeed characterizes the average dynamics of the x-iterates in (<ref>), while the first relation in Lemma <ref>(iii) shows that the constant zero function in the ODE characterizes the average dynamics of the θ-iterates in (<ref>). These two parts of Lemma <ref> thus fulfill the convergence-in-mean condition (A.8.2.7) in <cit.>. Regarding other conditions in that theorem, the simple constraint set B_θ× B_x satisfies the condition required, and so does the stepsize {β_n} due to Assumption <ref>. The rest of the conditions correspond to the conditions (i)-(iii) we gave in Section <ref>. (In particular, (A.8.2.1) and (A.8.2.5) correspond to our uniform integrability condition (i); (A.8.2.4) corresponds to our tightness condition (ii); and (A.8.2.3) corresponds to our continuity condition (iii).) These conditions are all satisfied by (<ref>), as discussed earlier.

Thus we can apply <cit.> to the iterates (<ref>) and obtain the following conclusions. 
Consider the sequence of continuous-time processes {(θ^n(·), x^n(·))}_n ≥ 0 where for each n, θ^n(·) and x^n(·) are piecewise constant interpolations of the θ-iterates and x-iterates, respectively, with the n-th iterates placed at t=0:

    θ^n(t) = θ_n+m   and  x^n(t) = x_n+m    for  t ∈[∑_i=0^m-1β_n+i  ,  ∑_i=0^mβ_n+i),  m ≥ 0.

Then by <cit.>, for each subsequence of {(θ^n(·), x^n(·))}_n ≥ 0, there is a further subsequence {(θ^n_j(·), x^n_j(·))}_j ≥ 0 that converges in distribution
[Involved here is the convergence of probability measures on the space of ^2d-valued functions on [0, +∞) or (-∞, + ∞) that are right-continuous with left limits, where the space of these functions is endowed with the Skorohod metric, which makes it a complete separable metric space <cit.>.]

to (θ(·), x(·)), where (θ(·), x(·)) has absolutely continuous paths satisfying the ODE (<ref>) and moreover, almost surely, its path lies in the limit set of the ODE (<ref>). 
As discussed in Section <ref>, with the initial x_0, _0 ∈{ϕ()}, we are only interested in those solutions of the ODE with initial x(0) and hence the entire solution x(·) in {ϕ()}. For all initial conditions x(0) ∈ B_x ∩{ϕ()}, by our assumption on B_x, the limit set of the ODE (<ref>) is the set {(θ, x̅(θ) ) |θ∈ B_θ} (Lemma <ref>).
So, almost surely, θ(·) ≡θ(0) ∈ B_θ and x(·) ≡x̅(θ(0)).

Let us now prove (<ref>).
Suppose it is not true. Then lim sup_n →∞[  x_n - x̅ (θ_n) ] > 0, so there must exist some ϵ > 0 and a subsequence {n_j}_j ≥ 1 of integers such that 

    [  x_n_j - x̅ (θ_n_j) ] > ϵ,     ∀  j ≥ 1.
 
Consider the sequence of piecewise constant interpolated processes {(θ^n_j(·), x^n_j(·))}_j ≥ 1. By the preceding proof, there exists a further subsequence, which we denote by {(θ^n'_j(·), x^n'_j(·))}_j ≥ 1, that converges in distribution to a process (θ(·), x(·)) with the properties described above.
More conveniently, as in <cit.>, let us work with the Skorohod representation of these processes. 
Specifically, by <cit.>, there exist some probability space and processes {(θ̃^n'_j(·), x̃^n'_j(·))}_j ≥ 1 and (θ̃(·), x̃(·)) defined on that probability space such that
 


  (i) each of the tilde processes has the same probability distribution as the corresponding process without tilde;

  (ii) {(θ̃^n'_j(·), x̃^n'_j(·))}_j ≥ 1 converges almost surely to (θ̃(·), x̃(·)).


Because the limit process (θ̃(·), x̃(·)) has continuous paths, the convergence of (θ̃^n'_j(·), x̃^n'_j(·)) to (θ̃(·), x̃(·)) in (ii) is uniform on any compact interval of t.
We then have x̃(·) ≡x̅(θ̃(0)) a.s. and 

    θ̃^n'_j(0) -  θ̃(0)  + x̃^n'_j(0) - x̅(θ̃(0) )  0,     as  j →∞.

Since x̅(θ) is a continuous function of θ, the above implies that
x̃^n'_j(0) - x̅( θ̃^n'_j(0) ) 0 as j →∞. In turn, since all the processes lie in the bounded set B_θ× B_x, 
by the bounded convergence theorem, we have lim_j →∞[ x̃^n'_j(0) - x̅( θ̃^n'_j(0) ) ] = 0, which is the same as 

    lim_j →∞[  x^n'_j(0) - x̅(θ^n'_j(0) ) ]  = lim_j →∞[  x_n'_j - x̅(θ_n'_j) ] = 0,
  
because (θ̃^n'_j(·), x̃^n'_j(·)) has the same probability distribution as (θ^n'_j(·), x^n'_j(·)). But the last equality above contradicts (<ref>). Therefore, (<ref>) must hold.

We now consider the slow time-scale determined by {α_n} and rewrite the θ-iterates (<ref>) of the algorithm as

    θ_n+1    = Π_B_θ( θ_n + α_n (g(θ_n, x̅ (θ_n), Z_n)  + Δ_n ) ), 
    where    Δ_n     = g(θ_n, x_n, Z_n) - g(θ_n, x̅ (θ_n), Z_n).

We view this as a single-time-scale algorithm and apply again <cit.> or more precisely, the proof of that theorem. 
As before, the constraint set B_θ and the stepsize {α_n} satisfy the conditions in the theorem, and the uniform integrability and tightness conditions in the theorem are entailed by our conditions (i)-(ii) and (vi) listed in Section <ref>, which are all satisfied by the algorithm. Another condition is the continuity of g(θ, x̅(θ), z) in θ uniformly in z in a compact set, which corresponds to the condition (A.8.2.3) in <cit.> and which is implied by our condition (iii) and the uniform continuity of the function x̅(θ) on the bounded set B_θ. What remains to be shown is that there is a mean ODE corresponding to (<ref>).

The desired mean ODE is (<ref>), namely,

    θ̇(t) = g̅(θ(t)) + z(t),     z(t) ∈ - _B_θ(θ(t)),
  
where z(t) is the boundary reflection term.
Lemma <ref>(ii) relates the averages of {g(θ, x̅ (θ), Z_n)} in (<ref>) to the function g̅(θ) in the ODE (<ref>), and it fulfills the condition (A.8.2.7) in <cit.>. 
The Δ_n terms in (<ref>)-(<ref>) are noise terms. We need to show that eventually their effects become negligible so that the average dynamics of the algorithm is characterized by the desired mean ODE. Specifically, if we can show for two arbitrary, fixed positive numbers t < T,

    lim sup_n →∞[ ∑_i= m(n, t)^m(n,T)α_i Δ_i ] = lim sup_n →∞[ ∑_i= m(n, t)^m(n,T)α_i (g(θ_i, x_i, Z_i) - g(θ_i, x̅(θ_i), Z_i) ) ] = 0,

then we can proceed as in the proof of <cit.> to obtain its conclusions.
[The term Δ_n here corresponds to the noise term β_n in <cit.>. Our condition on Δ_n is weaker than the condition (A.8.2.6) in <cit.> for noise terms. This is why, instead of directly applying the theorem, we need to follow its proof. The part of the proof related to the treatment of the noise terms appears in <cit.> (presented there are the proof arguments for the constant-stepsize case, which, apart from notational differences, are essentially the same as the arguments for the diminishing-stepsize case). What we need to do here is to use the property of the noise terms to eliminate the B(·) term in the equation (2.6) of that proof <cit.>. The condition we give above is sufficient for this purpose.]

For the preceding relation to hold, it is sufficient that

    lim sup_n →∞∑_i= m(n, t)^m(n,T)α_i  [   g(θ_i, x_i, Z_i) - g(θ_i, x̅(θ_i), Z_i) ] = 0.
 
Let us prove (<ref>) using (<ref>) and the uniform integrability of {g(θ_n, x_n, Z_n)} and {g(θ_n, x̅ (θ_n), Z_n)}.

For any ϵ, by the tightness of {_n} (implied by Prop. <ref>(ii)), there exists K_ϵ > 0 such that 

    inf_n ≥ 0(_n  > K_ϵ) < ϵ.
 
Since {g(θ_n, x_n, Z_n)} and {g(θ_n, x̅ (θ_n), Z_n)} are uniformly integrable (as verified in the previous subsection), by <cit.>, there exist constants η_ϵ which depend on ϵ, such that

    sup_n ≥ 0[ g(θ_n, x_n, Z_n)·( _n  > K_ϵ) ] < η_ϵ,     sup_n ≥ 0[  g(θ_n, x̅ (θ_n), Z_n) ·( _n  > K_ϵ)  ] < η_ϵ,
 
and η_ϵ→ 0 as ϵ→ 0.

The function g(θ, x, z) is Lipschitz continuous in x uniformly w.r.t. (θ, z) ∈ B_θ× D for a compact set D in the space of z. Hence, for some constant c_ϵ depending on K_ϵ,

    [  g(θ_n, x_n, Z_n) - g(θ_n, x̅(θ_n), Z_n) ·( _n ≤ K_ϵ) ] ≤ 
    c_ϵ [  x_n - x̅(θ_n) ],   ∀  n ≥ 0.

Combining this with (<ref>), we have

    lim_n →∞[  g(θ_n, x_n, Z_n) - g(θ_n, x̅(θ_n), Z_n) ·( _n ≤ K_ϵ) ] = 0.
 
Now ∑_i= m(n, t)^m(n,T)α_i  [  g(θ_i, x_i, Z_i) - g(θ_i, x̅(θ_i), Z_i) ] is bounded above by

    (T - t + α_m(n,t)) ·sup_m(n, t) ≤ i ≤ m(n,T)[  g(θ_i, x_i, Z_i) - g(θ_i, x̅(θ_i), Z_i) ·( _i ≤ K_ϵ) ] 
        + (T - t+ α_m(n,t)) ·sup_m(n, t) ≤ i ≤ m(n,T)[ ( g(θ_i, x_i, Z_i)   +  g(θ_i, x̅ (θ_i), Z_i) ) ·( _i  > K_ϵ) ].
   
(To see this, recall that by the definition (<ref>) of m(n, ·), ∑_i= m(n, t)^m(n,T)α_i ≤ T-t + α_m(n,t).) So, letting first n →∞ and then ϵ→ 0, and using first (<ref>) and (<ref>) (together with the fact that the stepsize is diminishing), and using then the fact lim_ϵ→ 0η_ϵ = 0, we obtain (<ref>).

Thus we have fulfilled the conditions to apply the proof of <cit.>. From it we obtain the conclusion that there exists a 
a sequence of positive numbers T_n →∞ such that for any ϵ > 0,

    lim sup_n →∞( θ_i ∉N_ϵ(L_B_θ),  some i ∈ [n, m(n, T_n) ] )  = 0,

where L_B_θ is the limit set of the ODE (<ref>). 
This is just the set Θ_opt by Lemma <ref>, so we obtain (<ref>). Finally, since {θ_n} is bounded and (<ref>) implies dist(θ_n, Θ_opt) → 0 in probability, the convergence is also in mean by <cit.>: lim_n →∞[ dist(θ_n, Θ_opt) ]= 0.




  §.§.§ Convergence proof for the constant-stepsize case
 

The two-time-scale GTDa algorithm (<ref>)-(<ref>) can be carried out using constant stepsizes instead of diminishing stepsizes, i.e., with α_n = α, β_n=β for all n and α < < β. An analogue of Theorem <ref> is given below for this constant stepsize algorithm. Imagine that we run the algorithm simultaneously for all possible stepsize parameters. The theorem concerns the asymptotic property of the algorithm, as the stepsize parameters α, β approach 0 in a way such that α/β→ 0. In the theorem the shorthand notation “i ∈ [0, T_α,β/α ]” refers to an iteration i with 0 ≤ i ≤ T_α,β/α.

 
Let Assumption <ref> hold. Let {(θ^α,β_n, x^α,β_n)} be generated by the two-time-scale GTDa algorithm (<ref>)-(<ref>) with constant stepsizes α_n = α and β_n = β for all n, where α << β. 
Moreover, let the constraint set B_x satisfy the condition (<ref>), and let the initial x^α,β_0=x_0, _0 ∈{ϕ()}.
Then for each given initial condition, the following holds: For any integers n_α such that α  n_α→∞ as α→ 0,
there exist positive numbers {T_α, β|α, β > 0} with T_α, β→∞ as (β, α/β) → 0, such that for any ϵ > 0,

    lim sup_β→ 0,  α/β→ 0( θ^α,β_n_α + i∉N_ϵ(Θ_ opt),  some i ∈ [0, T_α,β/α ] )  = 0.
  



The proof of Theorem <ref> given below parallels that of Theorem <ref> and uses essentially the same arguments. 
We will refer to the previous proof frequently to avoid repetition. First of all, we want to make sure that several conditions are met by the algorithm, so that we can apply the theorems from <cit.> for constant stepsize algorithms in our proof. These conditions are the same as the conditions given in Section <ref>, except that in the uniform integrability condition, iterates generated with different stepsize parameters all need to be considered. In particular, Conditions (ii)-(v) and (vii) remain the same. In Conditions (i) and (vi), the iterates (θ_n, x_n) now need to be indexed also by the stepsize parameters used to generate them. Specifically, besides those sets in Conditions (i) and (vi) that involve fixed (θ, x), the following sets of random variables are required to be uniformly integrable:

    {g(θ_n^α,β, x_n^α,β, Z_n) | n ≥ 0, α > 0, β > 0 },     {k(θ_n^α,β,x_n^α,β,Z_n)+_nω_n+1| n ≥ 0, α > 0, β > 0 }, 
       {g(θ_n^α,β, x̅(θ_n^α,β), Z_n) |  n ≥ 0, α > 0, β > 0  }.

As before, that these sets are uniform integrable follows from the uniform integrability of {_n} given in Prop. <ref>(ii).
Regarding other conditions used in the proof below, Remark <ref> given at the end of Section <ref> applies here as well.


Consider an arbitrary sequence of stepsize parameters (α^(m), β^(m)), m ≥ 1, that satisfies (α^(m), β^(m)) → 0 and α^(m)/β^(m)→ 0 as m →∞. For each m, let n_m = n_α for α = α^(m), and let (θ_n^(m), θ_n^(m)), n ≥ 0, denote the iterates generated by the algorithm with the stepsizes (α^(m), β^(m)).  

As before we view the algorithm as a single-time-scale algorithm, first for the fast time-scale and then for the slow time-scale. At the fast time-scale, for each m, we let g_m = α^(m)/β^(m)· g, and rewrite the algorithm as

    ( [ θ_n+1^(m); x_n+1^(m) ]) = Π_B_θ× B_x( [           θ_n^(m) + β^(m)  g_m(θ_n^(m), x_n^(m), Z_n); x_n^(m) + β^(m)( k(θ_n^(m), x_n^(m), Z_n) + _n ω_n+1) ]).

At the slow time-scale, for each m, we rewrite the θ-iterates of the algorithm as

    θ_n+1^(m)    = Π_B_θ( θ_n^(m) + α^(m) (g(θ_n^(m), x̅ (θ_n^(m)), Z_n)  + Δ_n^(m) ) ), 
    where    Δ_n^(m)    = g(θ_n^(m), x_n^(m), Z_n) - g(θ_n^(m), x̅ (θ_n^(m)), Z_n).

In analyzing (<ref>), our goal is to show the following analogue of (<ref>):

    lim sup_m →∞sup_i ≥ n_m[  x_i^(m) - x̅(θ_i^(m)) ] = 0.

We will then use this relation to bound the effects of the noise terms Δ_n^(m) in (<ref>) at the slow time-scale, similarly to the previous diminishing-stepsize case.

We consider first (<ref>) at the fast time-scale. Its associated mean ODE is again (<ref>).  We proceed with arguments paralleling those given in the previous proof, except that in place of <cit.>, we now apply <cit.> for single-time-scale, constant stepsize algorithms and we verify that the conditions in <cit.> are met.
[The verification step is the same as that given immediately after (<ref>) in the previous proof, except that we now use the second (instead of the first) relation in Lemma <ref>(iii) to characterize the average dynamics of the θ-iterates at the fast time-scale.]

With the latter theorem, we obtain the following conclusions. 
Let n'_m, m ≥ 1, be any sequence of integers satisfying β^(m) n'_m →∞ as m →∞.
For each m, let (θ^(m)(·), x^(m)(·)) be piecewise constant interpolations of the iterates (θ^(m)_n'_m+i, x^(m)_n'_m+i) given by 

    θ^(m)(t) = θ^(m)_n'_m+i   and  x^(m)(t) = x^(m)_n'_m+i    for  t ∈[ i β^(m), (i+1)  β^(m)),   i ≥ 0.

Then, by <cit.>, for each subsequence of {(θ^(m)(·), x^(m)(·))}_m ≥ 1, there is a further subsequence that converges in distribution to 
a process (θ(·), x(·)), where (θ(·), x(·)) has the same property as before. Namely, (θ(·), x(·)) has absolutely continuous paths satisfying the ODE (<ref>) and moreover, almost surely, its path lies in the limit set of the ODE (<ref>). 
As before, with the initial x_0, _0 ∈{ϕ()}, we are only interested in those solutions of the ODE with the initial x(0) ∈ B_x ∩{ϕ()}. For such initial conditions, by our assumption on B_x, the limit set of the ODE (<ref>) is {(θ, x̅(θ) ) |θ∈ B_θ} (Lemma <ref>).
So, almost surely, θ(·) ≡θ(0) ∈ B_θ and x(·) ≡x̅(θ(0)). 

We use the preceding conclusions to prove (<ref>). Suppose (<ref>) does not hold. Then there exists ϵ > 0 and a subsequence m_j and integers n'_j ≥ n_m_j such that

    [  x_n'_j^(m_j) - x̅(θ_n'_j^(m_j)) ] > ϵ,     ∀  j ≥ 1.

Now in the preceding conclusions, let n'_m_j  be n'_j (note that n'_j satisfies the requirement β^(m_j)  n'_j →∞ as j →∞, because β^(m_j)/α^(m_j)→∞ and α^(m_j) n'_j  ≥α^(m_j) n_m_j→∞). By the preceding proof, from the subsequence of interpolated processes {(θ^(m_j)(·), x^(m_j)(·))}_j ≥ 1, we can select a further subsequence that has the weak convergence properties described above. Denote the latter subsequence by {(θ^(m'_j)(·), x^(m'_j)(·))}. Then, as in the previous proof, working with the Skorohod representation of these processes allows us to conclude that 

    [  x^(m'_j)_n'_j - x̅(θ^(m'_j)_n'_j) ] → 0   as j →∞.

This contradicts (<ref>); therefore, (<ref>) must hold.

We now consider (<ref>) at the slow time-scale, and apply the proof of <cit.> to connect it with the desired mean ODE (<ref>). We proceed with arguments paralleling those given immediately after (<ref>) in the previous proof. As before, in order for the conclusions of <cit.> to hold for (<ref>), we need to show that the noise terms Δ_n^(m) in (<ref>) are well-behaved. Specifically,
we want to show that for two arbitrary, fixed positive numbers t < T,

    lim sup_m →∞[ ∑_i = n_m+ ⌊ t/α^(m)⌋^n_m + ⌊ T/α^(m)⌋α^(m)Δ_i^(m)] = 0.

Clearly, it is sufficient if we show

    lim sup_m →∞∑_i = n_m+ ⌊ t/α^(m)⌋^n_m + ⌊ T/α^(m)⌋α^(m)·[   g(θ_i^(m), x_i^(m), Z_i) - g(θ_i^(m), x̅(θ_i^(m)), Z_i) ] = 0.

To prove this, we proceed as in the last part of the previous proof (that is, the proof immediately after (<ref>)), using (<ref>) in place of (<ref>), and also using, as before, the uniform integrability of the sets 
{g(θ_n^α,β, x_n^α,β, Z_n) | n ≥ 0, α > 0, β > 0 } and {g(θ_n^α,β, x̅(θ_n^α,β), Z_n) |  n ≥ 0, α > 0, β > 0  }, together with the fact that for z in a compact set and θ∈ B_θ, the function g(θ, x, z) is Lipschitz continuous in x uniformly w.r.t. (θ, z).

Having verified the required conditions in order to apply the proof of <cit.>, we obtain the conclusions of <cit.> for the sequences of iterates {(θ_n^(m), θ_n^(m))}_n ≥ n_m, m ≥ 1.  
In particular, since Θ_opt is the limit set of the ODE (<ref>) (Lemma <ref>), we have that for any T > 0 and ϵ > 0,

    lim sup_m →∞( θ^(m)_n_m + i∉N_ϵ(Θ_opt),  some i ∈ [0, T/α^(m) ] )  = 0.
  
To finish the proof, we now show that (<ref>), together with the arbitrariness of the stepsize sequence {(α^(m), β^(m))}, implies the desired conclusion (<ref>). To this end, for each (α, β, ϵ, T), with c = α/β, define

    f(β,  c, ϵ, T) = ( θ^α, β_n_α + i∉N_ϵ(Θ_opt),  some i ∈ [0, T/α ] ).

Consider the sets A_j = { (β, c) |β≤ 1/j,   c ≤ 1/j }, j ≥ 1.
For each T, ϵ > 0, by (<ref>), 

    sup_(β, c) ∈ A_j f(β, c, ϵ, T) → 0  as  j →∞.

(Otherwise, we would be able to choose a sequence {(α^(m), β^(m))} with β^(m)→ 0 and α^(m)/β^(m)→ 0 that violates (<ref>); a contradiction.) This implies that for each m ≥ 1, there exists j_m such that

    sup_(β, c) ∈ A_j f(β, c, 1/m, m) ≤1/m,     ∀ j ≥ j_m.
   
Choose a strictly increasing sequence j_m, m ≥ 1, with the above property. 
Define sets B_m = A_j_m∖ A_j_m+1.  Then define the positive numbers T_α,β in the theorem as follows: 
for all (α, β) such that (β, α/β) ∈ B_m, let T_α, β = m. (Define T_α,β arbitrarily if (α, β) belongs to none of the sets {B_m}.)
With this definition of T_α,β, let us show that (<ref>) hold. For any ϵ > 0, there exists m̅ with ϵ > 1/m̅. 
If (α, β) is such that (β, α/β) ∈ B_m for some m ≥m̅,
the probability in the left-hand side (l.h.s.) of (<ref>) is 

    ( θ^α, β_n_α + i∉N_ϵ(Θ_opt),  some i ∈ [0, T_α,β/α ] ) = f(β, αβ, ϵ, m) ≤ f(β, αβ, 1m, m) ≤1/m,

where the last inequality follows from (<ref>) and our construction of B_m. If we let β→ 0 and α/β→ 0, then the corresponding points (β, α/β) enter sets B_m with m increasing to +∞. So the preceding relation shows that the limit on the l.h.s. of (<ref>) equals 0.





  §.§.§ Averaged iterates for the constant-stepsize case
 

We mention a result about the averaged iterates for the constant-stepsize case α_n = α, β_n=β with α < < β. It complements Theorem <ref> by characterizing the asymptotic behavior of averaged iterates for a fixed pair of stepsize parameters (α, β). Averaged iterates, such as 

    θ̅^α,β_n = 1/n - n_0∑_i=n_0^n-1θ^α,β_i
  
for some given n_0 ≥ 0, often behave better than the original iterates. Theoretical results regarding the acceleration of convergence by averaging are discussed in <cit.> and <cit.> (see also the references in the latter book). In our case, we do not yet have a rate of convergence analysis for the algorithms, so we cannot prove the acceleration of convergence for the averaged iterates.
[However, we did observe in a numerical study of ETD that averaging has great advantages in both the diminishing-stepsize case and the constant-stepsize case. For details of the study, see the companion note to the paper <cit.>.]

The result below about the averaged iterates (<ref>) is similar to a result proved in <cit.> in the ETD context and follows from the same analysis.

Let X_n = (S_n, y_n, _n, θ^α,β_n, x^θ,β_n) (treat y_n as a dummy variable in the case of state-dependent λ). 
Consider the m-step version of the Markov chain {X_n}: {(X_n, X_n+1, … X_n+m-1)}_n≥ 0. 
Denote the set of its invariant probability measures by ℐ^α,β_m. (The set ℐ^α,β_m is always nonempty, and it can have multiple elements, despite that the state-trace process has a unique invariant probability measure.)
For the invariant probability measures in ℐ^α,β_m, consider their marginal distributions on the space of the m θ-variables, and let ℐ̅^α,β_m denote the set of all these marginals.  
For ϵ > 0, write N'_ϵ(Θ_ opt) for the open ϵ-neighborhood of Θ_ opt,
and write [N'_ϵ(Θ_ opt)]^m for the Cartesian product of m copies of  N'_ϵ(Θ_ opt).

The following theorem is analogous to <cit.>. The second part of the theorem bounds, in expectation, the maximal deviation from Θ_ opt of m consecutive averaged iterates. The bound is expressed in terms of the minimal probability mass that the distributions in ℐ̅^α,β_m assign to [N'_ϵ(Θ_ opt)]^m. The first part of the theorem says that for an arbitrarily small ϵ, this minimal probability mass increases to 1, as the stepsize parameters (α, β), as well as their ratio, approach zero.

 
In the setting of Theorem <ref>, consider the sequence of averaged iterates {θ̅^α,β_n} given by (<ref>). Then the following hold for any ϵ > 0 and m ≥ 1:



  (i) lim inf_β→ 0,  α/β→ 0 inf_μ∈ℐ̅^α,β_mμ( [N'_ϵ(Θ_ opt)]^m ) = 1
and more strongly, with m_α = ⌈mα⌉,

    lim inf_β→ 0,  α/β→ 0 inf_μ∈ℐ̅^α,β_m_αμ( [N'_ϵ(Θ_ opt)]^m_α) = 1.


  (ii) For each initial condition of the algorithm and each stepsize pair (α, β),

    lim sup_n →∞[ max_n ≤ i < n+m dist( θ̅_i^α,β, Θ_ opt) ] ≤ϵκ^α,β_m + 2   r_B_θ ( 1 - κ_m^α,β),

where r_B_θ is the radius of B_θ, and κ_m^α,β = inf_μ∈ℐ̅^α,β_mμ( [N'_ϵ(Θ_ opt)]^m ).

 


We briefly explain how the theorem is proved. The starting point of this analysis is the observation that with constant stepsizes, the iterates jointly with the states/memory states and traces, {(S_n, y_n, _n, θ^α,β_n, x^θ,β_n)}, form a weak Feller Markov chain. By exploiting this fact—recall that weak Feller Markov chains have nice ergodicity properties <cit.>, one can combine the convergence result of Theorem <ref>
[More precisely, we use a version of Theorem <ref> in which for each (α, β), the initial (S_0, y_0, _0, θ^α,β_0, x^θ,β_0) is distributed according to an invariant probability measure of the Markov chain {(S_n, y_n, _n, θ^α,β_n, x^θ,β_n)}.]

with ergodicity properties of weak Feller Markov chains <cit.> and with the ergodicity property of the state-trace process given in Section <ref>, to obtain the characterization of the asymptotic behavior of the averaged iterates {θ̅^α,β_n} given in the above theorem. 
Because such an analysis has already been carried out in detail, in the context of the ETD algorithm, in <cit.> and the same proof arguments can be applied here, we do not repeat the proofs. We only remark that the analysis given in <cit.> for ETD concerns the distances from the θ-iterates to a single desired limit point. These distances are replaced, in the case here, by the distances from the θ-iterates to the set Θ_opt. Because the distance function, dist(θ, Θ_opt), is a convex function of θ (since the set Θ_opt is convex), the same analysis given in <cit.> applies here with this replacement of the distant function. 


    


 §.§ GTDb
 
Consider now a constrained version of the two-time-scale GTDb algorithm (<ref>)-(<ref>):

    θ_n+1    =  Π_B_θ( θ_n + α_n (_n δ_n(v_θ_n) -   ρ_n (1 - λ_n+1)   _n+1ϕ(S_n+1)  ·_n^ x_n ) ), 
    
    x_n+1    = Π_B_x( x_n + β_n (_n δ_n(v_θ_n) - ϕ(S_n) ϕ(S_n)^ x_n) ).

It differs from the previous constrained GTDa algorithm only in the θ-iteration. Under the same conditions for GTDa, we show that it is minimizing J(θ) over B_θ with stochastic gradient-descent and has the same convergence properties.

The analysis is almost identical to that of GTDa given in the previous subsection. First, we write the iterates equivalently as 

    θ_n+1    = Π_B_θ( θ_n + α_n ( g(θ_n, x_n, Z_n) + _n ω_n+1) ), 
    
          x_n+1    = Π_B_x( x_n + β_n ( k(θ_n, x_n, Z_n) + _n ω_n+1) ),

where Z_n and the function k(·) are the same as in the case of GTDa, and only the function g(·) is now different. It is defined according to (<ref>) as follows: with z=(s, y, , s') (treat y as a dummy variable in the case of state-dependent λ), for state-dependent λ,

    g(θ, x, z) =  δ̅(s, s', v_θ) - ρ(s, s') (1 - λ(s'))  (s') ϕ(s') ·^ x,
 
and for history-dependent λ, 

    g(θ, x, z) =  δ̅(s, s', v_θ) -  ρ(s, s') (1 - λ(f(y, s'), ))  (s') ϕ(s') ·^ x,
 
where f is the function in (<ref>) that determines the next memory state.
We then consider the mean ODEs associated with the algorithm. For the fast time-scale, we calculated k̅(θ, x) = _ζ[ k(θ, x, Z_0) ] already in Section <ref>. For the slow time-scale, let x̅(θ) = x_θ as before (cf. (<ref>)-(<ref>) and (<ref>)), and let us calculate g̅(θ) = _ζ[ g(θ, x̅(θ), Z_0) ] for each fixed θ.
Using the expressions (<ref>) and (<ref>) given in Prop. <ref> and using also the second expression (<ref>) of ∇ J(θ), we have

    g̅(θ) = _ζ[ g(θ, x̅(θ), Z_0) ]     =  _ζ[_0  δ̅_0(v_θ) ] -  _ζ[   ρ_0 (1 - λ_1)  _1ϕ(S_1) ·_0^x̅(θ)   ]  
        = Φ^Ξ  ( v_θ - v_θ) - (Φ^Ξ Φ)^ x_θ
        = - ∇ J(θ),
 
which is also the same as in the GTDa case. Therefore, if we impose the same condition (<ref>) on the constraint set B_x, the mean ODEs for the fast and slow time-scales are the same ODEs (<ref>) and (<ref>) for GTDa that we discussed in Section <ref>.

Next we check if the algorithm satisfies the conditions listed in Section <ref> and in Remark <ref>; if so, then we can apply the same convergence proof for GTDa. Clearly all the conditions that do not involve the function g are satisfied, since only g is different from the previous case. Those involving g are also satisfied, and it is straightforward to verify them using the above definition of g and the fact that for each (θ, x), the function g(θ, x, ·) is Lipschitz continuous in the trace variable . So we will omit the details except for one subtle point in the case of history-dependent λ. In that case, g(θ, x, z) is given by (<ref>), and for g(θ, x, ·) to be Lipschitz continuous in the trace variable , we need the term ρ(s, s') λ(f(y, s'), ) (s') ϕ(s') ·^ x, as a function of  for given x and (s, y, s'), to be Lipschitz continuous. This follows from Condition <ref>(i), which ensures the Lipschitz continuity of the function ↦λ(y̅, ) for each given memory state y̅, in the case of history-dependent λ.

We can now conclude that the convergence theorems we proved for GTDa also hold for GTDb:


 
Consider the two-time-scale GTDb algorithm (<ref>)-(<ref>). Then in the case of diminishing (constant) stepsize, under the same conditions in Theorem <ref> (Theorem <ref>), the conclusions of Theorem <ref> (Theorems <ref> and <ref>) hold. 






 §.§ Biased Variant of GTD
 

For the case of state-dependent λ, we discuss in this subsection a biased variant algorithm that can “robustfy” the off-policy TD algorithms against the high variance issue in off-policy learning. The price it pays for being more robust is that some bias is introduced, so that instead of approaching the set Θ_opt, it can only approach a neighborhood of Θ_opt, with the size of the neighborhood depending on the degree of its “bias.”  The idea has been applied to the ETD algorithm in <cit.> and to the off-policy LSTD algorithms in <cit.>, with its effectiveness observed experimentally and reported in those references.
In the present case of gradient-based TD algorithms, their variant algorithms can be regarded as approximate (stochastic) gradient-descent algorithms. 
 
In what follows, let us first describe the biased variant of GTD, before we explain further its motivation. A convergence analysis of the variant algorithm will be given after that, starting with its associated mean ODEs.

The biased variant we consider is, for the GTDa algorithm (<ref>)-(<ref>), given by

    θ_n+1    = Π_B_θ( θ_n + α_n  ρ_n (ϕ(S_n) - _n+1ϕ(S_n+1) ) · h(_n)^ x_n  ), 
    
       x_n+1    = Π_B_x( x_n + β_n (h(_n) ·δ_n(v_θ_n) - ϕ(S_n) ϕ(S_n)^ x_n) ),
 
where we have replaced _n in the iterates with a certain bounded continuous function h(_n) of _n, in order to robustify GTDa.
A similar variant for the GTDb algorithm (<ref>)-(<ref>) is 

    θ_n+1 = Π_B_θ( θ_n + α_n ( h(_n)  δ_n(v_θ_n) -  ρ_n (1 - λ_n+1)   _n+1ϕ(S_n+1) · h(_n)^ x_n ) ),

with the same iteration (<ref>) for x_n+1. 

Regarding the function h, besides boundedness, we require it to be Lipschitz continuous. 
In order to relate the degree of its “bias” to the behavior of the variant algorithm, we will consider a family of functions h_K parametrized by K > 0, each of which can serve as h, and for each K > 0, we require that h_K : → is a bounded Lipschitz continuous function such that

    h_K() ≤  ∀ ∈^d,   and   h_K() =   if ≤ K.

The parameter K reflects the degree of bias: the larger K is, the smaller the bias. We shall also assume that

    h_K() ∈{ϕ()},     ∀ ∈{ϕ()}.

This is mostly for simplicity: when the matrix Φ has full column rank, (<ref>) is satisfied trivially; when Φ does not have full column rank, it allows us to keep exploiting the nice subspace {ϕ()} and focus the analysis on important issues, without being sidetracked. A simple example of functions with the properties (<ref>)-(<ref>) are functions that downscale  when _2 is too large. Simply truncating  componentwise using some large thresholds also works when Φ has full column rank.

Let us now explain the motivation for using bounded h(_n) in place of _n in the algorithms. With state-dependent (or constant) λ, in general, {_n} can have unbounded variances and are also naturally unbounded in common off-policy learning situations (see a related result and its discussion in <cit.>). One can understand this unboundedness behavior based on the ergodicity of the state-trace process: if the invariant probability measure ζ has an unbounded support on the trace space, then, since the process is ergodic, {_n} will surely visit every part of the support however far away it is from the origin and however small its probability mass is under ζ. When {_n} visits such a distant part, its magnitude can stay large for many consecutive iterations since _n's are calculated iteratively. This will result in drastic changes in the (θ_n, x_n)-iterates, causing the algorithms to “forget” rapidly what they have “learned,” if no measure is taken to prevent such drastic changes from happening. If we replace _n by h(_n) in the (θ_n, x_n)-iterates, a “bias” is introduced when _n visits the distant part of its space on which h() “downsizes” . But if that part has a small probability mass under ζ, the “total bias” introduced by using h(_n) will also be small. 
This is the motivation of the above variant algorithm, and it is also reflected in the convergence analysis that we give below (cf. the proof of Lemma <ref> below).




  §.§.§ Associated mean ODEs
 

We start by calculating the functions involved in the desired mean ODEs. We do this for the biased variant of GTDa; the case of GTDb is similar. 
We shall assume Assumption <ref> and the conditions (<ref>)-(<ref>) on the Lipschitz continuous functions h, h_K throughout the analysis; to be concise, we will not mention these conditions explicitly in the lemmas we derive below.
First, we rewrite the algorithm (<ref>)-(<ref>) as

    θ_n+1    = Π_B_θ( θ_n + α_n   g_h(θ_n, x_n, Z_n) ),  
    
          x_n+1    = Π_B_x( x_n + β_n (k_h(θ_n, x_n, Z_n) + h(_n)  ω_n+1 ) ),

where, with z=(s,,s'),

    g_h(θ, x, z) =  ρ(s, s') (ϕ(s) - (s') ϕ(s') ) · h()^ x,    k_h(θ, x, z) = h()  δ̅(s, s', v_θ) - ϕ(s) ϕ(s)^ x.

For each fixed (θ, x), since g_h(θ, x, z) and k_h(θ, x, z) are bounded functions of z, _ζ[ g_h(θ, x, Z_0) ] and _ζ[ k_h(θ, x, Z_0) ] are well-defined and finite. For the fast time-scale, we define

    k̅_h(θ, x)  := _ζ[ k_h(θ, x, Z_0) ]  
          = _ζ[ h(_0)  δ̅_0(v_θ) ] - _ζ[ ϕ(S_0) ϕ(S_0)^]   x 
         = _ζ[ h(_0)  δ̅_0(v_θ) ] - Φ^Ξ Φ  x.

It is an affine function of (θ, x). Observe the following:

 
For each θ, the linear system of equations, k̅_h(θ, x) = 0, x ∈{ϕ()},
has a unique solution x̅_h(θ), and it is a continuous function of θ.


First, we show _ζ[ h(_0)  δ̅_0(v_θ) ] ∈{ϕ()} (otherwise k̅_h(θ, x) = 0, x ∈{ϕ()} does not have a solution). In view of the condition (<ref>) on h(·), it is sufficient to show that w.r.t. the invariant probability measure ζ of the state-trace process {(S_n, _n)}, the set ×{ϕ()} has measure 1.  
By Theorem <ref>(i), from any initial condition of (S_0, _0), the sequence of occupation probability measures converge almost surely to ζ. On the other hand, if we let _0 ∈{ϕ()}, then by the definition (<ref>) of {_n}, _n  ∈{ϕ()} for all n, so the occupation probability measures all assign measure 1 to the set ×{ϕ()}. It then follows that ζ must also assign measure 1 to this set. Consequently, _ζ[ h(_0)  δ̅_0(v_θ) ] ∈{ϕ()}. From this it follows that the equation Φ^Ξ Φ  x = _ζ[ h(_0)  δ̅_0(v_θ) ] has a unique solution in {ϕ()}, or equivalently, the solution to k̅_h(θ, x) = 0, x ∈{ϕ()} is unique. 

To see that the solution x̅_h(θ) is continuous in θ, note that x̅_h(θ)_2 ≤_ζ[ h(_0)  δ̅_0(v_θ) ] _2 /c, where c is the smallest nonzero eigenvalue of the matrix Φ^Ξ Φ. Since _ζ[ h(_0)  δ̅_0(v_θ) ] is an affine function of θ, the preceding bound on x̅_h(θ)_2 implies that for any point θ', as θ→θ', x̅_h(θ) lies in a bounded set, and then the limit of any of its convergent subsequences must be a solution to k̅_h(θ', x) = 0, x ∈{ϕ()}, by the continuity of k̅_h(·). But the latter equation has a unique solution x̅_h(θ'), so we must have x̅_h(θ) →x̅_h(θ') as θ→θ'. This proves the continuity of x̅_h(·).



Henceforth, let x̅_h(θ) be as in Lemma <ref>.
The desired mean ODE for the fast time-scale is

    ( [ θ̇(t);  ẋ(t) ]) 
     =  ( [                         0; k̅_h (θ(t), x(t) ) + z(t) ]),     θ(0) ∈ B_θ,   z(t) ∈ - _B_x(x(t)),
 
where z(t) is the boundary reflection term. Similar to the condition (<ref>) on B_x for GTDa, we shall require
[Note that a sufficient condition for (<ref>) is that
radius(B_x) ≥sup_θ∈ B_θ_ζ[ h(_0)  δ̅_0(v_θ) ] _2  /c,
where c is the smallest nonzero eigenvalue of the matrix Φ^Ξ Φ.]


    B_x ⊃{x̅_h(θ) |θ∈ B_θ}.
 
This condition ensures that for an initial condition θ(0)=θ and x(0) ∈{ϕ()}, the solution x(t) of the ODE (<ref>) converges to x̅_h(θ) as t →∞. Moreover, we have an analogue of Lemma <ref> about the limit set of the ODE (<ref>), which follows from the same proof of Lemma <ref> (with k̅_h, x̅_h in place of k̅, x̅) and which will be needed in analyzing the variant algorithm at the fast time-scale:


 
Let the constraint set B_x satisfy (<ref>). Let x̅_h(θ) be as in Lemma <ref>. Then for all initial x(0) ∈ B_x ∩{ϕ()} and θ(0) ∈ B_θ,
the limit set of the ODE (<ref>) is

    ⋂_t̅≥ 0  {(θ(t), x(t))  |  θ(0) ∈ B_θ,   x(0) ∈ B_x ∩{ϕ()},   t ≥t̅ }  = {(θ, x̅_h(θ)) |θ∈ B_θ}.



For the slow time-scale, we define, for each θ,

    g̅_h(θ)  :=  _ζ[ g_h(θ, x̅_h(θ), Z_0) ] =  _ζ[  ρ_0 (ϕ(S_0) - _1ϕ(S_1) ) · h(_0)^x̅_h(θ) ].

It is a continuous function of θ, since the function x̅_h(θ) is continuous.
The desired mean ODE is

    θ̇(t) = g̅_h(θ(t)) + z(t),     z(t) ∈ - _B_θ(θ(t)),

where z(t) is the boundary reflection term. 
We want to characterize the relation between its limit set and the set Θ_opt, when h is chosen from the family of functions h_K, K > 0 mentioned earlier and with large values of K (which correspond to small biases). 

Let x̅_K(·) and g̅_K(·) stand for the function x̅_h(·) and g̅_h(·), respectively, when h = h_K. 
The next lemma shows the approximate gradient-descent nature of the variant algorithm.


  
lim_ K →∞sup_θ∈ B_θg̅_K(θ) + ∇ J(θ)  = 0.
 


First, recall an expression of the gradient ∇ J(θ) from the analysis of the mean ODEs of GTDa in Section <ref> (cf. (<ref>)-(<ref>)): for each θ∈^d,  
∇ J(θ) = - _ζ[  ρ_0 (ϕ(S_0) - _1ϕ(S_1) ) ·_0^ x_θ], 
where x_θ is the unique solution of the equation Φ^Ξ Φ  x = _ζ[ _0  δ̅_0(v_θ) ] in {ϕ()}.
Then by (<ref>),

    g̅_K(θ) + ∇ J(θ)      =  _ζ[  ρ_0 (ϕ(S_0) - _1ϕ(S_1)  ) · h_K(_0)^x̅_K(θ)  ]
          -    _ζ[  ρ_0 (ϕ(S_0) - _1ϕ(S_1) ) ·_0^ x_θ] 
        =  _ζ[  ρ_0 (ϕ(S_0) - _1ϕ(S_1) ) · h_K(_0)^( x̅_K(θ) - x_θ) ] 
                    +  _ζ[  ρ_0 (ϕ(S_0) - _1ϕ(S_1) ) ·(h_K(_0) - _0   )^ x_θ].

So to prove the lemma, it suffices to prove that as K →∞,

    sup_θ∈ B_θx̅_K(θ) - x_θ→ 0   and  _ζ[  h_K(_0) - _0 ]  → 0.

Since _ζ[  h_K(_0) - _0 ] ≤ 2 _ζ[ _0 (_0 > K) ] by (<ref>), the second relation in (<ref>) follows from _ζ[ _0  ] < ∞ (Theorem <ref>(ii)). To show the first relation in (<ref>), note that by the definition of x̅_K(θ) (cf. Lemma <ref>), x̅_K(θ) - x_θ is the unique solution of the following equation (in x) in the subspace {ϕ()}:

    Φ^Ξ Φ  x = _ζ[ (h_K(_0)  - _0 ) ·δ̅_0(v_θ) ].

This implies that x̅_K(θ) - x_θ_2 ≤_ζ[ (h_K(_0)  - _0 ) ·δ̅_0(v_θ) ] _2 /c, where c is the smallest nonzero eigenvalue of the matrix Φ^Ξ Φ. On the other hand, 

    sup_θ∈ B_θ_ζ[  h_K(_0)  - _0 · |δ̅_0(v_θ)| ]  → 0   as K →∞,

since sup_θ∈ B_θsup_s, s' ∈ | δ̅(s, s', v_θ) | < ∞ and lim_K →∞_ζ[ h_K(_0)  - _0 ]  = 0 as just proved. Therefore, sup_θ∈ B_θx̅_K(θ) - x_θ→ 0 as K →∞, proving the first relation in (<ref>). The proof is now complete.


Below is an analogue of Lemma <ref> about the limit set of the ODE (<ref>), which will be needed in analyzing the variant algorithm at the slow time-scale:

 
For any ϵ > 0, there exists K_ϵ > 0 such that if K ≥ K_ϵ, then with g̅ = g̅_K,
the limit set of the ODE (<ref>) satisfies
⋂_ t̅≥ 0 {θ(t)   |  θ(0) ∈ B_θ,   t ≥t̅ }⊂ N_ϵ ( Θ_ opt).



Consider a solution θ(·) of (<ref>) with g̅_h = g̅_K for some K. At time t, denote D=_B_θ(θ(t)) and recall that the boundary reflection term z(t) = - Π_Dg̅_K(θ(t)). Define z̃(t) = - Π_D(- ∇ J(θ(t))).
Then, by the non-expansiveness of the projection operator Π_D,

    z(t) - z̃(t) _2 ≤g̅_K(θ(t)) + ∇ J(θ(t)) _2.
   
We now calculate V̇(θ(t)) for the function V(θ) = J(θ):

    V̇(θ(t))     = ⟨∇ J(θ(t)),  g̅_K(θ(t)) + z(t) ⟩
        =   ⟨∇ J(θ(t)),    - ∇ J (θ(t)) + z̃(t) ⟩ + ⟨∇ J(θ(t)),  g̅_K(θ(t)) + z(t) + ∇ J (θ(t)) - z̃(t)  ⟩
       ≤⟨∇ J(θ(t)),    - ∇ J (θ(t)) + z̃(t) ⟩  + ∇ J(θ(t)) _2 · g̅_K(θ(t)) + z(t)  + ∇ J(θ(t)) -  z̃(t) _2  
       ≤⟨∇ J(θ(t)),    - ∇ J (θ(t)) + z̃(t) ⟩  + ∇ J(θ(t)) _2 ·   2   g̅_K(θ(t)) + ∇ J(θ(t)) _2   ,

where the last inequality follows from (<ref>) and the triangle inequality.
As we showed in the proof of Lemma <ref> for GTDa, the first term in (<ref>) is always nonpositive and moreover,
for any 0 < ϵ≤sup_θ∈ B_θ J(θ) - inf_θ∈ B_θ J(θ), there exists η_ϵ > 0 such that

    ⟨∇ J(θ(t)),    - ∇ J (θ(t)) + z̃(t) ⟩≤ - η_ϵ  if  J(θ(t)) ≥inf_θ∈ B_θ J(θ) +ϵ.

For the second term in (<ref>), we can upper-bound it by 

    Δ_K : = sup_θ∈ B_θ∇ J(θ) _2 · 2  sup_θ∈ B_θg̅_K(θ) + ∇ J(θ) _2,

which, by Lemma <ref>, converges to 0 as K → +∞. Therefore, we can choose K̅_ϵ sufficiently large so that η_ϵ - Δ_K ≥η_ϵ/2 for all K ≥K̅_ϵ. Together with (<ref>) and (<ref>), this shows that for any solution of the ODE corresponding to such K, there exists a time τ_ϵ (which depends on η_ϵ but not on K) such that whenever θ(t) is outside the set 
E_ϵ := {θ| J(θ) ≤inf_θ∈ B_θ J(θ) + ϵ}∩ B_θ, it will visit E_ϵ before the amount of time τ_ϵ has elapsed.

Now assume that at some moment t_0, θ(t_0) ∈ E_ϵ. Let us check how far away from the set E_ϵ the solution θ(t) can wander before it revisits this set. During such an excursion, by (<ref>), the maximal amount of increase in V(θ) relative to V(θ(t_0)) is bounded by τ_ϵΔ_K. Hence, once the solution hits E_ϵ, it will not leave the set E_ℓ, where ℓ =  τ_ϵΔ_K + ϵ.

Finally, to finish the proof, consider any given ϵ' > 0 (it is the given ϵ in the lemma). There exists a sufficiently small ϵ̃> 0 such that E_ϵ̃⊂ N_ϵ'(Θ_opt).
[Otherwise, there would be a sequence of points {θ_i} in B_θ that converges to θ_∞ and has the property that lim_i →∞ J(θ_i) = J(θ_∞) = inf_θ∈ B_θ J(θ) and yet θ_∞∉Θ_opt, a contradiction.]

We first choose ϵ < ϵ̃ and then choose a threshold K̅' ≥K̅_ϵ sufficiently large so that ℓ < ϵ̃ and hence E_ℓ⊂ E_ϵ̃ for all K ≥K̅'. (This is possible because in the expression of ℓ above, we have Δ_K → 0 as K →∞ by Lemma <ref>.)
It then follows that for all K ≥K̅', the limit set of the ODE (<ref>) with g̅ = g̅_K must be contained in E_ϵ̃⊂ N_ϵ'(Θ_opt).




  §.§.§ Convergence properties


We are now ready to address the convergence properties of the variant algorithm. 
It is straightforward to check that all the conditions used in our convergence proof of GTDa are satisfied by the variant algorithm, by using the definition of the functions g_h, k_h and by using the results about the desired mean ODEs for the fast and slow time-scales that we just derived.
[In particular, among the conditions listed in Section <ref>, the uniform integrability conditions are trivially satisfied because the boundedness of the function h() makes g_h(θ, x, z) and k_h(θ, x, z) bounded on B_θ× B_x ×. The functions g_h(θ, x, z) and k_h(θ, x, z) are simply affine functions of (θ, x) for each z. By their definitions, the uniform continuity condition (iii) on g_h and k_h are also trivially satisfied, and so does the Lipschitz continuity condition on g_h(θ, ·, z) mentioned in Remark <ref>, which is required in the convergence proof. For the “averaging conditions” (iv) and (vii), as the proof of Lemma <ref> showed, they are satisfied if for each θ and x, the functions k_h(θ, x, z) and g_h(θ, x̅(θ), z) are Lipschitz continuous in the trace variable . This is true since h() is Lipschitz continuous by assumption.]

We can thus apply the convergence proof for GTDa to the variant algorithm for each choice h = h_K, with the corresponding condition (<ref>) on the constraint set B_x. (We remark that the threshold for the radius will depend on K, but is bounded above for all K because of (<ref>).)
The result is that the conclusions of Theorems <ref>-<ref> for GTDa hold also in this case, with the limit set of the slow-time-scale ODE (<ref>) in place of the set Θ_ opt. We then combine these conclusions with Lemma <ref> to obtain Theorem <ref>(i)-(ii) given below. 


The theorem is stated for both GTDa and GTDb variants—the analysis of the GTDb variant is essentially the same as that of the GTDa variant we gave.
The conclusions about the biased variant algorithms are very similar to those about GTD given earlier (cf. Theorems <ref>-<ref>). The only difference is that here, in order for the algorithms to approach a given small neighborhood of Θ_ opt, K needs to be sufficiently large (so that the bias is sufficiently small). 

The property of the averaged iterates is addressed in the part (iii) of the theorem, where '_ϵ(Θ_ opt) is the open ϵ-neighborhood of Θ_ opt, and the set ℐ̅^α,β_m of probability measures is as defined in Section <ref> before Theorem <ref>.  Like the latter theorem, the proof of this part is essentially the same as that given in <cit.> in the context of ETD and therefore omitted. (The result in the part (iii) is analogous to <cit.> for biased variants of ETD.) Compared with Theorem <ref> for GTDa, the only difference is that here, for a desired small ϵ-neighborhood '_ϵ(Θ_ opt), in order for the minimal probability mass κ_m^α,β for this set to increase to 1 as (α, β) → 0 and α/β→ 0, K needs to be sufficiently large (i.e., the bias needs to be sufficiently small).

 
For the case of state-dependent λ, consider the two-time-scale biased GTD algorithm (<ref>)-(<ref>) or (<ref>)-(<ref>). In the algorithm, let the function h ∈{ h_K | K > 0}, a family of bounded Lipschitz continuous functions that satisfy (<ref>)-(<ref>).
In addition, let the same conditions in Theorem <ref> (Theorem <ref>) hold for the case of diminishing (constant) stepsize, except that the condition on the constrained set B_x is replaced by (<ref>) for the chosen h=h_K.
Then for each initial condition of the algorithm, the following hold:



  (i) In the case of diminishing stepsize, for each ϵ > 0, there exists K_ϵ > 0 such that if K ≥ K_ϵ, then 
it holds for some sequence of positive numbers T_n →∞ that 

    lim sup_n →∞( θ_i ∉N_ϵ(Θ_ opt),  some i ∈ [n, m(n, T_n) ] )  = 0.
  

  (ii) In the case of constant stepsize, for each ϵ > 0, there exists K_ϵ > 0 with the following property. If K ≥ K_ϵ, then for any integers n_α that satisfy α  n_α→∞ as α→ 0,
there exist positive numbers {T_α, β|α, β > 0} with T_α, β→∞ as β→ 0 and α/β→ 0, such that,

    lim sup_β→ 0,  α/β→ 0( θ^α,β_n_α + i∉N_ϵ(Θ_ opt),  some i ∈ [0, T_α,β/α ] )  = 0.


  (iii) In the case of constant stepsize, consider the averaged iterates {θ̅^α,β_n} defined by (<ref>). Then 
for each ϵ > 0, there exists K_ϵ > 0 such that if K ≥ K_ϵ, then it holds for all m ≥ 1 that
lim inf_β→ 0,  α/β→ 0 inf_μ∈ℐ̅^α,β_mμ( [N'_ϵ(Θ_ opt)]^m ) = 1,
and more strongly, with m_α = ⌈mα⌉,

    lim inf_β→ 0,  α/β→ 0 inf_μ∈ℐ̅^α,β_m_αμ( [N'_ϵ(Θ_ opt)]^m_α) = 1.

Furthermore, regardless of the choice of K, for each ϵ > 0 and each stepsize pair (α, β),

    lim sup_n →∞[ max_n ≤ i < n+m dist( θ̅_i^α,β, Θ_ opt) ] ≤ϵκ^α,β_m + 2   r_B_θ ( 1 - κ_m^α,β),

where r_B_θ is the radius of B_θ, and κ_m^α,β = inf_μ∈ℐ̅^α,β_mμ( [N'_ϵ(Θ_ opt)]^m ). 






 §.§ Extension to a Composite Scheme of Setting λ-Parameters
 

In this subsection we explain a composite scheme of setting the λ-parameters, 
which builds upon the state-dependent and history-dependent λ considered so far in the paper but is more general than them.
With this composite scheme, a larger set of generalized Bellman operators can be utilized for the approximate policy evaluation task.
We will explain this and show how the GTD algorithms can be simply modified to minimize the projected-Bellman-error objective function J(θ), for the Bellman operators  associated with this scheme. A convergence theorem for the GTD algorithms will be given at the end. 

We shall use primarily the GTDa algorithm as an illustrative example, and mention the GTDb algorithm near the end, which is very similar. One can apply the same arguments given below to other algorithms discussed earlier or to be discussed in the next section. 



  §.§.§ Multiple concurrent trace processes and associated Bellman operators
 

To explain the composite scheme of setting λ and its underlying idea, it is convenient to start with an illustrative example of an algorithm that uses such a scheme. Consider the GTDa algorithm (<ref>)-(<ref>). 
Let (θ_n, x_n) be generated according to these same formulae, but with the trace iterates _n calculated differently as follows. 
Partition the state space  into disjoint subsets, say, = ∪_i=1^ℓ_i. 
Associate each subset _i with a way of setting the λ-parameters λ_n^(i) and with a trace sequence _n^(i) generated according to the recursion

    ^(i)_n = λ_n^(i)_n ρ_n-1^(i)_n-1 + ϕ(S_n) ·(S_n ∈_i).

Then take their sum to be the trace _n used in the algorithm:

    _n = ∑_i=1^ℓ^(i)_n.
 
As will be seen shortly, the idea of the composite scheme is quite general. Here, for the purpose of reusing the previous analyses, however, we shall restrict attention to the case where for each i ≤ℓ, the choice of λ_n^(i) is in the two classes (state-dependent and history-dependent λ) discussed in Section <ref> and obeys the conditions given there. Specifically, either λ_n^(i) = λ^(i)(S_n) for a given function λ^(i), or λ_n^(i) is determined through (<ref>) based on the memory states and the previous trace in the i-th process: 

    y_n = f(y_n-1, S_n),     λ^(i)_n=λ^(i)(y_n, ^(i)_n-1).

In the latter case, we require that the memory states {y_n} satisfy Condition <ref>, and the λ^(i) function satisfies Condition <ref>. 

The idea behind (<ref>)-(<ref>) is that by decomposing the trace _n in this manner, one can compose a generalized Bellman operator  from selected component mappings of other Bellman operators and minimize the projected-Bellman-error objective function J(θ) for that . As a simple example, if = _1 ∪_2 and T^(0), T^(1) are the Bellman operators (for the target policy) associated with the constant λ = 0 and λ = 1, respectively, one can employ in J(θ) a composite  given by

    ( v)(s) = (T^(0) v)(s)    if s ∈_1;       ( v)(s) = (T^(1) v)(s)    if s ∈_2.

To minimize J(θ) for this  using GTDa, one just modifies the definition of _n in the algorithm as shown above; namely, let _n = ^(1)_n + ^(2)_n, where ^(1)_n and ^(2)_n are updated according to (<ref>) with λ_n^(1) = 0 and λ_n^(2) = 1, respectively. Such composite  is not possible with the form of state-dependent or history-dependent λ-parameters we considered earlier. By using such Bellman operators in the objective function J(θ), one can better regulate what information to use for approximating the value function v_π at specific states. 

This idea of setting λ and its implementation through (<ref>)-(<ref>) were proposed by the author in <cit.>. The properties of the associated state-trace processes and LSTD algorithms were further analyzed in <cit.>, with the second reference extending the scheme to include history-dependent λ's. We refer the reader to <cit.> (especially Section 3.1 and the discussion before Corollary 3.1 in Section 3.2 therein) for the details of the generalized Bellman operator  associated with (<ref>)-(<ref>). As before, for the purpose of this paper, we do not need the details of , and what we care about is that the algorithms with the above redefinition of the trace iterates are indeed minimizing J(θ) for the corresponding Bellman operator . So we will only describe  at an abstract level in terms of other Bellman operators considered earlier, before we proceed to discuss the convergence properties of the GTD algorithms.

With the new definition (<ref>)-(<ref>) for the trace iterates, instead of a single state-trace process, there are now ℓ concurrent processes, {(S_n, y_n, _n^(i)) }, i ≤ℓ, which share the same state and memory state variables (treat the memory states y_n as dummy variables if state-dependent λ's are used in the recursion (<ref>) for the i-th trace process).
Since the choice of λ in each of these state-trace processes is assumed to satisfy the same conditions as required before, there is a Bellman operator T^(λ^(i)) associated with the i-th process for each i ≤ℓ (cf. Section <ref>), and the theorems given in Section <ref> all hold for each of these ℓ state-trace processes individually.

Consider a Bellman operator  composed from the component mappings of T^(λ^(i)), i ≤ℓ, as follows: for any v ∈^||,

    ( v)(s)  := ( T^(λ^(i)) v )(s)    for s ∈_i.

This is the generalized Bellman operator corresponding to the above composite scheme of setting the λ-parameters. Let us now proceed to show that the correspondingly modified GTD algorithms minimize the projected-Bellman-error objective function J(θ), for this Bellman operator , and that the convergence properties we derived earlier GTD continue to hold for the modified algorithms.



  §.§.§ Convergence analysis for GTD


To present the analysis, we continue to use the GTDa algorithm as an illustrative example. (The corresponding GTDb algorithm will be addressed at the end; see (<ref>)-(<ref>).)
Substituting the new definition (<ref>) for _n, GTDa now becomes

    θ_n+1    = Π_B_θ( θ_n + α_n  ∑_i=1^ℓρ_n (ϕ(S_n) - _n+1ϕ(S_n+1) ) · (_n^(i))^ x_n  ),  
    
       x_n+1    = Π_B_x( x_n + β_n ( ∑_i=1^ℓ_n^(i)δ_n(v_θ_n) - ϕ(S_n) ϕ(S_n)^ x_n) ).
 
The first step is to identify the mean ODEs. By Theorem <ref>, for each i ≤ℓ, the Markov chain {(S_n, y_n, _n^(i)) } has a unique invariant probability measure ζ^(i). Since conditioned on the states and memory states, the evolution of traces in each of the ℓ processes is independent of one another, the joint state-trace process {(S_n, y_n, _n^(1), …, _n^(ℓ) )} also has a unique invariant probability measure ζ̃, which is determined by the collection ζ^(i), i ≤ℓ. Denote expectation w.r.t. the stationary joint state-trace process by _ζ̃. To identify the functions in the mean ODEs, we calculate the expectations of several functions of the states and traces, for those functions involved in the algorithm, w.r.t. the stationary joint state-trace process {(S_n, y_n, _n^(1), …, _n^(ℓ) )}.

Consider first the functions involved in the iteration (<ref>) at the fast time-scale. 
We have _ζ̃[ ϕ(S_0) ϕ(S_0)^] = Φ^Ξ Φ as before.
For each fixed θ and each i ≤ℓ, applying Prop. <ref> to the i-th state-trace process, we have

    _ζ̃[ _0^(i) δ̅_0(v_θ) ] = _ζ^(i)[ _0^(i) δ̅_0(v_θ) ] =  (Φ^(i))^Ξ (T^(λ^(i)) v_θ - v_θ),
 
where Φ^(i) is the matrix whose s-th row equals ϕ(s)^ if s ∈_i and equals the zero vector otherwise (cf. the definition (<ref>) of _n^(i)). Now by the definition of Φ^(i) and the definition (<ref>) of , 

    ∑_i = 1^ℓ(Φ^(i))^Ξ (T^(λ^(i)) v_θ - v_θ)  = Φ^Ξ ( v_θ - v_θ),

so we obtain that for each fixed (θ, x),

    _ζ̃[ ∑_i = 1^ℓ_0^(i) δ̅_0(v_θ) ] - _ζ̃[ ϕ(S_0) ϕ(S_0)^] x =  Φ^Ξ ( v_θ - v_θ)  - Φ^Ξ Φ  x  = : k̅(θ, x).

Note that the function k̅(θ, x) here is exactly the one for GTDa in Section <ref>, apart from involving a different .

Thus, for the mean ODE associated with the fast time-scale, we can proceed as in Section <ref>: For each θ, define x̅(θ) to be the unique solution x_θ to k̅(θ, x) = 0, x ∈{ϕ()}. The mean ODE is the same as (<ref>) (apart from a different definition of ). If we place the same condition (<ref>) on the constraint set B_x (with x_θ being defined w.r.t. the Bellman operator  here), then as before, for all initial conditions x(0) ∈{ϕ()}, the solution x(t) of the ODE converges to x̅(θ(0)) and the limit set of the ODE is characterized by Lemma <ref>.


Consider now the iteration (<ref>) at the slow time-scale. In terms of the function g defined in (<ref>) earlier, with Z^(i)_n = (S_n, y_n, _n^(i), S_n+1), we can write (<ref>) equivalently as

    θ_n+1  = Π_B_θ( θ_n + α_n  ∑_i = 1^ℓ  g(θ_n, x_n, Z^(i)_n) ).

For each fixed (θ, x) and each i ≤ℓ, by Prop. <ref> applied to the i-th state-trace process, we have

    _ζ̃[ g(θ, x, Z^(i)_n) ]     = _ζ^(i)[ g(θ, x, Z^(i)_n) ] 
        =   _ζ^(i)[  ρ_0 (ϕ(S_0) - _1ϕ(S_1) ) · (_0^(i))^ x ] 
        = [ (Φ^(i))^Ξ  (I - P^(λ^(i)))   Φ]^ x,

where Φ^(i) is as defined above, and P^(λ^(i)) is the substochastic matrix in the operator T^(λ^(i)). 
Thus,

    _ζ̃[  ∑_i = 1^ℓ  g(θ, x, Z^(i)_n) ]      =  ∑_i = 1^ℓ [ (Φ^(i))^Ξ  (I - P^(λ^(i)))   Φ]^ x  
        = [ Φ^Ξ  (I - )  Φ]^ x,

where  is the substochastic matrix in the affine operator , and the second equality follows from the definition of Φ^(i) and the definition (<ref>) of . Then for x = x̅(θ), (<ref>) yields

    _ζ̃[  ∑_i = 1^ℓ  g(θ, x̅(θ), Z^(i)_n) ] = [ Φ^Ξ  (I - )  Φ]^x̅(θ)  =: g̅(θ)

and g̅(θ) = - ∇ J(θ) by the gradient expression (<ref>). This is also the same as in Section <ref>, apart from that a different operator  in involved in defining the functions g̅ and J here. The mean ODE for the slow time-scale is thus the same as (<ref>), and its limit set, by Lemma <ref>, is the set 

    Θ_ opt = _θ∈ B_θ J(θ) = _θ∈ B_θ12Π_ξ( v_θ - v_θ) _ξ^2,

for the Bellman operator  defined in (<ref>).

Finally, to apply the same convergence proofs given in Sections <ref>-<ref>, we need to check if the algorithm here meets the conditions required by those analyses, including uniform integrability, continuity, and averaging conditions, etc.
To see which conditions to verify, it is convenient to work with the same formulae used earlier for the GTDa algorithm, with the trace now being the “total trace” _n = ∑_i = 1^ℓ^(i)_n:

    θ_n+1    = Π_B_θ( θ_n + α_n   g(θ_n, x_n, Z_n) ), 
    
          x_n+1    = Π_B_x( x_n + β_n (k(θ_n, x_n, Z_n) + _n ω_n+1 ) ).
  
In the above, we take Z_n to be (S_n, y_n, _n, S_n+1) together with _n^(i), i ≤ℓ. We take the functions g, k to be the ones defined earlier in (<ref>), whose values depend only on the state transition and the “total trace” _n. Since these are the same functions as before and only the trace iterates are different, what we need to verify are just those conditions that involve the traces iterates {_n}.

Consider first the uniform integrability conditions (i) and (vi) listed in Section <ref>. As mentioned there, these conditions are satisfied if {_n} is uniformly integrable. In the case here, by Prop. <ref>(ii), for each i ≤ℓ, {_n^(i)} is uniformly integrable. Then, since _n = ∑_i = 1^ℓ_n^(i), as a simple consequence of the definition of uniform integrability, {_n} is uniformly integrable as well. So the uniform integrability conditions (i) and (vi) are met. Similarly, the tightness condition (ii) listed in Section <ref> is met, because Prop. <ref>(ii) implies the tightness of each {_n^(i)}, i ≤ℓ, and as a result, both {_n} and the joint trace process {(_n, _n^(1), …, _n^(ℓ))} are also tight. 

What remain to be verified are the averaging conditions (iv) and (vii) listed in Section <ref>, as well as Lemma <ref>(iii), which corresponds to part of the averaging conditions needed for analyzing the θ-iterates at the fast time-scale. These conditions are also satisfied, thanks to the properties of the ℓ concurrent state-trace processes. We prove this in the lemma below, which is an analogue of Lemma <ref>. In the lemma  denotes the space of Z_n. 

 
In the above context, for each θ∈ B_θ, x ∈ B_x, the following hold:



  (i) For each compact set D ⊂,

    lim_m, n →∞1/m∑_i=n^n+m-1_n [ k(θ, x, Z_i) - k̅(θ, x) ] (Z_n ∈ D) = 0   in mean.


  (ii) For each compact set D ⊂,

    lim_m, n →∞1/m∑_i=n^n+m-1_n [ g(θ, x̅(θ), Z_i) - g̅(θ) ] (Z_n ∈ D) = 0   in mean.


  (iii)If {α_n} and {β_n} satisfy Assumption <ref>, then

    lim_m, n →∞1/m∑_i=n^n+m-1_n [ (α_i/β_i) · g(θ, x, Z_i) ] = 0   in mean.

For any positive sequence c_j → 0 as j →∞,

    lim_m, n, j →∞1/m∑_i=n^n+m-1_n [ c_j · g(θ, x, Z_i) ] = 0   in mean.





For j ≤ℓ, let Z_n^(j) = (S_n, y_n, _n^(j), S_n+1) (treat the memory state y_n as a dummy variable if the j-th trace process is generated with state-dependent λ). Let D^j be the projection of the compact set D on the space of Z_n^(j); note that D^j is also compact.
Write k(θ, x, Z_n) = ∑_j = 1^ℓ k^o(θ, x, Z_n^(j)), where the function k^o is given by 
k^o(θ, x,  (s, y, , s') ) =  δ̅(s, s', v_θ) - 1ℓϕ(s) ϕ(s)^ x.
For each (θ, x), let k̅^o_j(θ, x) = _ζ^(j) [ k^o(θ, x, Z_0^(j)) ], 
and since the function k^o(θ, x, ·) is Lipschitz continuous in the trace variable , applying Prop. <ref>(i) to each of the ℓ state-trace processes, we have that for each j ≤ℓ and the compact set D^j, 

    lim_m, n →∞1/m∑_i=n^n+m-1( k^o(θ, x, Z_i^(j)) -  k̅^o_j(θ, x)  ) ·(Z_n^(j)∈ D^j) = 0   in mean.

Combining the relation (<ref>) for all j ≤ℓ, and using also the fact that D^j's are the projections of the set D, we obtain

    lim_m, n →∞1/m∑_i=n^n+m-1∑_j =1^ℓ( k^o(θ, x, Z_i^(j)) -  k̅^o_j(θ, x)  ) ·(Z_n ∈ D) = 0   in mean.

Since ∑_j =1^ℓ k^o(θ, x, Z_i^(j)) = k(θ, z, Z_i) and ∑_j =1^ℓk̅^o_j(θ, x) = k̅(θ, x) as we calculated earlier (cf. (<ref>)-(<ref>)), we see that (<ref>) is just the desired relation in the part (i) of the lemma except for the conditional expectation _n. The part (i) then follows from (<ref>) by Jensen's inequality.

The part (ii) follows from a similar argument, by writing g(θ, x̅(θ), Z_n) = ∑_j = 1^ℓ g(θ, x̅(θ), Z_n^(j)) and using the calculations given in (<ref>)-(<ref>). 
Similarly, for the part (iii), we write g(θ, x, Z_n) = ∑_j = 1^ℓ g(θ, x, Z_n^(j)) and apply Prop. <ref>(ii) to the function g(θ, x, ·) and each of the ℓ state-trace processes, and we then add up the results to obtain the part (iii).


We have now verified all the conditions required in the convergence proofs, so we can conclude that the convergence theorems given earlier for GTDa hold in the present context as well.
Regarding GTDb, if we employ the preceding composite scheme of setting λ, then with the “total trace” being _n = ∑_i=1^ℓ_n^(i),
the algorithm becomes 

    θ_n+1    =  Π_B_θ( θ_n + α_n [ _n δ_n(v_θ_n) -   ∑_i=1^ℓρ_n (1 - λ_n+1^(i))  _n+1ϕ(S_n+1) · (_n^(i))^ x_n  ] ), 
    
    x_n+1    = Π_B_x( x_n + β_n (_n δ_n(v_θ_n) - ϕ(S_n) ϕ(S_n)^ x_n) ).

The analysis of this algorithm follows the same line of argument given above, apart from some non-essential differences in calculations and definitions.
We now summarize the results for both GTD algorithms in the following theorem.

 
Consider the two-time-scale GTDa algorithm (<ref>)-(<ref>) or GTDb algorithm (<ref>)-(<ref>). Assume the following:



  (i) Condition <ref> holds.

  (ii) For each i ≤ℓ, the i-th trace process {_n^(i)} is generated by (<ref>) either with state-dependent λ-parameters or with history-dependent λ-parameters of the form (<ref>), where the memory states {y_n} and the function λ^(i)(·) satisfy Conditions <ref> and <ref>, respectively.

  (iii) The constraint set B_x satisfies the condition (<ref>) for the Bellman operator  defined by (<ref>).

  (iv) The initial x_0 ∈{ϕ()} and for each i ≤ℓ, the initial _0^(i)∈{ϕ()}.
 

Then for diminishing stepsizes {α_n}, {β_n} that satisfy Assumption <ref>, the conclusions of Theorem <ref> hold, and for constant stepsizes α_n = α, β_n= β with α < < β, the conclusions of Theorems <ref> and <ref> hold, where the set Θ_ opt in the theorems is now given by (<ref>) for the Bellman operator  in (<ref>).
 


 
Finally, we note that one can also apply the “robustification” idea discussed in the previous Section <ref> to derive biased variants for the GTD algorithms that employ the composite scheme of setting λ. Convergence properties of these “robustified” variants can be analyzed similarly, by combining the arguments given in the present and previous subsections.





§ CONVERGENCE ANALYSES II
 

This section continues our analyses of gradient-based TD algorithms using the weak convergence method. 
Section <ref> is about the Mirror-Descent TD algorithms, and Section <ref> is about the single-time-scale GTDa algorithm and its biased variant, which use a minimax approach to solve the minimization problem for off-policy policy evaluation. 
Section <ref> revisits the two-time-scale GTDa algorithm and its biased variant, and relaxes the conditions on their constraint set B_x by using the minimax problem formulation and related results derived in Section <ref>.
As before, we consider constrained algorithms with slowly diminishing or constant stepsize. 
For the objective function, we will treat the case where it has an additional regularizer that is smooth and convex, to demonstrate that the convergence analysis remains essentially the same. 

We will use the results of the previous sections to show that the average dynamics of the algorithms are characterized by their associated mean ODEs. Most of the efforts in our convergence analysis will be spend on the study of solution properties of these mean ODEs. In particular, for the Mirror-Descent TD algorithms, we derive a suitable form of the constraint sets to ensure desired convergence properties. For the single-time-scale GTDa algorithm, we use a general theory on differential inclusion with maximal monotone mappings to analyze the limit set of its mean ODE, and we also show that its “robustified” biased variant solves minimax problems that approximate the one solved by GTDa.

As before, convergence theorems will be stated close to where they are finally proved. For quick access to the results of this section, here is the list:




  * Mirror-Descent GTD and Mirror-Descent TD: Theorems <ref> and <ref>, respectively;

  * single-time-scale GTDa and its biased variant: Theorems <ref> and <ref>, respectively;

  * an improved result for two-time-scale GTDa and its biased variant: Theorem <ref>.




 §.§ Mirror-Descent TD Algorithms
 
In this subsection we consider the mirror-descent variant of the TD algorithms. As mentioned in the introduction, combining the mirror-descent idea <cit.> with TD learning was proposed in <cit.> (see also <cit.>). We shall call the variant algorithms MD-GTD for GTD algorithms, and MD-TD for ordinary TD algorithms. 

In what follows, let ψ^*: ^d → be a differentiable convex function such that ψ^*(θ^*) grows to +∞ at a faster than linear rate, as θ^*→ +∞. Such a function is said to be co-finite and has the property that lim_i →∞∇ψ^*(θ^*_i)   = + ∞ for every sequence {θ^*_i} such that lim_i →∞θ^*_i   = + ∞ <cit.>. 
The θ^*-space on which ψ^* is defined is dual to the θ-space we have been working on thus far.
[The notation ψ^* may look cumbersome with its superscript. We use it because dual to the convex function ψ^* is its conjugate on the θ-space, which is usually denoted by the same symbol but without the superscript ^*, although our analysis will not involve this conjugate function.]

One can think of a point θ^* being a “mirror image” of the point θ = ∇ψ^*(θ^*). 
The co-finiteness of ψ^* implies that for any θ, its pre-image under ∇ψ^* is non-empty: (∇ψ^*)^-1 (θ) ≠∅.
[Because (∇ψ^*)^-1 (θ) is precisely the nonempty set of optimal solutions to inf_θ^*{ψ^*(θ^*) - ⟨θ, θ^* ⟩}, whose objective function is not only convex and finite everywhere but also coercive due to the co-finiteness of ψ^*.]

Thus every θ∈^d is “accessible” from the θ^*-space via its mirror images.



  §.§.§ MD-GTD
 

Let J_p(θ) : = J(θ) + p(θ), where J(θ) is the projected-Bellman-error objective function as before, and p(θ): ^d → is a differentiable convex function that serves as a regularizer. In the convergence analysis given later, we will require that inf_θ J_p(θ) has a unique optimal solution, which can be ensured by a strictly convex p(·), for instance. 

Note that to minimize J_p instead of J, one can modify the GTD algorithms by simply subtracting the gradient term α_n ∇ p(θ_n) in the θ-iteration. In the case of GTDa, for example, change (<ref>) to:

    θ_n+1  = Π_D_θ( θ_n + α_n  ρ_n (ϕ(S_n) - _n+1ϕ(S_n+1) ) ·_n^ x_n   - α_n ∇ p(θ_n) )  .
    

The two-time-scale MD-GTDa algorithm we consider is given by

    x_n+1    = Π_B_x( x_n + β_n (_n δ_n(v_θ_n) - ϕ(S_n) ϕ(S_n)^ x_n) ), 
    θ_n+1^*     = Π_D_θ^*( θ_n^* + α_n  ρ_n (ϕ(S_n) - _n+1ϕ(S_n+1) )  ·_n^ x_n  - α_n ∇ p(θ_n) )  ,
    θ_n+1    = ∇ψ^*(θ_n+1^*) ,

with θ_0 = ∇ψ^*(θ^*_0) for some given initial θ^*_0.  
This algorithm calculates x_n like GTDa, at the fast time-scale determined by {β_n}. At the slow time-scale determined by {α_n}, the algorithm obtains θ_n indirectly from its “mirror image” θ_n^*, which is updated according to (<ref>), a formula that would be identical to the slow-time-scale iteration (<ref>) in GTDa if we remove all the superscripts ^* in it. The set D_θ^*⊂^d is some large enough compact set to constrain the θ^*-iterates. We assume for now that it is a large convex set with a smooth boundary; a specific choice of D_θ^* that can ensure the desired convergence will be discussed later.
Note that if {θ_n^*} is bounded, then so is {θ_n}.
[Since any convex differentiable function is continuously differentiable <cit.>, ∇ψ^* is continuous on ^d and therefore maps bounded sets to bounded sets.]


The MD-GTDb algorithm uses the same formulae (<ref>) and (<ref>) to compute x_n+1 and θ_n+1. To compute θ^*_n+1, it uses a formula that resembles the slow-time-scale iteration (<ref>) in GTDb:

    θ^*_n+1 = Π_D_θ^*( θ_n^* + α_n  (_n δ_n(v_θ_n) -  ρ_n (1 - λ_n+1)   _n+1ϕ(S_n+1) ·_n^ x_n  ) - α_n ∇ p(θ_n) ),

where θ_n = ∇ψ^*(θ_n^*).
The convergence analyses of the two MD-GTD algorithms are essentially the same. We will use MD-GTDa to show how we carry out this analysis, because it is simpler notation-wise. (One can also apply the mirror-descent idea on the x-iteration, working with the x^*-iterates in addition to the θ^*-iterates. We shall not consider these fancier algorithms here, although the line of analysis would be similar.)

Associated mean ODEs


Let us focus on the iterates {(θ^*_n, x_n)} and discuss the mean ODEs associated with them, for the MD-GTDa algorithm (<ref>)-(<ref>). Using the functions k(·), g(·) defined earlier for GTDa in (<ref>),
we write (<ref>)-(<ref>) equivalently as follows:
with θ_n = ∇ψ^*(θ_n^*),

    θ_n+1^*      = Π_D_θ^*( θ^*_n + α_n   g(θ_n, x_n, Z_n) - α_n ∇ p(θ_n) ), 
    
      x_n+1    =   Π_B_x( x_n + β_n ( k( θ_n, x_n, Z_n) + _n ω_n+1) ).
  
Consider first the fast time-scale determined by {β_n}. 
Based on the reasoning and calculation given in Sections <ref>, the desired mean ODE for this time-scale is

    ( [ θ̇^*(t);    ẋ(t) ]) 
     =  ( [                              0; k̅(∇ψ^*(θ^*(t)), x(t) ) + z(t) ]),     θ^*(0) ∈ D_θ^*,   z(t) ∈ - _B_x(x(t)),
 
where z(t) is the boundary reflection term, and the function k̅(θ, x) is as defined before in (<ref>). Note that in this ODE, the function k̅(∇ψ^*(·), ·) is continuous, since the gradient mapping ∇ψ^* is continuous (cf. Footnote <ref>).
As before, for each θ, let x_θ be the unique solution to the linear system of equations, k̅(θ, x) = 0, x ∈{ϕ()}. 
For each θ^*,  define 

    x̅(θ^*) : = x_θ   for θ = ∇ψ^*(θ^*).
 
Then, as shown in Sections <ref>, by making the radius of B_x sufficiently large, it can be ensured that for any initial x(0) ∈{ϕ()}, the solution of the ODE (<ref>) has the desired behavior that as t →∞, x(t) converges to x_θ for θ = ∇ψ^*(θ^*(0)), or equivalently, x(t) →x̅(θ^*(0)). Moreover, it can be ensured that the limit set of (<ref>) is {(θ^*, x̅(θ^*) |θ^* ∈ D_θ^*}, similar to Lemma <ref> and with the same proof.





Let us defer the precise condition on the radius of B_x and proceed to consider the desired mean ODE for the slow time-scale.
For fixed θ^*, with θ = ∇ψ^*(θ^*), we have

    _ζ[  g(θ, x̅(θ^*), Z_0) ] =  _ζ[  g(θ, x_θ, Z_0) ] =  - ∇ J (θ),

as calculated earlier in (<ref>). Thus, with g̅(θ) = - ∇ J(θ) as before, define 

    g̅_p(θ) = g̅(θ) - ∇ p(θ) = - ∇ J(θ) - ∇ p(θ) = - ∇ J_p(θ),

and we see that the desired mean ODE for the slow time-scale is given by

    θ̇^*(t) = (g̅_p ∘∇ψ^*) (θ^*(t) ) + z(t),     z(t) ∈ - _D_θ^*(θ^*(t)),

where _D_θ^*(θ^*(t)) denotes the normal cone of D_θ^* at the point θ^*(t), and z(t) is the boundary reflection term. 

Next we examine the solution properties of this ODE (<ref>). We would like it to yield solutions that lead to points θ^* that are “mirror images” of those points θ with small values of J_p(θ). Based on the idea of mirror-descent methods <cit.>, we know this is the case when there is no constraint set D_θ^*. So we will aim at finding a suitable choice of the constraint set D_θ^*, so that it does not “ruin” the good property of mirror-descent.

Choice of the constraint set and the solution property of the corresponding ODE


To study the solution property of the ODE (<ref>), let us pick an optimal solution θ_opt of the problem inf_θ J(θ), and consider a candidate Lyapunov function for the ODE, which is from the original idea behind mirror descent methods <cit.>:

    V(θ^*) = ψ^*(θ^*) - ⟨θ^*, θ_opt⟩.
   
The co-finiteness of ψ^* ensures that V is bounded below on ^d.
Let θ^*(·) be a solution of (<ref>), and let θ(t) = ∇ψ^*(θ^*(t)). 
We have

    V̇(θ^*(t))     = ⟨θ̇^*(t),  ∇ψ^*(θ^*(t)) - θ_opt⟩
        = ⟨ - ∇ J_p( θ(t)) + z(t) ,  θ(t) - θ_opt⟩
       ≤ J_p( θ_opt) - J_p(θ(t))  + ⟨  z(t)  ,  θ(t) - θ_opt⟩,

where the last inequality follows from the inequality J_p( θ_opt) ≥ J_p(θ(t)) + ⟨∇ J_p ( θ(t) ),  θ_opt - θ(t)  ⟩ implied by the convexity of the function J_p. 
Thus, when the boundary reflection term z(t) = 0, as is the case when θ^*(t) is in the interior of D_θ^*, 
we have the desired relation that V̇(θ^*(t)) < 0 if θ(t) is not an optimal solution of inf_θ J_p(θ). 
We would like this relation to hold also when θ^*(t) is on the boundary of D_θ^*, so that V can serve as a Lyapunov function. 
This leads us to make a simple but natural choice of the constraint set D_θ^* as follows. 


Let D_θ^* be a level set of ψ^*,
D_θ^* = {θ^* |ψ^*(θ^*) ≤ℓ}, for some finite number ℓ > inf_θ^*ψ^*(θ^*). 
Then the normal cone of D_θ^* at a boundary point θ^* is just _D_θ^*(θ^*) = { a ∇ψ^*(θ^*) | a ≥ 0 }. If θ^*(t) is on the boundary of D_θ^*, the boundary reflection term z(t) = - a ∇ψ^*(θ^*(t)) = - a  θ(t) for some a ≥ 0, and the second term in (<ref>) becomes

    - a ⟨θ(t) ,  θ(t) - θ_opt⟩≤ - a  θ(t) _2 ·( θ(t) _2 - θ_opt_2 ).

The upperbound on the r.h.s. is nonpositive if we make D_θ^* sufficiently large so that all its boundary points θ^* satisfy ∇ψ^*(θ^*) _2 ≥θ_opt_2 (which is possible since ∇ψ^*(θ^*) _2 → + ∞ as θ^* _2 → + ∞; cf. the co-finiteness property of ψ^* mentioned at the beginning). Then we will have the desired relation V̇(θ^*(t)) ≤ J_p( θ_opt) - J_p(θ(t)) at all points θ^*(t).

We can now state precisely our assumptions on the constraint sets D_θ^* and B_x.
Define

    Θ̃_opt = _θ∈^d J_p(θ).
   

[Conditions on the constraint sets of MD-GTD algorithms]  



  (i) The constraint set D_θ^* = {θ^* |ψ^*(θ^*) ≤ℓ}, where ℓ is large enough so that for some optimal solution θ_ opt∈Θ̃_ opt, ∇ψ^*(θ^*) _2 ≥θ_ opt_2 for all points θ^* on the boundary of D_θ^*.

  (ii) With D_θ = {∇ψ^*(θ^*) |θ^* ∈ D_θ^*}, the constraint set B_x satisfies
B_x ⊃{ x_θ|θ∈ D_θ}.
[As before, a sufficient condition is that radius(B_x) ≥sup_θ∈ D_θΦ^Ξ  ( v_θ - v_θ) _2 /c, where c > 0 is the smallest nonzero eigenvalue of the matrix Φ^Ξ Φ.]




Note that in the above, θ_opt∈ D_θ always (hence D_θ∩Θ̃_ opt≠∅; i.e., D_θ^* includes at least one “mirror image” of an optimal θ).
This is a generic property of the function ψ^*, as the next lemma shows.

 
Given θ̅, let D_θ^* = {θ^* |ψ^*(θ^*) ≤ℓ} where ℓ is such that ∇ψ^*(θ^*) _2 ≥θ̅_2 for all points θ^* on the boundary of D_θ^*. Then there exists at least one point θ̅^* ∈ D_θ^* with ∇ψ^*(θ̅^*) = θ̅. In particular, in Assumption <ref>, θ_ opt∈ D_θ.


As discussed at the beginning of the section, the co-finiteness of ψ^* implies that the set of points {θ̅^* |∇ψ^*(θ̅^*) = θ̅} is nonempty. If all these points are in D_θ^*, we are done. Otherwise, some point θ̅^* from this set lies outside D_θ^*. Let us find a point θ^* on the boundary of D_θ^* with ∇ψ^*(θ^*) = θ̅, thereby proving the lemma. Consider the boundary point θ^* = Π_D_θ^*θ̅^*. 
Then θ̅^* - θ^* ∈_D_θ^*(θ^*), so θ̅^* - θ^* = a ∇ψ^*(θ^*) for some a > 0.  
Let f(t) = ψ^*(θ̅^* + t (θ^* - θ̅^*)) for t ∈.
We have 

    f'(0)  = ⟨∇ψ^*(θ̅^*),  θ^* - θ̅^* ⟩ = - a  ⟨θ̅,  ∇ψ^*(θ^*) ⟩,
       f'(1)  = ⟨∇ψ^*(θ^*), θ^* - θ̅^* ⟩= - a  ∇ψ^*(θ^*) _2^2.

Since f is convex, f'(0) ≤ f'(1) by <cit.> and hence 
⟨θ̅,  ∇ψ^*(θ^*) ⟩≥∇ψ^*(θ^*) _2^2. But ∇ψ^*(θ^*) _2 ≥θ̅_2 since θ^* is on the boundary of D_θ^*. These two inequalities, together with the Cauchy-Schwarz inequality, imply ∇ψ^*(θ^*)  = θ̅. This completes the proof.


The next lemma is about the behavior of the solutions to the ODE (<ref>) when the constraint set D_θ^* satisfies Assumption <ref>(i). We will use its second part on the limit set of the ODE to obtain a convergence theorem for the MD-GTD algorithms. 


 
Under Assumption <ref>(i) on the constraint set D_θ^*, every solution θ^*(·) of the ODE (<ref>) satisfies that 
θ^*(t) converges to the compact set ( ∇ψ^*)^-1 (Θ̃_ opt) ∩ D_θ^* as t →∞. 
If, in addition, Θ̃_ opt = {θ_ opt} (i.e., inf_θ J_p(θ) has a unique optimal solution), then
the limit set of the ODE (<ref>) is ( ∇ψ^*)^-1 (θ_ opt) ∩ D_θ^*.



Consider the function V(·) defined in (<ref>) for the point θ_opt in Assumption <ref>(i). 
Let θ^*(·) be a solution of (<ref>), and let θ(t) = ∇ψ^*(θ^*(t)) for t ≥ 0. As discussed earlier, by (<ref>) and Assumption <ref>(i) on D_θ^*,

    V̇(θ^*(t)) ≤ J_p( θ_opt) - J_p(θ(t)) ≤ 0.
   
From this it follows that as t →∞, θ^*(t) converges to the compact set
[To see this, let θ̅^* ∈ D_θ^* be any point such that for some sequence of times t_n →∞, θ^*(t_n) converges to θ̅^* as n →∞. 
If J_p(∇ψ^*( θ̅^*)) > J_p( θ_opt), then since J_p ∘∇ψ^* is a continuous function, there exist sufficiently small ϵ,  δ > 0 such that J_p(∇ψ^*(θ^*)) > J_p( θ_opt) + ϵ for all θ^* ∈ N_δ(θ̅^*) (the δ-neighborhood of θ̅^*). Then, since V is bounded below on D_θ^*, by (<ref>), the total amount of time the solution θ^*(·) spends in N_δ(θ̅^*) must be finite. But the solution θ^*(·) is Lipschitz continuous (because by Lemma <ref>, g̅_p(∇ψ^*(θ^*(t))) + z(t) is the projection of g̅_p(∇ψ^*(θ^*(t))) on the tangent cone of D_θ^* at θ^*(t), and therefore, for all t, g̅_p(∇ψ^*(θ^*(t))) + z(t)_2 ≤g̅_p(∇ψ^*(θ^*(t)))_2 ≤sup_θ^* ∈ D_θ^*g̅_p(∇ψ^*(θ^*)) _2 < ∞). 
So, whenever its path enters inside the smaller neighborhood N_δ/2(θ̅^*), there is some fixed amount of time, say Δ, that it has to spend inside N_δ(θ̅^*) before it can exit N_δ(θ̅^*). This means it will not visit N_δ(θ̅^*) any more after some finite time t, a contradiction to θ^*(t_n) →θ̅^* with t_n →∞. Therefore, we must have J_p(∇ψ^*( θ̅^*)) = J_p( θ_opt).
]


    {θ^* ∈ D_θ^*|  J_p( θ_opt) - J_p(∇ψ^*(θ^*)) = 0 }   = ( ∇ψ^*)^-1 (Θ̃_ opt) ∩ D_θ^*.


Now consider the limit set of (<ref>) in the case Θ̃_ opt = {θ_ opt}. 
Clearly, it contains the set ( ∇ψ^*)^-1 (θ_ opt) ∩ D_θ^*, since for any θ̅^* in the latter set, g̅_p(∇ψ^*(θ̅^*)) = - ∇ J_p(θ_ opt) = 0 by the optimality condition and hence θ^*(·) ≡θ̅^* is a solution of (<ref>). 
To show the other direction of inclusion, 
define level sets D_η, E_ϵ for η, ϵ  > 0 by

    D_η : = {θ^* ∈  D_θ^*| V(θ^*) ≤inf_θ̃^* V(θ̃^*) + η},    E_ϵ : = {θ^* ∈  D_θ^*| J_p(∇ψ^*(θ^*)) ≤ J_p( θ_opt) + ϵ}.

Note that (∇ψ^*)^-1 (θ_ opt) is the set of optimal solutions to inf_θ^* V(θ^*), and it is nonempty, convex and compact. 
The sets D_η are also convex and compact, with ∩_η > 0 D_η = (∇ψ^*)^-1 (θ_ opt) ∩ D_θ^*.
Since the function J_p ∘∇ψ^* is continuous, the sets E_ϵ are also compact and moreover, by the uniqueness of θ_opt, ∩_ϵ > 0 E_ϵ = (∇ψ^*)^-1 (θ_ opt) ∩ D_θ^*.

For each η, there must exist ϵ sufficiently small such that E_ϵ⊂ D_η. 
Otherwise, we could find a sequence of points θ^*_n ∈ E_ϵ_n with ϵ_n → 0, but with V(θ^*_n) - inf_θ^* V(θ^*) > η for all n. Then any limit point θ^*_∞ of this bounded sequence {θ^*_n} must lie in ∩_ϵ > 0 E_ϵ and hence in (∇ψ^*)^-1 (θ_ opt), yet it must also satisfy V(θ^*_∞) - inf_θ^* V(θ^*) ≥η, which is impossible.

For any given η, consider this set E_ϵ⊂ D_η.
Since V is bounded on D_θ^*, (<ref>) implies that there exists a finite time t_ϵ such that every solution of (<ref>) must visit the set E_ϵ at least once before time t_ϵ;
i.e., there is a time t_0 ≤ t_ϵ with θ^*(t_0) ∈ E_ϵ. Since V(θ^*(t)) is nonincreasing and E_ϵ⊂ D_η, 
this means that θ^*(t) ∈ D_η for all t ≥ t_ϵ and all solutions θ^*(·) of (<ref>). 
Consequently, the limit set of (<ref>) must satisfy
⋂_ t̅≥ 0  {θ^*(t)   |  θ^*(0) ∈ D_θ^*,    t ≥t̅ }⊂∩_η > 0 D_η = (∇ψ^*)^-1 (θ_ opt) ∩ D_θ^*.
This prove the other direction of inclusion and completes the proof.




Convergence results



We now use the preceding results about the limit sets of the two ODEs to characterize the asymptotic behavior of MD-GTD. Besides Assumption <ref> on the constraint sets D_θ^* and B_x, we shall also assume, as in the second part of Lemma <ref>, that inf_θ J_p(θ) has a unique optimal solution θ_ opt. 
It is straightforward to verify that the iterates {(θ^*_n, x_n)} generated by MD-GTD satisfy all the conditions for applying the convergence proofs given earlier for GTDa.
[Although the conditions are now on the (θ^*, x)-space instead of the (θ, x)-space, they are essentially the same as before. For example, the “averaging condition” (iv) listed in Section <ref> is now a convergence-in-mean condition on {k(∇ψ^*(θ^*), x, Z_n)} for each (θ^*, x). This is the same as the “averaging condition” on {k(θ, x, Z_n)} for each θ = ∇ψ^*(θ^*) and each x, which we already treated in Lemma <ref>. As another example, in the case here, we need that for (θ^*, x) ∈ D_θ^*× B_x and z in a compact set, the function g(∇ψ^*(θ^*), x, z) - ∇ p(∇ψ^*(θ^*)) is Lipschitz continuous in x uniformly w.r.t. (θ^*, z) (cf. Remark <ref>). This is the same as the function g(θ, x, z) being Lipschitz continuous in x uniformly w.r.t. (θ, z) in a compact set, the same property that we verified earlier in the GTDa case.]

Thus the conclusions of Theorems <ref>-<ref> hold for {(θ^*_n, x_n)}. In particular, about the asymptotic behavior of {θ_n^*}, the conclusions of those theorems hold with Θ_ opt replaced by ( ∇ψ^*)^-1 (θ_ opt) ∩ D_θ^*, the limit set of the ODE (<ref>). We can then translate these results about the asymptotic behavior of {θ_n^*} to that of {θ_n}, by using the following observation: Since the gradient mapping ∇ψ^* is continuous and the set D_θ^* is bounded, for any ϵ > 0, there exists η_ϵ > 0 (with η_ϵ→ 0 as ϵ→ 0), such that 

    θ^* ∈ N_η_ϵ(( ∇ψ^*)^-1 (θ_ opt) ∩ D_θ^*)   ⟹  θ = ∇ψ^*(θ^*) ∈  N_ϵ(θ_ opt).

This gives us the convergence results for MD-GTD with both diminishing and constant stepsizes stated in the following convergence theorem. 

The theorem also includes conclusions about the averaged θ-iterates in the case of constant stepsize. The line of proof for this part is the same as that explained after Theorem <ref> in Section <ref>. It uses the observation that with a given pair of constant stepsizes (α, β), the MD-GTD iterates together with the states and traces, (S_n, y_n, _n, θ^*_n, θ_n, x_n), jointly form a weak Feller Markov chain (this proof is similar to the proof of <cit.>). The proof then combines this observation with those conclusions about the asymptotic behavior of the θ-iterates, and with ergodicity properties of weak Feller Markov chains. As before, we omit the proof, since it is essentially the same proof given in <cit.>. 




 
Consider the MD-GTDa algorithm (<ref>)-(<ref>) or the MD-GTDb algorithm given by (<ref>), (<ref>), and (<ref>),  with the initial x_0, _0 ∈{ϕ()}. Let Assumptions <ref> and <ref> hold, and let θ_ opt be the unique optimal solution to inf_θ J_p(θ). 
Then for diminishing stepsizes {α_n}, {β_n} that satisfy Assumption <ref>, the conclusions of Theorem <ref> about the θ-iterates hold with θ_ opt in place of Θ_ opt. For the case of constant stepsize, the conclusions of both Theorem <ref> (about the θ-iterates) and Theorem <ref> (about the averaged θ-iterates) hold with θ_ opt in place of Θ_ opt.



 
For the convergence of GTD algorithms, we do not need any condition on the minimization problem inf_θ∈ B_θ J_p(θ); in particular, we do not need B_θ to contain an optimal solution of the unconstrained minimization problem inf_θ J_p(θ). The condition in the above theorem for MD-GTD is more stringent, as it requires not only that D_θ contains an optimal solution of the unconstrained problem inf_θ J_p(θ), but also that inf_θ J_p(θ) has a unique optimal solution. This condition is used in Lemma <ref> to ensure that the limit set of the ODE (<ref>) has desirable properties—if the latter can be obtained under a weaker condition on the minimization problem, the condition in the theorem can be correspondingly weakened.


 
We mention that in the case of state-dependent λ, one can also consider biased variants of the MD-GTD algorithms. Such variants can be analyzed by combining the preceding analysis with the reasoning given in Section <ref>. 


 
When the λ's are set according to the composite scheme described in Section <ref>, the changes in the MD-GTD algorithms are similar to those in the GTD algorithms described earlier. In particular, with _n = ∑_i=1^ℓ_n^(i), where _n^(i) is updated according to (<ref>) for each i ≤ℓ, MD-GTDa is still given by (<ref>)-(<ref>), and MD-GTDb becomes

    x_n+1    = Π_B_x( x_n + β_n (_n δ_n(v_θ_n) - ϕ(S_n) ϕ(S_n)^ x_n) ),  
    θ^*_n+1    = Π_D_θ^*( θ_n^* + α_n  (_n δ_n(v_θ_n) -   ∑_i=1^ℓρ_n (1 - λ^(i)_n+1)   _n+1ϕ(S_n+1) · (_n^(i))^  x_n ) - α_n ∇ p(θ_n) ), 
    θ_n+1    = ∇ψ^*(θ_n+1^*).
    
By combining the preceding analysis with the reasoning given in Section <ref>, it is straightforward to show that the conclusions of Theorem <ref> hold for these MD-GTD algorithms as well, where the Bellman operator  in J(θ) and in Assumption <ref> is now given by (<ref>).




  §.§.§ MD-TD


In the rest of this subsection, we discuss an algorithm that combines the mirror-descent method with the original TD method. This MD-TD algorithm is given by

    θ_n+1^*  = Π_D_θ^*( θ_n^* + α_n  _n  δ_n(v_θ_n)  ),      θ_n+1  = ∇ψ^*(θ_n+1^*),

with θ_0 = ∇ψ^*(θ_0^*) for some given initial θ_0^*.
It is a single-time-scale algorithm like TD, and computes the θ^*-iterates using a formula that resembles the TD algorithm.

We now derive the mean ODE associated with the θ^*-iterates, and we will introduce a few conditions to ensure desired solution properties. 
By Prop. <ref>, for each fixed θ,

    _ζ[ _0 δ̅_0(v_θ) ] = Φ^Ξ  ( v_θ - v_θ) = : g̅(θ).
   
So the projected mean ODE associated with (<ref>) is

    θ̇^*(t) = (g̅∘∇ψ^*) (θ^*(t) ) + z(t),     z(t) ∈ - _D_θ^*(θ^*(t)),

where z(t) is the boundary reflection term.
We shall impose a strong condition on g̅, in order to ensure desired convergence:

  
The linear equation g̅(θ) = Φ^Ξ  ((Φθ)  - Φθ) = 0 has a unique solution θ_ TD, and the matrix C=Φ^Ξ  ( - I) Φ is negative definite (i.e., for some c > 0, θ^ C θ≤ - c θ_2^2 for all θ∈^d).


Suppose Assumption <ref> holds. In addition, suppose the constraint set D_θ^* is a level set of ψ^* that satisfies Assumption <ref>(i) with θ_TD in place of θ_opt. Note that by Lemma <ref>, θ_TD∈ D_θ : = {∇ψ^*(θ^*) |θ^* ∈ D_θ^*} (i.e., in D_θ^* there is at least one “mirror image” of θ_TD, through which the algorithm can “access” θ_TD).

As a candidate Lyapunov function, consider the function

    V (θ^*) = ψ^*(θ^*) - ⟨θ^*, θ_TD⟩.

Let θ^*(·) be a solution of (<ref>), and let θ(t) = ∇ψ^*(θ^*(t)). 
Note that since g̅(θ_TD) = 0, g̅(θ) = C (θ - θ_TD).
Then similar to (<ref>) in the previous MD-GTD case, 
we have

    V̇(θ^*(t))     = ⟨g̅( θ(t))  + z(t),  ∇ψ^*(θ^*(t)) - θ_TD⟩
        =  ⟨ C (θ(t) - θ_TD),  θ(t) - θ_TD⟩  + ⟨  z(t)  ,  θ(t) - θ_TD⟩
       ≤⟨ C (θ(t) - θ_TD),  θ(t) - θ_TD⟩
       ≤ - c  θ(t) - θ_TD_2^2,

for some constant c > 0.
In the above, the second equality uses the fact g̅(θ) = C (θ - θ_TD). The second to last inequality uses the fact that our choice of the constraint set D_θ^* ensures that the term
⟨  z(t)  ,  θ(t) - θ_TD⟩≤ 0 if θ^*(t) is on the boundary of D_θ^* (cf. the discussion before Assumption <ref>(i) in the previous case), so this term is always nonpositive. The last inequality follows from the negative definiteness of the matrix C under Assumption <ref>. Thus V̇(θ^*(t)) < 0 for all θ(t) ≠θ_TD as desired.

Furthermore, similar to Lemma <ref>, we have that the limit set of the ODE (<ref>) is
[The limit set contains (∇ψ^*)^-1( θ_TD) ∩ D_θ^* since each point in the latter set corresponds to a constant solution to the ODE. The other direction of inclusion follows from the last part of the proof of Lemma <ref>, with θ_TD in place of θ_opt and with the function θ↦ - c  θ - θ_TD_2^2 serving as the function J_p in the proof.]


    ⋂_ t̅≥ 0  {θ^*(t)   |  θ^*(0) ∈ D_θ^*,    t ≥t̅ } = (∇ψ^*)^-1( θ_TD) ∩ D_θ^* .

As θ^*(t) approaches this limit set, the corresponding θ(t)=∇ψ^*(θ^*(t)) converges to θ_ TD, since for any ϵ > 0, there exists η_ϵ > 0 such that 

    θ^* ∈ N_η_ϵ(( ∇ψ^*)^-1 (θ_ TD) ∩ D_θ^*)   ⟹  θ = ∇ψ^*(θ^*) ∈  N_ϵ(θ_ TD)

(cf. the discussion preceding (<ref>)).
Likewise, if the iterates θ_n^* of the MD-TD algorithm approach the above limit set, the corresponding iterates θ_n = ∇ψ^*(θ_n^*) approach the point θ_TD. 

We now proceed to the convergence proof for MD-TD. We first establish the connection between the average dynamics of the algorithm and the ODE (<ref>) using stochastic approximation theory, which will allow us to translate the solution property of the mean ODE (<ref>) derived above to the asymptotic behavior of the θ^*-iterates. We then use (<ref>) to obtain convergence properties for the θ-iterates. For the first step of this proof, we apply <cit.> in the case of diminishing stepsize and <cit.> in the case of constant stepsize.  The conditions of these theorems are similar to those listed in Section <ref>, and it is straightforward to show that they are satisfied.
[We omit the details here, because in the subsequent Section <ref>, for another single-time-scale algorithm (GTDa), we will describe similar conditions and explain how to verify them. In particular, see Footnote <ref> therein.]

This gives us the convergence results about the θ-iterates generated by MD-TD stated in the following convergence theorem. Those conclusions in the theorem about the averaged θ-iterates, in the constant-stepsize case, are then obtained through a procedure similar to that described before Theorem <ref> in the previous subsection for MD-GTD (see also the explanation of the proof of Theorem <ref> in Section <ref>).



 
Consider the MD-TD algorithm (<ref>). Let Assumptions <ref> and <ref> hold, and let D_θ^* satisfy Assumption <ref>(i) with θ_ TD in place of θ_ opt.
Then, in the case of diminishing stepsize, with the stepsizes {α_n} satisfying those conditions given in Assumption <ref> for the fast-time-scale stepsizes, the conclusions of Theorem <ref> about the θ-iterates hold with θ_ TD in place of Θ_ opt. 
In the case of constant stepsize, the conclusions of both Theorem <ref> (about the θ-iterates) and Theorem <ref> (about the averaged θ-iterates) hold with θ_ TD in place of Θ_ opt, and with obvious modifications to remove all the references to the fast-time-scale stepsize parameter β.
 





 §.§ A Minimax Approach: Single-Time-Scale GTDa
 

In this subsection we analyze a constrained single-time-scale GTDa algorithm and its biased variant.
As in Section <ref>, we shall consider the regularized objective function J_p(θ) = J(θ) + p(θ), where p(·) is a differentiable convex function (for the reason that the analysis is the same with or without this regularizer p(·)). The goal of the single-times-scale GTDa algorithm is, as before, to solve the minimization problem inf_θ∈ B_θ J_p(θ), where B_θ is the constraint set. It computes {(θ_n, x_n)} according to

    θ_n+1    = Π_B_θ( θ_n + α_n  ρ_n (ϕ(S_n) - _n+1ϕ(S_n+1) ) ·_n^ x_n  - α_n ∇ p(θ_n) ),  
    
       x_n+1    = Π_B_x( x_n + α_n (_n δ_n(v_θ_n) - ϕ(S_n) ϕ(S_n)^ x_n) ),

using the same stepsize sequence {α_n} in both iterations. Although the algorithm looks almost identical to its two-time-scale counterpart, it is based on a different, minimax approach to minimize J_p(θ), as first pointed out in <cit.>, and its behavior can thus also be quite different. 

To analyze the asymptotic behavior of the algorithm, we will focus on the solution properties of its associated mean ODE. 
Indeed, it is straightforward to apply stochastic approximation theory to this single-time-scale algorithm and prove that its average dynamics is characterized by the mean ODE, by using the analyses given in Sections <ref> and <ref>. (The details of this proof step will be explained in Section <ref> after we state the convergence theorem.) So the solution properties of the mean ODE will show us how the algorithm behaves and how the constraint sets can affect it.

The mean ODE associated with the algorithm (<ref>)-(<ref>) is

    ( [ θ̇(t);  ẋ(t) ]) 
     =  ( [ g̅(θ(t), x(t)) - ∇ p(θ(t));            k̅(θ(t), x(t) ) ]) +  ( [ z_1(t); z_2(t) ]),         z_1(t) ∈ - _B_θ(θ(t)),   z_2(t) ∈ - _B_x(x(t)),

where z_1(t), z_2(t) are the boundary reflection terms, and the functions g̅, k̅ are given by

    g̅(θ, x) = ( Φ^Ξ  (I - ) Φ)^ x,     ̨̅(θ, x) = Φ^Ξ  ( v_θ - v_θ) - Φ^Ξ Φ  x.

(The derivations of the above are the same as those given at the beginning of Section <ref> for GTDa.) 
In order to study its solutions, we will need to first clarify its relation with a minimax approach to solving the convex optimization problem inf_θ∈ B_θ J_p(θ). 




  §.§.§ Solution properties of the associated mean ODE
 

Let us start by converting inf_θ∈ B_θ{ J(θ) + p(θ)} to a minimax problem.
Notice the identity relation that for all v ∈^||, 12 v _ξ^2  = sup_y {⟨ y, v ⟩_ξ - 12 y ^2_ξ}, where
the maximization can be over the approximation subspace Ł_ϕ only, if v ∈Ł_ϕ:

    12 v ^2_ξ    = sup_ y ∈Ł_ϕ{⟨ y, v ⟩_ξ - 12 y ^2_ξ } 
     =  sup_ x ∈^d{⟨Φ x, v ⟩_ξ - 12Φ x ^2_ξ } 
         =  sup_ x ∈{ϕ()}{⟨Φ x, v ⟩_ξ - 12Φ x ^2_ξ }.
 
We have J(θ) = 12 v _ξ^2 for v = Π_ξ ( v_θ - v_θ) ∈Ł_ϕ. Then, using also the fact that ⟨Φ x, Π_ξ v ⟩_ξ = ⟨Φ x, v ⟩_ξ for any v ∈^||, we can write inf_θ∈ B_θ{ J(θ) + p(θ)} equivalently as 

    inf_θ∈ B_θsup_x ∈^d{⟨Φ x,  v_θ - v_θ⟩_ξ - 12Φ x ^2_ξ  + p(θ) }.
  
Let us denote the function above by ψ^o, to simplify notation:

    ψ^o(θ, x) : = ⟨Φ x,  v_θ - v_θ⟩_ξ - 12Φ x ^2_ξ  + p(θ).
 
This function ψ^o is a continuous convex-concave function: convex in θ for each x and concave in x for each θ. 
Since B_θ is compact, by <cit.>, the minimax problem (<ref>) has a saddle-value.
Note also that due to the term - 12Φ x^2_ξ, the function inf_θ∈ B_θψ^o(θ, ·) (which is everywhere finite) is strictly concave and coercive on the subspace {ϕ()}, whereas it is constant over {ϕ()}^⊥. This implies that in {ϕ() } the minimax problem (<ref>) has a unique dual optimal solution,

    x_ opt =  _x ∈{ϕ()}{inf_θ∈ B_θψ^o(θ, x) }.

Then by <cit.>, the set Θ_ opt×{ x_ opt} is a subset of saddle points of the problem (<ref>), where   

    Θ_ opt = _θ∈ B_θ{ J(θ) + p(θ) } = _θ∈ B_θ{sup_x ∈^dψ^o(θ, x) }.


Instead of (<ref>), below we shall focus on the minimax problem

    inf_θ∈ B_θsup_x ∈ B_xψ^o(θ, x),
  
where we have added the compact constraint set B_x for the maximizer, the same constraint set in the algorithm and its mean ODE. 
Its mean ODE (<ref>) is related to the minimax problem (<ref>), as we will show below.

By <cit.>, the minimax problem (<ref>) has a nonempty set of saddle points, which we denote by D_θ× D_x. The set D_θ is just the compact set of optimal solutions to the minimization problem inf_θ∈ B_θ{sup_x ∈ B_xψ^o(θ, x) }, whereas the set D_x is the compact set of optimal solutions to the dual maximization problem: sup_x ∈ B_x{inf_θ∈ B_θψ^o(θ, x) }. 
The minimax problem (<ref>) has a unique dual optimal solution x̅ in {ϕ() } (in other words, D_x ∩{ϕ() } = {x̅}). This follows from the strict concavity and coercivity of the function inf_θ∈ B_θψ^o(θ, ·) on {ϕ()} and its constancy over {ϕ()}^⊥ mentioned above. 
These saddle points of (<ref>) can be related to the saddle points of the original minimax problem (<ref>) as follows.


 
If x_ opt∈ int(B_x) (the interior of B_x), then D_θ×{x̅} =Θ_ opt×{ x_ opt}.
 
 
The proof of this lemma is short, but we will give it at the end of this subsection, where we will have all the definitions needed in the proof. The main effort of our analysis is to prove that D_θ×{x̅} is the limit set of the ODE (<ref>), as stated in the following proposition. We will spend the rest of this subsection to prove it.

 
For initial conditions x(0) ∈{ϕ()}, the limit set of the ODE (<ref>) is the subset of saddle points, D_θ×{x̅}, of the minimax problem (<ref>), where x̅ is the unique dual optimal solution of (<ref>) in {ϕ()}. 



To setup the stage to prove Prop. <ref>, we shall need some properties of the ODE (<ref>) as well as some basic properties of the minimax problem (<ref>). 
First, to apply the theory of convex analysis to (<ref>), it is more convenient to represent the constraint sets by functions and rewrite (<ref>) equivalently as 

    inf_θ∈^dsup_x ∈^dψ(θ, x)
 
for an extended-real-valued convex-concave function ψ(θ, x) given by

    ψ(θ, x)      := ψ^o(θ, x) + δ_B_θ(θ) - δ_B_x(x) 
          = ⟨Φ x,  v_θ - v_θ⟩_ξ - 12Φ x ^2_ξ  + p(θ)+ δ_B_θ(θ) - δ_B_x(x),

where for a set D, δ_D(·) is the indicator function that takes the value 0 on D and +∞ outside D.
[Outside B_θ× B_x, ψ takes either the value +∞ or -∞, and there are two consistent ways to assign these values to ψ there; see <cit.>. Either way can be taken here as the convention to handle the difference δ_B_θ(θ) - δ_B_x(x) when both terms are +∞. Such choices do not affect the subsequent analysis because the effective domain of ψ is B_θ× B_x (see <cit.> on equivalent saddle-functions).]

The set of subgradients of ψ at (θ, x) is 

    ∂ψ(θ, x) = ∂_θψ(θ, x) ×∂_x ψ(θ, x)
 
where, expressed in terms of the function ψ^o or the functions g̅, k̅ in (<ref>),

    ∂_θψ(θ, x)     = ∇_θψ^o(θ, x) + _B_θ(θ) = - g̅(θ, z) + ∇ p(θ) + _B_θ(θ),  
    ∂_x ψ(θ, x)     = ∇_x ψ^o(θ, x) - _B_x(x) = k̅(θ, x) - _B_x(x).
    
(If either θ∉B_θ or x ∉B_x, ∂ψ(θ, x) = ∅ by definition.) 
Note that (θ, x) is a saddle-point of (<ref>) if and only if (0,0) ∈∂ψ(θ, x). 
We shall also need  the set-valued mapping  from ^d ×^d to ^d ×^d defined by 

    (θ, x) : = ∂_θψ(θ, x) ×( - ∂_x ψ(θ, x) ).
 
By <cit.>, (·) is a maximal monotone mapping.
[A set-value mapping G from ^m to ^m is monotone if ⟨ x - x', y - y' ⟩≥ 0 for every y ∈ G(x) and y' ∈ G(x'). 
It is maximal monotone if for every (x', y') that satisfies ⟨ x - x', y - y' ⟩≥ 0 for all (x,y) ∈gph  G, we have (x', y') ∈gph  G, where gph  G = {(x, y) | y ∈ G(x), x ∈^m } is the graph of G.]

We will soon use a general result on differential inclusions with such mappings to help us in analyzing the ODE (<ref>). 
To this end, let us first relate the mapping  to the ODE.

Comparing the r.h.s. of the ODE (<ref>) with the expressions (<ref>)-(<ref>) of the subgradients, we see that if (θ(t), x(t)) = (θ, x), then

    g̅(θ, x) - ∇ p(θ) + z_1  ∈ - ∂_θψ(θ, x),     k̅(θ, x) + z_2  ∈∂_x ψ(θ, x),

where z_1, z_2 are the boundary reflection terms in (<ref>). Moreover, using Lemma <ref>, we can relate the above two vectors in the r.h.s. of the ODE (<ref>) to the set (θ,x) as follows:

 
Given (θ, x) ∈ B_θ× B_x, let z_1 = - Π_D_1 ( g̅(θ, x) - ∇ p(θ) ) for D_1 = _B_θ(θ), and let z_2 = - Π_D_2k̅(θ, x) for D_2 = _B_x(x). 
Then 

    (g̅(θ, x) - ∇ p(θ) + z_1,  k̅(θ, x) + z_2) ∈ - (θ, x)
 
and it is the vector in - (θ, x) that has the minimal ·_2-norm.



Applying Lemma <ref> with D=B_θ, h = g̅(θ,x) - ∇ p(θ) and y=-z_1, and using also the expression (<ref>), we see that g̅(θ, x) - ∇ p(θ) + z_1 is the minimal-norm vector in the set - ∂_θψ(θ, x). Then applying Lemma <ref> with D=B_x, θ = x, h = k̅(θ,x) and y=-z_2, and using also the expression (<ref>), we see that k̅(θ, x) + z_2 is the minimal-norm vector in the set ∂_x ψ(θ, x). It then follows that (g̅(θ, x) - ∇ p(θ) + z_1,  k̅(θ, x) + z_2) is the minimal-norm vector in - (θ, x). 


We are now ready to study the limit set of the ODE (<ref>). For this part of the proof, we use y(·) to denote a solution (θ(·), x(·)) of the ODE, in order to simplify notation. 

We shall use a very general result from <cit.> concerning solutions to differential inclusions with maximal monotone mappings. 
By <cit.>, for a maximal monotone mapping , the differential inclusion ẏ(t) ∈ - (y(t)) has a unique absolutely continuous solution for any initial condition in the domain of , and this solution has the following properties:



  (i) For almost all t, ẏ(t) is the minimal-norm (w.r.t. the Euclidean norm ·_2) vector in - (y(t)).

  (ii) The norm of the minimal-norm vector of (y(t)) is nonincreasing as t increases.

  (iii) If y_1(·), y_2(·) are two solutions corresponding to two different initial conditions, then

    y_1(t) -  y_2(t)  _2 ≤ y_1(0) - y_2(0) _2,     ∀  t ≥ 0.



For the maximal monotone mapping  in our case, we see that the solutions to the differential inclusion with  are just the solutions to the mean ODE (<ref>), so the solutions y(·) of the ODE have the properties listed above. We now use these properties (especially, the third one) to study the limit set of the ODE (<ref>). By definition this is the set ∩_τ≥ 0 { y(t) | y(0) ∈ B_θ× B_x, t ≥τ} as we recall.
For any set D ∈ B_θ× B_x and t ≥ 0, let F^t(D) = { y(t) | y(0) ∈ D }, i.e., the union of y(t) for all initial conditions in D. We say D is invariant (w.r.t. the ODE) if F^t(D) = D for all t ≥ 0.

 
The limit set of the ODE (<ref>) is E : = ∩_t ≥ 0 F^t(B_θ× B_x) (a nonempty compact set), and it is the largest invariant subset of B_θ× B_x.



If t_1 ≤ t_2, F^t_2(B_θ× B_x) ⊂ F^t_1(B_θ× B_x). By the property (iii) above, for a compact set D, F^t(D) is closed for each t. Hence for any τ≥ 0, 

    { y(t) | y(0) ∈ B_θ× B_x, t ≥τ} = ( ∪_t ≥τF^t(B_θ× B_x) ) = F^τ(B_θ× B_x),
 
so the limit set of the ODE is ∩_τ≥ 0 F^τ(B_θ× B_x) = E.  As the intersection of nonempty compact sets, this set is nonempty and compact.
That E is the largest invariant subset of B_θ× B_x follows from <cit.> (which is a general result on semiflows).
[Alternatively, one can prove the invariance of E by using the solution property (iii) of the differential inclusion listed above. Specifically, for each t ≥ 0, by the definition of E, it is clear that F^t(E) ⊂ E. To prove the reverse inclusion, suppose for some t ≥ 0 and y̅∈ E, y̅∉F^t(E). Fix that t, and let Δ be the set of all initial conditions from which y̅ can be reached at time t; i.e., Δ = {ŷ|ŷ∈ B_θ× B_x  and y(t) = y̅ if y(0) = ŷ}. Then Δ∩ E = ∅. By the property (iii) listed above, Δ is closed and therefore compact. Since E is by definition the intersection of nested nonempty compact sets, ∩_τ≥ 0 F^τ(B_θ× B_x),  in order to have Δ∩ E = ∅, we must have Δ∩ F^τ(B_θ× B_x) = ∅ for some τ≥ 0.  This in turn implies that y̅∉F^t+τ(B_θ× B_x) and hence y̅ cannot be in E, a contradiction. Thus we must also have F^t(E) ⊃ E for all t ≥ 0, proving that F^t(E)=E for all t ≥ 0.]



By the above lemma F^t(E) = E for all t ≥ 0. The next lemma is a consequence of this invariance of E and the solution property (iii) above. Its implications, especially, the following Cor. <ref>, will be useful for our study of what points can be in E.  

 
Let y̅∈ E, and let y(·) be the solution of the ODE (<ref>) with the initial condition y(0)=y̅. Then there exist a sequence of times τ_i →∞ such that y(τ_i) →y̅ as i →∞.



In this proof, abusing notation, we let F^t also stand for the mapping that maps an initial condition ŷ of the ODE (<ref>) to its solution at time t.
Thus, in particular, F^t(ŷ) = y̅ means y(t)=y̅ if y(0) = ŷ.

Since E is invariant by Lemma <ref>, for each t ≥ 0, there exists some y_t ∈ E such that F^t(y_t) = y̅. Since E is compact, we can choose an increasing sequence t_i →∞ such that y_t_i→ y_∞∈ E. Then

    F^t_i(y_∞) - y̅_2 =  F^t_i(y_∞) - F^t_i(y_t_i) _2 ≤ y_∞ - y_t_i_2  → 0  as i →∞,

where the last inequality follows from the solution property (iii) of the differential inclusion given earlier. Thus F^t_i(y_∞) →y̅ as i →∞.
For any j > i, by the solution property (iii) again,

    F^t_j - t_i (y̅) - F^t_j(y_∞) _2 =    F^t_j - t_i( F^t_i (y_t_i ) ) - F^t_j(y_∞) _2 ≤ y_t_i - y_∞_2.
    
Now choose an increasing sequence of integers j_i such that j_i > i and τ_i :=t_j_i - t_i →∞ as i →∞. We then have

    F^τ_i(y̅) - y̅_2    ≤ F^τ_i(y̅)  - F^t_j_i(y_∞) _2 +  F^t_j_i(y_∞) - y̅_2 
       ≤ y_t_i - y_∞_2 +  F^t_j_i(y_∞) - y̅_2  → 0   as  i →∞.
 
This proves the lemma.


 
Let y(·) be a solution of the ODE (<ref>) with y(0) ∈ E. Then, for any continuous function V(·) on B_θ× B_x such that V(y(t)) is nonincreasing in t, V(y(t)) is constant over t ≥ 0. 


We are now ready to prove Prop. <ref>, which says essentially that E ∩ (^d ×{ϕ()}) is the subset of saddle points, D_θ×{x̅}, of the minimax problem (<ref>), where x̅ is its unique dual optimal solution in {ϕ()}. We will use the preceding corollary together with the special properties of the convex-concave function ψ^o  to prove this. (In particular, besides the uniqueness of x̅, one property of ψ^o we will use is that the only term in ψ^o in which θ and x “interact” is a bilinear function of (θ, x); cf. (<ref>).)


Let E^o be the limit set of the ODE for all initial conditions θ(0) ∈ B_θ, x(0) ∈ B_x ∩{ϕ()}.
Any saddle point (θ̃, x̃) of the minimax problem (<ref>) lies in E. This is because (0,0) ∈∂ψ(θ̃, x̃) and so, in view of Lemma <ref>, the solution of the ODE (<ref>) from (θ̃, x̃) is just θ(·)≡θ̃,  x(·) ≡x̃, implying that (θ̃, x̃) is in the limit set of the ODE, which is E by Lemma <ref>.
Thus, D_θ×{x̅}⊂ E^o ⊂ E ∩ (^d ×{ϕ()}). To prove the proposition, we now use Cor. <ref> and proof by contradiction to show that E ∩ (^d ×{ϕ()}) cannot contain other points than those saddle points D_θ×{x̅}.

First, suppose (θ̂, x̂) ∈ E where x̂∈{ϕ()} and x̂≠x̅. Let y(·) be the solution of the ODE (<ref>) with y(0) = (θ̂, x̂). Since x̂∈{ϕ()}, the x-component of y(t) lies in {ϕ()} for all t; since y(·) is continuous, there is a nonzero t̅ such that for all t < t̅, the x-component of y(t) never equals x̅.
For any t ≥ 0, consider V̇(y(t)) for the function 
V(θ, x) = 12θ - θ̅_2^2 + 12 x - x̅_2^2, where θ̅∈ D_θ so that (θ̅, x̅) is a saddle point of (<ref>).
To simplify notation, write 

    k̅(θ, x) = A θ + b - C x     and    g̅(θ, x) = - A^ x,

where A = Φ^Ξ (P^(λ) - I ) Φ, C = Φ^Ξ Φ, and the vector b is the constant term
[Recall the expression (<ref>) of the operator : v = r_π^(λ) +  v; the vector b here is given by b=Φ^Ξ   r_π^(λ).]

in the affine function k̅(θ, x) (cf. (<ref>)).
Since (θ̅, x̅) is a saddle point of (<ref>), (0,0) ∈∂ψ(θ̅, x̅), so for some z̅_1 ∈ - _B_θ(θ̅) and z̅_2 ∈ - _B_x(x̅), 

    A^x̅ + ∇ p(θ̅) - z̅_1 = 0,       A θ̅+ b - C x̅ + z̅_2 = 0
  
(cf. (<ref>)-(<ref>)).
At a point (θ, x) = y(t), let z_1 ∈ - _B_θ(θ), z_2 ∈ - _B_x(x) be the boundary reflection terms in (<ref>) at time t. We have   

    V̇(y(t))     = ⟨θ - θ̅,   - A^ x - ∇ p(θ) + z_1 ⟩  +  ⟨ x - x̅,   A θ + b - C x + z_2  ⟩≤ 0,

where the non-positivity follows from the monotonicity of the set-valued mapping (·) and the fact that (0,0) ∈ -(θ̅, x̅) and (- A^ x - ∇ p(θ) + z_1,   A θ + b - C x + z_2) ∈ -(θ, x) (Lemma <ref>).
Consider each term in the expression of V̇(y(t)) above. Using (<ref>), we have

    ⟨θ - θ̅, - A^ x - ∇ p(θ) + z_1 ⟩    = ⟨θ - θ̅,   - A^ x - ∇ p(θ) + z_1 ⟩ + ⟨θ - θ̅,   A^x̅ + ∇ p(θ̅) - z̅_1 ⟩
        = - ⟨θ - θ̅,   A^ (x - x̅) ⟩ - ⟨θ - θ̅,  ∇ p(θ) - ∇ p(θ̅) ⟩ +  ⟨θ - θ̅,   z_1 - z̅_1 ⟩, 
    ⟨ x - x̅,   A θ + b - C x + z_2 ⟩    = ⟨ x - x̅,   A θ + b - C x + z_2 ⟩ - ⟨ x - x̅,   A θ̅+ b - C x̅ + z̅_2 ⟩
        = ⟨ x - x̅,   A (θ - θ̅) ⟩ -  ⟨ x - x̅,   C (x - x̅) ⟩ + ⟨ x - x̅,   z_2 - z̅_2 ⟩,
 
and hence 

    V̇(y(t))     = - ⟨θ - θ̅,  ∇ p(θ) - ∇ p(θ̅) ⟩ +  ⟨θ - θ̅,   z_1 - z̅_1 ⟩ 
       -  ⟨ x - x̅,   C (x - x̅) ⟩  + ⟨ x - x̅,   z_2 - z̅_2 ⟩.

In the summation on the r.h.s., each term is nonpositive,
[For the first, second and forth terms, this is because ∇ p(·), _B_θ(·) and _B_x(·) are all monotone set-valued mappings <cit.>. For the third term, it is because -C is a symmetric negative semidefinite matrix.]

but the third term -  ⟨ x - x̅, C (x - x̅) ⟩ < 0 if x ∈{ϕ()} and x ≠x̅. 
Thus V(y(t)) is a nonincreasing function of t and for almost all t ∈ [0, t̅ ), V̇(y(t)) < 0, so V(y(t)) cannot remain constant. This contradicts Cor. <ref>. Therefore, points like (θ̂, x̂) cannot lie in E; i.e., if (θ̂, x̂) ∈ E and x̂∈{ϕ()}, then x̂ = x̅.

Next, suppose for some θ̂∉D_θ, (θ̂, x̅) ∈ E. 
Let y(·) be the solution with y(0) = (θ̂, x̅).
Since y(t) ∈ E by the invariance of E (Lemma <ref>) and its x-component x(t) ∈{ϕ()} for all t, we must have x(·) ≡x̅ by the preceding proof. This means that for all t, ẋ(t) = 0 and therefore by (<ref>) and (<ref>),

    0 = k̅(y(t)) + z_2(t) ∈∂ψ_x(y(t)),
  
where z_2(t) is the boundary reflection term. Since y(·) is continuous and its θ-component starts from θ̂ outside the compact set D_θ,  
there is a nonzero time t̅ such that for all t < t̅, y(t) are not saddle points of (<ref>).

Consider now V̇(y(t)) for the differentiable function V(θ,x) =  ⟨x̅, A θ⟩ + p(θ).
At the point (θ, x̅) = y(t), we have g̅(θ, x̅) = - A^x̅, so with z_1 ∈ - _B_θ(θ) being the boundary reflection term z_1(t) in (<ref>), we have

    V̇(y(t))     = ⟨ A^x̅ + ∇ p(θ),    - A^x̅  - ∇ p(θ) + z_1 ⟩
        = -  A^x̅ + ∇ p(θ) - z_1 _2^2 + ⟨ z_1,   - A^x̅- ∇ p(θ) + z_1 ⟩.

By Lemma <ref>, the second term is 0. So V̇(y(t)) ≤ 0, implying that V(y(t)) is a nonincreasing function of t. 
For t < t̅, since (θ, x̅) = y(t) is not a saddle point of (<ref>), the first term in (<ref>) is strictly less than 0 (otherwise, in view of (<ref>) and (<ref>), we would have (0,0) ∈∂ψ(y(t)), implying y(t) is a saddle point, a contradiction). 
Thus V̇(y(t)) < 0 for almost all t in the interval [0, t̅ ) and consequently V(y(t)) cannot remain constant. This contradicts Cor. <ref>. So we conclude that if (θ̂, x̅) ∈ E, then θ̂∈ D_θ.
This completes the proof.


Finally, let us give the proof of Lemma <ref>, which says that if x_ opt is in the interior of the constraint set B_x, then the limit set D_θ×{x̅} coincides with the desired set Θ_ opt×{ x_ opt}.


The point x_ opt is the unique optimal solution of the maximization problem sup_x ∈^d{inf_θ∈ B_θψ^o(θ, x) } in the subspace {ϕ()}.
So if x_ opt∈ B_x, we must have x̅ = x_ opt. 
Now suppose x̅ =  x_ opt∈int(B_x), the interior of B_x.
By the definition of a saddle point, we have that θ̅∈ D_θ if and only if

    ψ^o(θ, x̅) ≥ψ^o(θ̅, x̅) ≥ψ^o(θ̅, x),     ∀ θ∈ B_θ,  x ∈ B_x,
 
whereas θ̅∈Θ_ opt if and only if

    ψ^o(θ, x̅) ≥ψ^o(θ̅, x̅) ≥ψ^o(θ̅, x),     ∀ θ∈ B_θ,  x ∈^d.

Then any θ̅∈ D_θ must satisfy (<ref>), because the second inequality in (<ref>) together with the fact x̅∈int(B_x) implies that x̅ attains a local maximum of the concave function ψ^o(θ̅, ·), and hence x̅ must attain the global maximum of ψ^o(θ̅, ·). This shows D_θ⊂Θ_ opt. 
Of course, any θ̅∈Θ_ opt satisfies (<ref>) and is therefore in D_θ. This shows D_θ×{x̅} =Θ_ opt×{ x_ opt} if x_ opt∈int(B_x).





  §.§.§ Convergence properties
 

We can now state our convergence results for the single-time-scale GTDa algorithm. 
Recall that

    Θ_ opt = _θ∈ B_θ{ J(θ) + p(θ) },      x_ opt =  _x ∈{ϕ()}{inf_θ∈ B_θψ^o(θ, x) }.

The set
Θ_ opt×{x_ opt} is a subset of saddle points of the minimax problem (<ref>), which we want to solve and which places no constraints on the maximizer.  The set D_θ×{x̅} is a subset of saddle points of the minimax problem (<ref>) with the constraint set B_x for the maximizer. 

 
Consider the GTDa algorithm (<ref>)-(<ref>) under Assumption <ref>. Let the initial x(0), _0 ∈{ϕ()}. Then for each initial condition of the algorithm, the following hold:



  (i) In the case of diminishing stepsize, let the stepsizes {α_n} satisfy the conditions in Assumption <ref> for the fast-time-scale stepsizes. Then 
lim_n →∞[  dist( (θ_n, x_n),   D_θ×{x̅})] = 0. Moreover, there exists a sequence of positive numbers T_n →∞ such that for any ϵ > 0,

    lim sup_n →∞( (θ_i, x_i) ∉N_ϵ(D_θ×{x̅}),  some i ∈ [n, m(n, T_n) ] )  = 0.
  

  (ii) In the case of constant stepsize α_n = α for all n, for any integers n_α such that α  n_α→∞ as α→ 0,
there exist positive numbers {T_α|α > 0} with T_α→∞ as α→ 0, such that for any ϵ > 0,

    lim sup_α→ 0( (θ^α_n_α + i, x^α_n_α + i)  ∉N_ϵ(D_θ×{x̅}),  some i ∈ [0, T_α/α ] )  = 0.
  

  (iii) In the case of constant stepsize, for the averaged iterates {(θ̅^α_n, x̅^α_n)},
[Similar to (<ref>), these averaged iterates are given by θ̅^α_n = 1/n - n_0∑_i=n_0^n-1θ^α_i,  x̅^α_n = 1/n - n_0∑_i=n_0^n-1 x^α_i.] 

the conclusions of Theorem <ref> hold with the set D_θ×{x̅} in place of Θ_ opt, with the averaged (θ,x)-iterates in place of the averaged θ-iterates, and with obvious modifications to remove all the references to the fast-time-scale stepsize parameter β.


If the constraint set B_x contains x_ opt in its interior, then D_θ×{x̅} =Θ_ opt×{ x_ opt} in the above.




For the parts (i) and (ii), we first apply stochastic approximation theory to show that the average dynamics of the algorithm is characterized by the mean ODE (<ref>). Specifically, in the case of diminishing stepsize, we apply <cit.>, and in the case of constant stepsize, we apply <cit.>. The required conditions are straightforward to verify.
[In order to apply <cit.> to the algorithm (<ref>)-(<ref>), the conditions we need to verify include uniform integrability, tightness, continuity conditions as well as an “averaging condition,” similar to those listed in Section <ref>. In particular, for diminishing stepsize, using the notations of Section <ref> for GTDa, we need the following sets to be uniformly integrable 

    {( g(θ_n, x_n, Z_n) - ∇ p(θ_n),   k(θ_n, x_n, Z_n) + _n ω_n+1) }_n ≥ 0,  {( g(θ, x, Z_n) - ∇ p(θ),   k(θ, x, Z_n) ) | n ≥ 0, θ∈ B_θ, x ∈ B_x }.

Since the algorithm is constrained, the uniform integrability of these sets follows from the uniform integrability of {_n}, as explained in Section <ref>. For the same reason, the uniform integrability conditions are met in the case of constant stepsize (we do not write down those sets here, which are the same as the above except that they involve the (θ, x)-iterates generated with all stepsize parameters α.) 
The tightness condition on {Z_n} is satisfied as before. The continuity condition requires the functions g(θ, x, z) - ∇ p(θ) and k(θ, x, z) to be continuous in (θ, x) uniformly in z in a compact set. It is satisfied clearly as before. The functions g̅(θ, x) - ∇ p(θ) and k̅(θ, x) are required to be continuous, which are also clearly satisfied. Finally, the “averaging condition” requires that for each (θ, x) and each compact set D ⊂,

    lim_m, n →∞1/m∑_i=n^n+m-1_n [ g(θ, x,  Z_i) - g̅(θ, x) ] (Z_n ∈ D) = 0   in mean, 
       lim_m, n →∞1/m∑_i=n^n+m-1_n [ k(θ, x,  Z_i) - k̅(θ, x) ] (Z_n ∈ D) = 0   in mean.

This follows from the same proof of Lemma <ref>(i), since both g(θ, x, ·) and k(θ, x, ·) are Lipschitz continuous in the trace variable . Note that the regularizer p(θ) does not appear in the “averaging condition” at all, although ∇ p(θ) appears in the algorithm and its mean ODE. This is because ∇ p(θ) does not depend on z, so for fixed θ, ∇ p(θ) and _ζ [∇ p(θ)] canceled each other out in the first convergence-in-mean requirement above. Thus, for the constrained algorithm, the convergence analysis is the same with or without the regularizer p(·). 
]

These theorems give us the conclusions stated in the part (i) and (ii), respectively, with the set involved being the limit set of the ODE associated with the algorithm, and we then use the expression of the limit set given by Prop. <ref> to obtain the conclusions as stated. 

The procedure of deriving the part (iii) for the averaged (θ,x)-iterates is similar to that described in Section <ref> after Theorem <ref>. It uses the observation that with constant stepsize α, the iterates (θ_n^α, x^α_n) jointly with the states/memory states and traces form a weak Feller Markov chain, and the proof of this part is essentially the same as that given in <cit.>.

Finally, the last statement about when D_θ×{x̅} =Θ_ opt×{ x_ opt} follows from Lemma <ref>.



  

Recall that for the two-time-scale GTDa and GTDb algorithms in Section <ref>, to ensure desired convergence properties, we required the constraint set B_x to be large enough so that B_x ⊃{x_θ|θ∈ B_θ}, where x_θ is the unique solution to the linear system of equations, k̅(θ, x) =0, x ∈{ϕ()}. 
For each θ, an optimal solution to the maximization problem sup_xψ^o(θ, x) is just x_θ. 
Therefore, x_ opt must be in the set {x_θ|θ∈ B_θ}, and if int(B_x) ⊃{ x_θ|θ∈ B_θ}, then x_ opt∈int(B_x). This shows that, to ensure that the single-time-scale GTDa can approach the desired solution set Θ_ opt, a sufficient condition for B_x is one that is almost identical to the condition we imposed earlier for the two-time-scale GTD. 
But the latter sufficient condition is clearly far more stronger than the inclusion x_ opt∈int(B_x) needed by the single-time-scale GTDa. In particular, if x_ opt = 0, as is the case when the optimal solutions in Θ_ opt satisfy that J(θ) = 0 (i.e., they solve the projected Bellman equation), then we only need 0 ∈int(B_x), which is always true. 
Another remark is that, as we will show in Section <ref>, the weaker condition x_ opt∈int(B_x) is in fact also sufficient for the two-time-scale GTDa algorithm, because it can be viewed as a two-time-scale algorithm for solving the minimax problem (<ref>), with the maximizer operating at a faster time-scale than the minimizer.
This suggests an advantage of the single- or two-time-scale constrained GTDa algorithm over the two-time-scale constrained GTDb algorithm.


 Theorem <ref> also holds for the single-time-scale GTDa algorithm when the λ-parameters are set according to the composite scheme described in Section <ref>. The analysis is essentially the same as the one given in this subsection, apart from that a different Bellman operator  is involved.




  §.§.§ Biased variant
 

For the case of state-dependent λ, one can apply a biased variant of the single-time-scale GTDa algorithm, which is more robust. Like its two-time-scale counterpart (<ref>)-(<ref>) we discussed in Section <ref>, this algorithm uses a bounded function h() of traces in computing {(θ_n, x_n)}: 

    θ_n+1    = Π_B_θ( θ_n + α_n  ρ_n (ϕ(S_n) - _n+1ϕ(S_n+1) ) · h(_n)^ x_n - α_n ∇ p(θ_n) ), 
    
       x_n+1    = Π_B_x( x_n + α_n (h(_n) ·δ_n(v_θ_n) - ϕ(S_n) ϕ(S_n)^ x_n) ).
 
We shall assume that h is a bounded Lipschitz continuous function. 
As in Section <ref>, we consider a family of bounded Lipschitz continuous functions {h_K | K > 0} that satisfy (<ref>)-(<ref>), and we analyze the behavior of the above algorithm for h=h_K, as K increases (and the degree of bias correspondingly decreases). We will show that the algorithm solves minimax problems that are approximations of the minimax problem inf_θ∈ B_θsup_x ∈ B_xψ^o(θ, x) solved by GTDa.

The mean ODE associated  with the above biased GTDa has the same form as the ODE (<ref>) of GTDa, 

    ( [ θ̇(t);  ẋ(t) ]) 
     =  ( [ g̅_h(θ(t), x(t)) - ∇ p(θ(t));           k̅_h (θ(t), x(t) ) ]) +  ( [ z_1(t); z_2(t) ]),         z_1(t) ∈ - _B_θ(θ(t)),   z_2(t) ∈ - _B_x(x(t)),
 
where the functions g̅_h, k̅_h are defined according to (<ref>)-(<ref>) as

    g̅_h(θ, x) =   _ζ[  ρ_0 (ϕ(S_0) - _1ϕ(S_1) ) · h(_0)^ x ],   k̅_h(θ, x) =  _ζ[ h(_0)  δ̅_0(v_θ) ] - Φ^Ξ Φ  x.

Recall δ̅_0(v_θ) = ρ_0 r(S_0, S_1) + ρ_0 (_1ϕ(S_1) - ϕ(S_0) )^θ. Denote C = Φ^Ξ Φ, and define matrix A_h and vector b_h by

    A_h = _ζ[ h(_0)  ·ρ_0 (_1ϕ(S_1) - ϕ(S_0) )^],      b_h = _ζ[ h(_0)  ·ρ_0 r(S_0, S_1) ].

Then

    g̅_h(θ, x) = -A_h^ x,     k̅_h(θ, x) = A_h θ + b_h  - C x.

Note that as shown in the proof of Lemma <ref>, under the condition (<ref>) on the function h() (which requires h() ∈{ϕ()} for ∈{ϕ()}),
b_h ∈{ϕ()} and the column space of A_h is contained in {ϕ()}.

Now consider the minimax problem 

    inf_θ∈ B_θsup_x ∈ B_xψ^o_h(θ, x),
  
for the convex-concave function

    ψ^o_h(θ, x) : =  x^  ( A_h θ + b_h) -  12  x^ C x + p(θ).

The structure of this minimax problem and its relation with the ODE (<ref>) are exactly the same as that of the minimax problem (<ref>) and its relation with the ODE (<ref>). Thus, the same analysis given in Section <ref> for GTDa applies to the biased variant here.
In particular, Prop. <ref> holds in this case, with the minimax problem involved being (<ref>).
That is, for all initial x(0) ∈{ϕ()}, the limit set of the ODE (<ref>) is the subset of saddle points, D^h_θ×{x̅^h}, of the minimax problem (<ref>), where D^h_θ consists of its primal optimal solutions and x̅^h is its unique dual optimal solution in {ϕ()}, i.e.,

    D^h_θ = _θ∈ B_θ{sup_x ∈ B_xψ^o_h(θ, x) },   x̅^h =  _x ∈{ϕ()}{inf_θ∈ B_θψ^o_h(θ, x) }.

Moreover, regarding the convergence properties of the biased GTDa algorithm, the same conclusions of Theorem <ref>(i)-(iii) hold with D^h_θ×{x̅^h} in place of D_θ×{x̅}.

We now relate the minimax problem (<ref>) and its saddle points D^h_θ×{x̅^h} to the minimax problem (<ref>) solved by GTDa and its saddle points D_θ×{x̅}.

 
Let { h_K | K > 0} be a family of bounded Lipschitz continuous functions that satisfy (<ref>)-(<ref>). Then
sup_θ∈ B_θ, x ∈ B_x| ψ^o_h_K(θ, x) - ψ^o(θ, x) | → 0 as K →∞,
and for any ϵ > 0, there exist K_ϵ > 0 such that

    D^h_K_θ×{x̅^h_K}⊂ N_ϵ( D_θ×{x̅}),     ∀  K ≥ K_ϵ.




The proof is similar to that of Lemma <ref>. We can express the convex-concave function ψ^o as 
ψ^o(θ, x) = x^ (A θ + b) - 12 x^ C x + p(θ), where
A = _ζ[ _0  ·ρ_0 (_1ϕ(S_1) - ϕ(S_0) )^] and b = _ζ[ _0  ·ρ_0 r(S_0, S_1) ].
Then 

    ψ^o_h_K(θ, x) - ψ^o(θ, x)  = x^ (A_h_K - A) θ +  x^ (b_h_K - b).

To prove sup_θ∈ B_θ, x ∈ B_x| ψ^o_h_K(θ, x) - ψ^o(θ, x) | → 0 as K →∞, since θ and x are confined in bounded sets, it is sufficient to prove that 
A_h_K - A → 0 and b_h_K - b → 0 as K →∞. In turn, in view of the expressions of these matrices and vectors, it is sufficient to prove
E_ζ[   h_K(_0) - _0  ]  → 0 as K →∞, which we already proved in the proof of Lemma <ref>.

For the second part of the lemma, we use proof by contradiction. Suppose the statement does not hold. Then there exist some ϵ > 0, a sequence {K_i} with K_i →∞ as i →∞, and a sequence of points (θ_i, x_i) ∈ D^h_i_θ×{x̅^h_i} with h_i = h_K_i, such that (θ_i, x_i) ∉N_ϵ( D_θ×{x̅}) for all i.
Since all the points lie in the compact set B_θ× B_x,
by choosing a subsequence if necessary, we can assume that the sequence {(θ_i, x_i)} converges to (θ_∞, x_∞) ∈ B_θ× B_x.
Then x_∞∈{ϕ()} (since all the x_i's lie in this subspace), and (θ_∞, x_∞) ∉D_θ×{x̅}. 

On the other hand, each (θ_i, x_i) is a saddle point of the minimax problem (<ref>) for the corresponding function ψ^o_h_i. Therefore, 
(0,0) ∈∂ψ_h_i(θ_i, x_i), where ψ_h_i(θ, x) = ψ^o_h_i(θ, x) + δ_B_θ(θ) - δ_B_x(x); or equivalently,

    - ∇_θψ^o_h_i(θ_i, x_i) ∈_B_θ(θ_i),     ∇_x ψ^o_h_i(θ_i, x_i) ∈_B_x(x_i).
  
By <cit.>, the convergence of ψ^o_h_i to ψ^o proved above implies that as (θ_i, x_i) → (θ_∞, x_∞),

    ∇ψ^o_h_i(θ_i, x_i) →∇ψ^o(θ_∞,x_∞).

Then by the outer semicontinuity of the set-value mappings _B_θ(·) and _B_x(·) <cit.>, (<ref>) together with (<ref>) implies that

    - ∇_θψ^o(θ_∞, x_∞) ∈_B_θ(θ_∞),     ∇_x ψ^o(θ_∞, x_∞) ∈_B_x(x_∞)
 
or equivalently, (0,0) ∈∂ψ(θ_∞, x_∞) (where ψ(θ, x) = ψ^o(θ, x) + δ_B_θ(θ) - δ_B_x(x) as we recall). This shows that (θ_∞, x_∞) is a saddle point of the minimax problem (<ref>), so, since x_∞∈{ϕ()}, (θ_∞, x_∞) ∈ D_θ×{x̅}, a contradiction. This proves the second part of the lemma.


Combining the preceding lemma with the convergence properties of the algorithm mentioned earlier, we obtain the following convergence theorem.

 
For the case of state-dependent λ, consider the biased single-time-scale GTDa algorithm (<ref>)-(<ref>) under Assumption <ref>. In the algorithm, let the function h ∈{ h_K | K > 0}, a family of bounded Lipschitz continuous functions that satisfy (<ref>)-(<ref>), and let the initial x(0), _0 ∈{ϕ()}. Then for each ϵ > 0, there exists K_ϵ > 0 such that if h=h_K with K ≥ K_ϵ, the conclusions of Theorem <ref>(i)-(iii) hold for the algorithm (with “for any ϵ > 0” removed from the statements). 
Moreover, as in Theorem <ref>, if the constraint set B_x contains x_ opt in its interior, then in the conclusions, the set D_θ×{x̅} =Θ_ opt×{ x_ opt}.




  §.§.§ Further remarks
 

Finally, we remark on another variation of the GTDa algorithm and show how the analysis given in this subsection applies. In <cit.> the unconstrained single-time-scale GTDa algorithm can use an additional parameter η > 0 to scale the stepsizes α_n for the x-iterates:

    θ_n+1    = θ_n + α_n  ρ_n (ϕ(S_n) - _n+1ϕ(S_n+1) ) ·_n^ x_n,  
    
       x_n+1    = x_n + ηα_n (_n δ_n(v_θ_n) - ϕ(S_n) ϕ(S_n)^ x_n).
 
The analyses <cit.> of this linear update rule equate it to the following recursion via a change of variable x̃ = x/√(η) or x =  √(η) x̃:

    θ_n+1    = θ_n +  α_n  √(η)ρ_n (ϕ(S_n) - _n+1ϕ(S_n+1) ) ·_n^x̃_n, 
    x̃_n+1    = x̃_n + α_n √(η)(_n δ_n(v_θ_n) - √(η)ϕ(S_n) ϕ(S_n)^x̃_n).
 

A constrained version of the above algorithm, for minimizing the regularized objective function J(θ) + p(θ), is

    θ_n+1    = Π_B_θ( θ_n + α_n  ρ_n (ϕ(S_n) - _n+1ϕ(S_n+1) ) ·_n^ x_n - α_n ∇ p(θ_n) ),  
    
       x_n+1    = Π_B_x( x_n + ηα_n (_n δ_n(v_θ_n) - ϕ(S_n) ϕ(S_n)^ x_n) ).
 
Scaling the coordinates by 1/√(η) for the x-iterates as before, and with B̃_x = B_x/√(η) and with the correspondence x_n = √(η) x̃_n,  we see that (<ref>)-(<ref>) are equivalent to 

    θ_n+1    = Π_B_θ(  θ_n +  α_n  √(η)ρ_n (ϕ(S_n) - _n+1ϕ(S_n+1) ) ·_n^x̃_n - α_n ∇ p(θ_n) ), 
    x̃_n+1    = Π_B̃_x(  x̃_n + α_n √(η)(_n δ_n(v_θ_n) - √(η)ϕ(S_n) ϕ(S_n)^x̃_n) ).
 
Thus one can carry out this constrained algorithm in either form, and to analyze its behavior, it suffices to analyze the behavior of (<ref>)-(<ref>). The latter can be understood by recognizing the minimax problem associated with (<ref>)-(<ref>). 

This minimax problem is similar to the minimax problem (<ref>) discussed earlier and has the same structure. Similar to the derivation of (<ref>) and (<ref>) in Section <ref>, for any a > 0, we can express 1a J(θ) as the optimal value of a maximization problem using the identity relation  

    12 a v _ξ^2  = sup_y {⟨ y, v ⟩_ξ - a2 y ^2_ξ},     ∀  v ∈^||,

and then convert the minimization problem inf_θ∈ B_θ{ J(θ) + p(θ) } = inf_θ∈ B_θ{ a ·1a J(θ) + p(θ)} to the equivalent minimax problem

    inf_θ∈ B_θsup_x̃∈^d{ a  ⟨Φx̃,   v_θ - v_θ⟩_ξ - a^2/2Φx̃^2_ξ  + p(θ) }.
 
Adding the constraint set B̃_x for the maximizer then gives the minimax problem associated with (<ref>)-(<ref>):

    inf_θ∈ B_θsup_x̃∈B̃_x{ a  ⟨Φx̃,   v_θ - v_θ⟩_ξ - a^2/2Φx̃^2_ξ  + p(θ) }  where  a = √(η).
 

The two minimax problems above have the same structure as the minimax problems (<ref>) and (<ref>) discussed in Section <ref>, and the reasoning we gave there applies here.
In particular, the mean ODE associated with (<ref>)-(<ref>) corresponds to differential inclusion with the maximal monotone mapping associated with the minimax problem (<ref>), and all the results given in Sections <ref>-<ref> for the single-time-scale GTDa (where η = 1) hold for the algorithm (<ref>)-(<ref>), with the minimax problems (<ref>) and (<ref>) in place of (<ref>) and (<ref>), respectively. (These results then translate easily to the alternative form of the algorithm, (<ref>)-(<ref>), via the correspondence x_n = √(η) x̃_n.) Similarly, the results given in Section <ref>, which relate the biased variant of GTDa to GTDa in the case of η = 1, also hold for the biased variant of the algorithm (<ref>)-(<ref>) or (<ref>)-(<ref>) with η > 0, except that now the minimax problems involved are the problem (<ref>) and its approximations.

Although the line of convergence analysis is the same for different values of η, the behavior of the algorithm is certainly affected by η. This is reflected by the different minimax problems corresponding to different η, as shown above. In addition, we can also see the differences by considering two extreme cases. Suppose we run the algorithm with a small constant stepsize, and let us examine the algorithm of the form (<ref>)-(<ref>). If η is very large, the algorithm resembles the two-time-scale GTDa algorithm with the x-iterates (which correspond to the maximizer) running at the fast time-scale. If η < < 1, however, then the algorithm again resembles a two-time-scale algorithm but with the θ-iterates (which correspond to the minimizer) running at the fast time-scale.



 §.§ Two-Time-Scale GTDa Revisited
 

In this subsection we revisit the two-time-scale GTDa algorithm and its biased variant. Recall that in Section <ref> we regarded them as (stochastic) gradient-descent and approximate gradient-descent algorithms for minimizing J(θ). There, for GTDa, we required that the constraint set B_x satisfies the condition (<ref>): B_x ⊃{x_θ|θ∈ B_θ}, in order to ensure that the gradients ∇ J(θ) can be computed by the algorithm for all θ∈ B_θ. This condition is, however, unnecessarily strong, as we already remarked (cf. Remark <ref> near the end of Section <ref>). Using the minimax problem formulation, we can now address the behavior of the two-time-scale GTDa algorithm when B_x ⊅{x_θ|θ∈ B_θ} and the algorithm cannot compute all the gradients ∇ J(θ). In this case, as will be shown below, the algorithm is using gradient-ascent to solve the inner maximization problem in the minimax problem (<ref>) at the fast time-scale, and using gradient-descent to solve the outer minimization problem in (<ref>) at the slow time-scale. Thus we can weaken the condition on B_x and require merely that B_x contains x_ opt in its interior, without losing the convergence properties of GTDa derived earlier in Section <ref>. Similarly, for the two-time-scale biased variant of GTDa, the condition on B_x can also be weakened. 

The result is summarized in the theorem below. The objective function we consider here is the regularized objective function J_p as in Section <ref>, so the algorithms we refer to in the theorem have, in their θ-iterations, the gradient term - α_n ∇ p(θ_n), just as their single-time-scale counterparts discussed in Section <ref>. 

 
For the two-time-scale GTDa algorithm and its biased variant, Theorems <ref>-<ref> and <ref>-<ref> still hold if we replace the condition on B_x in these theorems by the weaker condition that x_ opt∈ int(B_x), where x_ opt is the unique dual optimal solution of the minimax problem (<ref>) in {ϕ()}.


In the rest of this subsection, we provide the proof details. To prove this theorem, we only need to re-characterize, under the new condition on B_x, the limit sets of the mean ODEs associated with the two algorithms. We will show that the fast-time-scale mean ODEs still have solution properties that satisfy the requirements in order to apply the ODE-based stochastic approximation analysis given earlier in Section <ref>. We will then consider the limit sets of the slow-time-scale mean ODEs, and show that in the case of GTDa, the limit set is the desired optimal solution set Θ_opt, and in the case of the biased variant, it is a set contained in a neighborhood of Θ_opt whose size decreases as the bias introduced by the algorithm decreases. Theorem <ref> then follow from these arguments.

Let us start with GTDa. Recall the minimax problem (<ref>): inf_θ∈ B_θsup_x ∈ B_xψ^o(θ, x), where B_x is a ball centered at the origin as before and with an arbitrary radius. 
For each θ, let

    J̃_p(θ) = sup_x ∈ B_xψ^o(θ, x) = sup_x ∈ B_x{⟨Φ x,  v_θ - v_θ⟩_ξ - 12Φ x ^2_ξ} + p(θ).
 
For the maximization problem above, denote by x̃(θ) its unique optimal solution in {ϕ()}. 

 
The convex function J̃_p(θ) is differentiable with ∇J̃_p(θ) = ∇_θψ^o(θ, x̃(θ)), and x̃(θ) is a continuous function of θ.



For any optimal solution x̂ of the maximization problem (<ref>), the projection of x̂ on {ϕ()} coincides with x̃(θ), so Φx̂ = Φx̃(θ). Consequently, ∇_θψ^o(θ, x̂) = ∇_θψ^o(θ, x̃(θ)) for all optimal solutions x̂ of (<ref>). It then follows from <cit.> that the function J̃_p(θ) is differentiable at θ with ∇J̃_p(θ) = ∇_θψ^o(θ, x̃(θ)), for all θ∈^d.

For the continuity of x̃(θ), note that as θ→θ̅, J̃_p(θ) = ψ^o(θ, x̃(θ)) →J̃_p(θ̅) =  ψ^o(θ̅, x̃(θ̅)) by the continuity of the function J̃_p. Let x_∞∈ B_x ∩{ϕ()} be the limit of any convergent subsequence of x̃(θ) as θ→θ̅. By the continuity of the functions ψ^o and J̃_p, we have J̃_p(θ̅) = ψ^o(θ̅, x_∞) and hence both x_∞ and x̃(θ̅) are optimal solutions in {ϕ()} for the maximization problem (<ref>) corresponding to θ̅. Since the latter has a unique optimal solution in {ϕ ()}, we must have x_∞ = x̃(θ̅). This proves that x̃(θ) →x̃(θ̅) as θ→θ̅.


We now show that at the fast time-scale of GTDa, the associated mean ODE (<ref>) has solutions (θ(·), x(·)) with θ(·) ≡θ(0) and x(t) →x̃(θ(0)) as t →∞. This is a counterpart of Lemma <ref> when the condition (<ref>) on B_x has been removed, and the proof is very similar to that of Lemma <ref>.

 
For all initial x(0) ∈ B_x ∩{ϕ()} and θ(0) ∈ B_θ,
the limit set of the ODE (<ref>) is {(θ, x̃(θ)) |θ∈ B_θ}.



For each θ∈ B_θ, with θ(·) ≡θ, consider a solution x(·) of the ODE (<ref>) with x(0) ∈{ϕ()}. Let V_θ(x) = 12 x - x̃(θ) _2^2, and we calculate V̇_θ(x(t)). Note that by the optimality of x̃(θ), there exists some z̃∈ - _B_x(x̃(θ)) such that ∇_x ψ^o(θ, x̃(θ)) + z̃ = 0. This is the same as b_θ - C x̃(θ) + z̃ = 0 for C = Φ^Ξ Φ and b_θ = Φ^Ξ  ( v_θ - v_θ). Since k̅(θ, x) = ∇_x ψ^o(θ, x) = b_θ - C x, we have 

    V̇_θ(x(t))     = ⟨ x(t) - x̃(θ),   b_θ - C x(t) + z(t) ⟩
        =  ⟨ x(t) - x̃(θ),   b_θ - C x(t) + z(t)  - ( b_θ - C x̃(θ) + z̃) ⟩
        = - ⟨ x(t) - x̃(θ), C(x(t) - x̃(θ)) ⟩ + ⟨ x(t) - x̃(θ),   z(t) - z̃⟩.
   
Since x(t) ∈{ϕ()}, the first term on the r.h.s. is bounded above by - c   x(t) - x̃(θ) _2^2, where c > 0 is the smallest nonzero eigenvalue of the symmetric positive semidefinite matrix C. The second term on the r.h.s. is nonpositive by the monotonicity of the set-valued mapping _B_x(·) <cit.>.
Thus, we have V̇_θ(x(t)) ≤ - c  x(t) - x̃(θ) _2^2. Since on B_x, {V_θ|θ∈ B_θ} are uniformly bounded, this implies that for any ϵ > 0, there is a time t_ϵ independent of θ such that x(t) - x̃(θ) _2^2  ≤ϵ for all t ≥ t_ϵ. The conclusion of the lemma then follows. 


Lemma <ref>, together with the continuity of x̃(θ) given in Lemma <ref>, shows that the fast-time-scale mean ODE satisfies the requirements in the convergence proofs given earlier in Sections <ref>-<ref>. We now consider the mean ODE associated with the slow time-scale. Since at the fast time-scale the x-iterates of GTDa “track” x̃(θ) instead of x_θ for the slowly varying θ-iterates, instead of the ODE (<ref>), the mean ODE for the slow time-scale is now given by

    θ̇(t) = g̅(θ(t), x̃(θ(t))) - ∇ p(θ(t))  + z(t),     z(t) ∈ - _B_θ(θ(t)),

where z(t) is the boundary reflection term. (The reasoning is the same as that given in Section <ref> when deriving (<ref>) and (<ref>).)
Since ∇_θψ^o(θ, x̃(θ))  = - g̅(θ, x̃(θ)) + ∇ p(θ) (cf. (<ref>)), by Lemma <ref>, the above ODE is the same as

    θ̇(t) = - ∇J̃_p(θ(t))  + z(t),     z(t) ∈ - _B_θ(θ(t)),

and it corresponds to using gradient-descent to minimize J̃_p on B_θ. So by the same proof of Lemma <ref> (with the function J̃_p in place of J), we see that the limit set of the ODE (<ref>) is the solution set _θ∈ B_θJ̃_p(θ), which, by Lemma <ref>, coincides with the desired set Θ_opt when x_ opt∈int(B_x). 
The first part of Theorem <ref> for GTDa then follows; that is, Theorems <ref>-<ref> and <ref> still hold for GTDa under the much weaker condition on B_x.


We now consider the two-time-scale biased variant of GTDa and apply the same reasoning to relax the condition (<ref>) on B_x in Theorem <ref> for this algorithm. Recall the minimax problem (<ref>) associated with the biased variant of the single-time-scale GTDa discussed in Section <ref>. 
We can view the two-time-scale biased variant algorithm as a two-time-scale algorithm for solving the same minimax problem, but with the maximizer operating at the fast time-scale. In particular, for each θ, recall that the inner maximization problem appearing in (<ref>) is

    J̃_p^h (θ) : = sup_x ∈ B_xψ^o_h(θ, x) = sup_x ∈ B_x{ x^  ( A_h θ + b_h) -  12  x^ C x } + p(θ),

where A_h, b_h are given by (<ref>) and C = Φ^Ξ Φ.
This maximization problem has a unique optimal solution in {ϕ()} (because b_h ∈{ϕ()} and the column space of A_h is contained in {ϕ()}; cf. Section <ref>). Let us denote this optimal solution by x̃_h(θ).

 
The convex function J̃_p^h(θ) is differentiable with ∇J̃_p^h(θ) = ∇_θψ^o_h(θ, x̃_h(θ)), and x̃_h(θ) is a continuous function of θ.


The proof of the lemma is the same as that of Lemma <ref>, except that in showing that ∇_θψ^o_h(θ, x̂) is the same for all optimal solutions x̂ of (<ref>), we use the fact that the column space of A_h is contained in {ϕ()} (cf. the discussion immediately after (<ref>)).

Consider now the mean ODE (<ref>) associated with the fast time-scale of the biased variant of GTDa. The next lemma characterizes the limit set of this ODE, and it is a counterpart of Lemma <ref> when the condition (<ref>) on B_x has been removed. The proof is the same as that of Lemma <ref> (with x̃_h(θ) in place of x̃(θ), A_h θ + b_h in place of b_θ, and k̅_h(θ, x) = ∇_x ψ^o_h(θ, x) in place of k̅(θ, x) = ∇_x ψ^o(θ, x)).

 
For all initial x(0) ∈ B_x ∩{ϕ()} and θ(0) ∈ B_θ,
the limit set of the ODE (<ref>) is
{(θ, x̃_h(θ)) |θ∈ B_θ}.


Finally, we consider the mean ODE associated with the slow time-scale of the biased variant of GTDa. In view of Lemma <ref>, instead of the ODE (<ref>), this mean ODE is now given by

    θ̇(t) = g̅_h(θ(t), x̃_h(θ(t))) - ∇ p(θ(t))  + z(t),     z(t) ∈ - _B_θ(θ(t)),

where the function g̅_h(θ, x) is as defined in (<ref>), and z(t) is the boundary reflection term.
(The reasoning is the same as that given in Section <ref> when deriving (<ref>).)
Since ∇_θψ^o_h(θ, x̃_h(θ))  = - g̅_h(θ, x̃_h(θ)) + ∇ p(θ) (cf. (<ref>) and (<ref>)), by Lemma <ref>, the above ODE is the same as

    θ̇(t) = - ∇J̃_p^h(θ(t))  + z(t),     z(t) ∈ - _B_θ(θ(t)).

It corresponds to using gradient-descent to minimize J̃_p^h on B_θ. Hence the limit set of this ODE is just the set of optimal solutions of the minimization problem, D^h_θ = _θ∈ B_θJ̃_p^h(θ) (this follows from the same proof of Lemma <ref> with the function J̃_p^h in place of J). 
We now use the approximation property given in Lemma <ref> to relate the set D^h_θ to the set D_θ = _θ∈ B_θ{sup_x ψ^o(θ, x)}, like what we did in Section <ref>.  Lemma <ref> shows that for h ∈{ h_K | K > 0} (the family of functions defined earlier for the biased variant algorithm), given any ϵ > 0, we have D^h_K_θ⊂ N_ϵ (D_θ) for sufficiently large K (equivalently, when the bias introduced by the algorithm is sufficiently small). Therefore, when x_ opt∈int(B_x) so that D_θ = Θ_opt (Lemma <ref>), D^h_K_θ⊂ N_ϵ (D_θ) for all K sufficiently large. Consequently, Theorem <ref> remains valid for the two-time-scale biased variant of GTDa if we replace its condition on B_x by the weaker condition x_ opt∈int(B_x). This establishes the second part of Theorem <ref>.





§ CONVERGENCE ANALYSES III: ALMOST SURE CONVERGENCE
 

In this section we analyze the asymptotic behavior of the gradient-based TD algorithms with respect to a stronger mode of convergence: almost sure convergence.
We consider primarily those constrained algorithms discussed in the previous sections, for diminishing stepsizes that satisfy the square-summable condition ∑_n ≥ 0α^2_n < ∞ and ∑_n ≥ 0β^2_n < ∞, in addition to Assumption <ref>.
We analyze two-time-scale algorithms in Section <ref> and single-time-scale algorithms in Section <ref>.
The convergence results we obtained are more limited in the case of state-dependent λ, and more satisfactory in the case of history-dependent λ where the trace iterates {_n} are bounded by the choice of λ. 
Finally, in Section <ref>, we give a result on the unconstrained single-time-scale GTDa algorithm for the case of history-dependent λ. 

As discussed in Section <ref>, one can analyze single-time-scale GTDa with a scaling parameter η > 0 for the x-iterates by using the same arguments for the algorithm without such scaling. So for notational simplicity, we will focus on the latter case where η = 1 and give detailed arguments for this case only. The implications for the case with scaling are as explained at the end of Section <ref> for constrained GTDa and in Remark <ref>, to be given in Section <ref>, for unconstrained GTDa.

The results of this section are listed here for quick access:



  * two-time-scale GTD and MD-GTD with history-dependent λ: Theorems <ref> and <ref>;

  * single-time-scale GTDa with history-dependent λ: Theorem <ref>;

  * single-time-scale GTDa and its biased variant with state-dependent λ: Theorem <ref>;

  * single-time-scale unconstrained GTDa with history-dependent λ: Theorem <ref>.






 §.§ Two-Time-Scale GTD and MD-GTD
 

We consider the regularized objective function J_p(θ) = J(θ) + p(θ) as in Section <ref>, where p(·) is a convex differentiable function on ^d. Let 
Θ_ opt = _θ∈ B_θ J_p(θ).
Recall that the GTD algorithms for minimizing J_p are the same as given before except that there is an extra gradient term -α_n ∇ p(θ_n) in the θ-iteration (cf. Section <ref> or Section <ref> below).

For two-time-scale GTD/MD-GTD algorithms, we only have almost sure convergence results for the case of history-dependent λ (including the use of the composite scheme), where the trace iterates {_n} are kept bounded by the choice of the λ-parameters (cf. Condition <ref>(ii)). This, of course, includes the special case of state-dependent λ in which one bounds the traces by using sufficiently small λ.

Before stating our convergence theorems, we collect the conditions of the theorems in the two lists below, for the GTD and MD-GTD algorithms, respectively. For GTDa and GTDb, the conditions are the same except for the condition on the constraint set B_x, which can be made weaker for GTDa (cf. Section <ref>). The weaker condition, as we recall, involves the point x_ opt, which is the unique dual optimal solution of the minimax problem (<ref>) in {ϕ()}.

 
For the two-time-scale GTD algorithms:



  (i) Condition <ref> holds.

  (ii) The λ-parameters are set according to either (<ref>) or (<ref>), where the memory states satisfy Condition <ref>, and the function λ(·) in (<ref>) or each function λ^(i)(·), i ≤ℓ in (<ref>) satisfies Condition <ref>.

  (iii) The stepsizes {α_n} and {β_n} satisfy Assumption <ref> and the square-summable condition, ∑_n ≥ 0α^2_n < ∞, ∑_n ≥ 0β^2_n < ∞.

  (iv) In the case of GTDa, the constraint set B_x contains x_ opt in its interior; and in the case of GTDb, the constraint set B_x satisfies the condition (<ref>).

  (v) The initial x_0, _0 ∈{ϕ()}.




 
Let {(θ_n, x_n)} be generated by any of the two-time-scale GTD algorithms discussed in Sections <ref>, <ref> and <ref>.  
Then under Assumption <ref>, for each given initial condition, {θ_n} converges almost surely to the set Θ_ opt.



For the two-time-scale MD-GTD algorithms, we impose the same conditions on the constraint sets and on the minimization problem inf_θ J_p(θ) as in Section <ref>: 


 
For the two-time-scale MD-GTD algorithms:



  (i) The conditions in Assumption <ref>(i)-(iii) and (v) hold.

  (ii) The constraint sets D_θ^* and B_x satisfy Assumption <ref>.

  (iii) The minimization problem inf_θ J_p(θ) has a unique optimal solution θ_ opt.
 





 
Let {(θ_n, x_n)} be generated by any of the two-time-scale MD-GTD algorithms discussed in Section <ref>.
Then under Assumption <ref>, for each given initial condition, {θ_n} converges almost surely to θ_ opt.



In the rest of this subsection, we prove the two theorems above. 
The proof arguments for MD-GTD are essentially the same as those for GTD; we will explain this after we give the proof details for GTD.




  §.§.§ Convergence proof for GTD
 

In this subsection we prove Theorem <ref> for the GTD algorithms. As before, for notational simplicity, we will use GTDa to show how we carry out the analysis. Because the condition on the constraint set B_x is stronger for GTDb and weaker for GTDa, and because the proof for GTDb is essentially the same as that for GTDa under the stronger condition on B_x, we will analyze GTDa under both conditions.

Recall the two-time-scale GTDa algorithm for minimizing J_p(θ):

    θ_n+1    = Π_B_θ( θ_n + α_n  ρ_n (ϕ(S_n) - _n+1ϕ(S_n+1) ) ·_n^ x_n - α_n ∇ p(θ_n) ),  
       
       x_n+1    = Π_B_x( x_n + β_n (_n δ_n(v_θ_n) - ϕ(S_n) ϕ(S_n)^ x_n) ).
 
In terms of the functions g, k defined in (<ref>), we can write it as

    θ_n+1    = Π_B_θ( θ_n + α_n   g(θ_n, x_n, Z_n) - α_n ∇ p(θ_n)), 
    
          x_n+1    = Π_B_x( x_n + β_n (k(θ_n, x_n, Z_n) + _n ω_n+1 ) ),

where ω_n+1 = ρ_n (R_n+1 - r(S_n, S_n+1) ).
Similar to the proof given earlier, we first consider the fast time-scale determined by the stepsize {β_n}, and write the algorithm in the form of a single-time-scale algorithm as 

    ( [ θ_n+1; x_n+1 ]) = Π_B_θ× B_x( [ θ_n + β_n   (α_n / β_n) ·( g(θ_n, x_n, Z_n) - ∇ p(θ_n) );                 x_n + β_n ( k(θ_n, x_n, Z_n) + _n ω_n+1) ]).

We want to show that its average dynamics is characterized by the mean ODE:

    ( [ θ̇(t);  ẋ(t) ]) 
     =  ( [                      0; k̅(θ(t), x(t) ) + z(t) ]),     θ(0) ∈ B_θ,   z(t) ∈ - _B_x(x(t)),
 
where the function k̅(·) is given by (<ref>) as before, z(t) is the boundary reflection term. 

We then consider the slow time-scale. For each θ, let us define x̅(θ) and g̅(θ), according to the condition placed on the constraint set B_x:
 


  * Under the stronger condition (<ref>) on B_x, let x̅(θ) = x_θ and g̅(θ) = - ∇ J(θ) as in Section <ref> (cf. (<ref>)-(<ref>)).

  * Under the weaker condition x_opt∈int (B_x), let x̅(θ) = x̃(θ) and g̅(θ) = g̅(θ, x̃(θ)) as in Section <ref>.

With these definitions, we write the θ-iterates at the slow time-scale equivalently as

    θ_n+1    = Π_B_θ( θ_n + α_n (g(θ_n, x̅ (θ_n), Z_n) - ∇ p(θ_n) + Δ_n ) ),   
    where    Δ_n     = g(θ_n, x_n, Z_n) - g(θ_n, x̅ (θ_n), Z_n).

Treating Δ_n as noise terms, we want to show that the average dynamics of (<ref>) is characterized by the mean ODE:

    θ̇(t) = g̅ (θ(t)) - ∇ p ( θ(t))+ z(t),     z(t) ∈ - _B_θ(θ(t)),

where z(t) is the boundary reflection term.

The ODEs (<ref>) and (<ref>) above are the same mean ODEs for GTDa as discussed before in Sections <ref> and <ref>. However, because the notion of convergence here is different and stronger, to show that the average dynamics of the algorithm is characterized by these mean ODEs, we need to verify a different set of conditions, which are more demanding (the reason that we required the traces to be bounded).


Conditions to verify


We will apply <cit.> in our convergence proof. This theorem concerns single-time-scale stochastic approximation algorithms with “exogenous noises.” We will apply it twice, first to the fast-time-scale iterates (<ref>) and then to the slow-time-scale iterates (<ref>).
In order to apply <cit.>, we need to verify the conditions listed below. Let us first explain some notations in the conditions. The ODE-based analysis works with continuous-time processes that are piecewise linear or constant interpolations of the discrete-time iterates. There is a correspondence between the continuous-time and the discrete-time, which is defined by the stepsizes used in the iterations.
Imagine placing consecutive iterations at moments on the continuous timeline that are separated by the stepsizes used in these iterations.
With respect to the stepsizes {β_n}, define, for each t ≥ 0,

    m_β(t) : = min{ n |∑_i=0^nβ_i > t },       m_β^-(t) : = max{ 0, m_β(t) - 1}.

Thus m_β(t) points to the first iteration occurred after time t, and m_β^-(t) the latest iteration before time t has elapsed. 
Abusing notation, we also denote

    m_β(n,t) = min{ j ≥ n | ∑_i=n^jβ_i > t },      m_β^-(n,t) = max{ n, m_β(n,t) - 1}.

These are similar to the “iteration pointers” above, but work with a continuous timeline that has the n-th iteration placed at time 0.
The above pointers with the subscript β will appear in the conditions for the fast-time-scale iterates. 
Likewise, with respect to the stepsizes {α_n}, we define “iteration pointers” m_α(t), m_α^-(t), etc.These pointers will appear in the conditions for the slow-time-scale iterates. 
Note that for any τ≥ 0, we have ∑_i=m_β(τ)^m^-_β(τ+t)β_i ≤ t and ∑_i=m_α(τ)^m^-_α(τ + t)α_i ≤ t.

The following conditions correspond to the conditions (A.6.1.1) and (A.6.1.3)-(A.6.1.7) in <cit.>.
[In our case, its condition (A.6.1.2) simply requires the function k(θ, x, z) to be continuous in (θ, x) for each z, and the function g(θ, x̅ (θ), z) to be continuous in θ for each z, which are clearly satisfied.]



Conditions for the fast time-scale:



  (i) sup_n ≥ 0[  k(θ_n, x_n, Z_n) + _n ω_n+1] < ∞, 
sup_n ≥ 0[ (α_n/β_n) · g(θ_n, x_n, Z_n) - ∇ p(θ_n) ] < ∞.

  (ii) For each (θ, x) ∈ B_θ× B_x, η > 0 and T > 0,

    lim_n →∞{sup_j ≥ nmax_0 ≤ t ≤ T∑_i=m_β(jT)^m^-_β(jT+t)β_i ( k(θ, x, Z_i) - k̅(θ, x) ) ≥η} = 0,
 

    lim_n →∞{sup_j ≥ nmax_0 ≤ t ≤ T∑_i=m_β(jT)^m^-_β(jT+t)α_i  ( g(θ, x, Z_i) - ∇ p(θ) )  ≥η} = 0.
 

  (iii) For each η > 0 and T > 0,

    lim_n →∞{sup_j ≥ nmax_0 ≤ t ≤ T∑_i=m_β(jT)^m^-_β(jT+t)β_i  _i ω_i+1≥η} = 0.


  (iv) There are nonnegative measurable functions f_1(θ, x), f_2(z) such that k(θ, x, z)≤ f_1(θ,x) f_2(z) for all (θ, x) ∈ B_θ× B_x, where
f_1 is bounded on B_θ× B_x. For each η > 0,

    lim_t → 0lim_n →∞{sup_j ≥ n∑_i=m_β(jt)^m^-_β(jt+t)β_i f_2( Z_i) ≥η} = 0.
 

  (v) There are nonnegative measurable functions f_3(θ, x), f_4(z) such that 

    k(θ, x, z) - k(θ', x', z) ≤ f_3(θ - θ', x - x') f_4(z),   ∀  (θ, x), (θ', x') ∈ B_θ× B_x,
 
where f_3 is bounded on B_θ× B_x and f_3(θ, x) → 0 as (θ, x) → 0, and 

    {lim sup_n →∞∑_i=n^m_β(n,t)β_i f_4( Z_i) < ∞} = 1,   for some t > 0.



Conditions for the slow time-scale:



  (i) sup_n ≥ 0[  g(θ_n, x_n, Z_n) - ∇ p(θ_n) ] < ∞.

  (ii) For each θ∈ B_θ, η > 0 and T > 0,

    lim_n →∞{sup_j ≥ nmax_0 ≤ t ≤ T∑_i=m_α(jT)^m^-_α(jT+t)α_i ( g(θ, x̅(θ), Z_i) - g̅(θ) ) ≥η} = 0.
 

  (iii) For each η > 0 and T > 0,

    lim_n →∞{sup_j ≥ nmax_0 ≤ t ≤ T∑_i=m_α(jT)^m^-_α(jT+t)α_i ( g(θ_i, x_i, Z_i)  - g(θ_i, x̅(θ_i), Z_i)  ) ≥η} = 0.


  (iv) There are nonnegative measurable functions f_1(θ), f_2(z) such that 

    g(θ, x̅(θ), z) - ∇ p(θ) ≤ f_1(θ) f_2(z),   ∀ θ∈ B_θ,
 
where f_1 is bounded on B_θ. For each η > 0 and T > 0,

    lim_t → 0lim_n →∞{sup_j ≥ n∑_i=m_α(jt)^m^-_α(jt+t)α_i f_2( Z_i) ≥η} = 0.
 

  (v) There are nonnegative measurable functions f_3(θ), f_4(z) such that 

    g(θ, x̅(θ), z) - ∇ p(θ) - (g(θ', x̅(θ'), z) - ∇ p(θ') ) ≤ f_3(θ - θ') f_4(z),   ∀ θ, θ' ∈ B_θ,
 
where f_3 is bounded on B_θ and f_3(θ) → 0 as θ→ 0, and 

    {lim sup_n →∞∑_i=n^m_α(n, t)α_i f_4( Z_i) < ∞} = 1,   for some t > 0.




Most of the above conditions can be quickly verified. 
Because we restrict attention to the case of history-dependent λ, by our choice of the λ's (cf. Condition <ref>(ii)), {_n} lies in a pre-determined bounded set. Since the iterates (θ_n, x_n) are also confined in bounded sets, the conditions (i) for both time-scales hold obviously. 
Consider now the conditions (iv)-(v) for the fast time-scale.
By the definition of k(·) (cf. (<ref>)), we have for some constant c > 0,

    k(θ, x, z)    ≤  c   (θ +   x +  1)  · (    + 1)  , 
     k(θ, x, z) - k(θ', x', z)    ≤ c   (  θ - θ' + x - x'  ) · ( + 1),

so in these conditions, we can let f_1(θ, x) = θ +   x +  1, f_3(θ, x) = θ +   x, and 
f_2(z) = f_4(z) =c   (  + 1). 
Since the traces {_n} lie in a pre-determined bounded set, f_2(Z_i) and f_4(Z_i) are also bounded for all i. 
Then, in view of the fact that ∑_i=m_β(jt)^m^-_β(jt+t)β_i ≤ t and ∑_i=n^m_β(n,t)β_i ≤ t, it is trivially true that the conditions (iv)-(v) for the fast time-scale are satisfied. 

The conditions (iv)-(v) for the slow time-scale can be similarly checked. 
Using the definition of g(·) (cf. (<ref>)), we have that for some constant c > 0 and for all θ∈ B_θ,

    g(θ, x̅(θ), z) - ∇ p(θ) ≤ c   (x̅(θ)  + ∇ p(θ)  + 1) · ( + 1).
  
Correspondingly, in the condition (iv) for the slow time-scale, we let f_1(θ)=x̅(θ)  + ∇ p(θ)  + 1 and f_2(z) = c   ( + 1). 
Since {_n} lie in a pre-determined bounded set and ∑_i=m_α(jt)^m^-_α(jt+t)α_i ≤ t , we see that the condition (iv) holds trivially.

Consider now the condition (v) for the slow time-scale. 
Note that ∇ p(θ) is uniformly continuous on the compact set B_θ (since ∇ p(·) is continuous on ^d).
This implies that the function u_p: _+ →_+,  

    u_p(ϵ) : = sup_θ, θ' ∈ B_θ,  θ - θ'≤ϵ∇ p(θ) - ∇ p(θ')

satisfies that u_p(ϵ) → 0 as ϵ→ 0. Similarly, since x̅(θ) is continuous in θ,
[In the case x̅(θ) = x̃(θ), the continuity of x̅(·) is shown by Lemma <ref>. In the case x̅(θ) = x_θ, one can use a simpler argument to show that x̅(·) is Lipschitz continuous. In this case, x̅(θ) - x̅(θ') is the unique solution to the linear system of equations (in x):
x ∈{ϕ()},  Φ^Ξ Φ x = Φ^Ξ  ( - I) Φ (θ - θ') = 0.
Therefore, x̅(θ) - x̅(θ') _2 ≤Φ^Ξ  ( - I) Φ (θ - θ') _2 /c, where c is the smallest nonzero eigenvalue of the matrix Φ^Ξ Φ. From this inequality it follows that x̅(·) is Lipschitz continuous.]

it is uniformly continuous on B_x and hence the function u_x̅: _+ →_+,

    u_x̅(ϵ) : = sup_θ, θ' ∈ B_θ,  θ - θ'≤ϵx̅(θ) - x̅(θ')
 
satisfies that u_x̅(ϵ) → 0 as ϵ→ 0. We then have that for some constant c > 0 and for all θ, θ' ∈ B_θ,

    g(θ, x̅(θ), z) - ∇ p(θ) - (g(θ', x̅(θ'), z) - ∇ p(θ') )    ≤ g(θ, x̅(θ), z) - g(θ', x̅(θ'), z) 
           + ∇ p(θ) -  ∇ p(θ')  
       ≤ c  x̅(θ) - x̅(θ') · + u_p( θ - θ')  
       ≤ c (u_x̅(θ - θ' )  + u_p( θ - θ') ) · ( + 1).
 
Now, in the condition (v) for the slow time-scale, let f_3(θ) = u_x̅(θ) + u_p(θ) and f_4(z) =c  ( + 1).
We see that as required, f_3(θ) → 0 as θ→ 0, and moreover, since {_n} lie in a pre-determined bounded set and ∑_i=n^m_α(n,t)α_i ≤ t, the conditions (v) also holds trivially.

That the algorithm satisfies the condition (iii) for the fast time-scale is a consequence of the fact that {_i ω_i+1} are martingale differences with bounded variances and {β_i} are square-summable (see <cit.>).
Consider now (<ref>) in the condition (ii) for the fast time-scale. In view of the boundedness of {_n}, g(θ, x, Z_i) - ∇ p(θ) ≤ c for some constant c, so for any j ≥ n,

    max_0 ≤ t ≤ T∑_i=m_β(jT)^m^-_β(jT+t)α_i ( g(θ, x, Z_i) - ∇ p(θ)) ≤ c ∑_i=m_β(jT)^m^-_β(jT+T)α_i ≤ c ∑_i=m_β(jT)^m^-_β(jT+T)β_i ·α_i/β_i≤ c   T ·sup_i ≥ m_β(nT)α_i/β_i,

which converges to 0 as n →∞ (since α_i/β_i → 0 as i →∞). Therefore, (<ref>) is satisfied.

We now prove that the algorithm satisfies (<ref>) and (<ref>) in the conditions (ii) as well. This proof is longer and uses a result from <cit.> on the convergence of certain random probability measures. Let us first explain this result.

A result for verifying the “averaging conditions” (<ref>) and (<ref>):


Focus, for now, on the case of history-dependent λ under Assumption <ref> (the composite scheme of setting λ will be treated later using the results we are going to derive first). Define random variables μ^β,t_τ,  μ^α,t_τ, τ≥ 0, t > 0, that take values in the space of probability measures on  as follows: for Borel subsets D ⊂,

    μ^β,t_τ(D) = 1/∑_i=m_β(τ)^m^-_β(τ+t)β_i ∑_i=m_β(τ)^m^-_β(τ+t)β_i  (Z_i ∈ D),     μ^α,t_τ(D) = 1/∑_i=m_α(τ)^m^-_α(τ+t)α_i∑_i=m_α(τ)^m^-_α(τ+t)α_i  (Z_i ∈ D).

Recall that the Markov chain {Z_i} has a unique invariant probability measure (which is just the probability distribution of Z_0 in the stationary state-trace process).
The following lemma concerns the convergence of these random probability measures to the invariant probability measure of {Z_i} as τ→∞. It is implied by <cit.> and Theorem <ref>(i).
[A condition in <cit.> is that the set of random probability measures {μ^β,t_τ|τ≥ 0, t > 0 } is almost surely tight, which is trivially satisfied in our case since {Z_i} lie in a bounded set. It is also assumed in <cit.> that the stepsize β_n is eventually nonincreasing; however, this assumption is used only once in the proof, as remarked in <cit.>. Where this assumption is used, the proof argument given in <cit.> also goes through if one assumes instead that β_n satisfies lim_n →∞ sup_0 ≤ j ≤ m_n| β_n+jβ_n - 1 | = 0 for some m_n →∞. Thus we can include the latter assumption as an alternative in Lemma <ref>. By <cit.>, for each t > 0, any limit point of {μ^β,t_τ}_τ≥ 0 must be an invariant probability measure of the weak Feller Markov chain {Z_i}. Since {Z_i} has a unique invariant probability measure by Theorem <ref>(i), we obtain Lemma <ref>.]

In the lemma {β_n} is a generic stepsize sequence, so the lemma also applies to μ^α,t_τ.

 
Consider the case of history-dependent λ under Assumption <ref>. Let {β_n} be positive stepsizes such that ∑_n ≥ 0β_n = ∞, ∑_n ≥ 0β^2_n < ∞, and either {β_n} is eventually nonincreasing or for some sequence of integers m_n →∞, 
lim_n →∞ sup_0 ≤ j ≤ m_n| β_n+jβ_n - 1 | = 0.
Then for each t > 0, almost surely, as τ→∞, μ^β,t_τ converges weakly to the unique invariant probability measure of {Z_i}.


We use this lemma to establish the following result, which leads directly to the verification of the “averaging conditions” (<ref>) and (<ref>) for both time-scales.

 
Let the conditions in Lemma <ref> hold. Let f(z) be a vector-valued function on  that is continuous in the trace variable , and let f̅ = _ζ [ f(Z_0) ]. Then for each η > 0 and T > 0,

    lim_n →∞{sup_j ≥ nmax_0 ≤ t ≤ T∑_i=m_β(jT)^m^-_β(jT+t)β_i ( f(Z_i) - f̅ ) ≥η} = 0.
 



Fix η and T. Denote
Δ_j,t = ∑_i=m_β(jT)^m^-_β(jT+t) β_i ( f(Z_i) - f̅  ),
so (<ref>) is the same as

    lim_n →∞{sup_j ≥ nmax_0 ≤ t ≤ TΔ_j,t≥η} = 0.

Since {_n} lies in a pre-determined bounded set by our choice of history-dependent λ under Assumption <ref>, for all i, f(Z_i) - f̅≤ c for some constant c. 
Let us replace the range of t in (<ref>) by a finite set of values. Let m be a positive integer such that c  T/m ≤η/2. If |t_1 - t_2| ≤ T/m, then 
Δ_j, t_1 - Δ_j, t_2≤ c   T/m ≤η/2. Therefore, to prove (<ref>), it is sufficient to prove for the finite set of t, E_m={ i T/m | i = 1, …, m}, that

    lim_n →∞{sup_j ≥ nmax_ t ∈ E_mΔ_j,t≥η/2 } = 0.

By the Markov inequality,

    lim_n →∞{sup_j ≥ nmax_ t ∈ E_mΔ_j,t≥η/2 }≤lim_n →∞ (2/η) ·[ sup_j ≥ nmax_ t ∈ E_mΔ_j,t].

Since sup_j ≥ nmax_ t ∈ E_mΔ_j,t≤ c   T, the relation (<ref>) will follow from (<ref>) and the bounded convergence theorem if we show that
lim_n →∞sup_j ≥ nmax_ t ∈ E_mΔ_j,t = 0 a.s., i.e.,  

    lim sup_n →∞max_t ∈ E_mΔ_n,t = 0    a.s.

So let us prove (<ref>).
 
Denote by ζ' the unique invariant probability measure of {Z_i}. By Lemma <ref>, with probability 1, 
it holds for all t ∈ E_m that μ^β,t_τ converges weakly to ζ' as τ→∞. 
Since {Z_n} lie in a compact set by our choice of λ, the supports of all these probably measures are contained in the same compact set.
Then for the continuous function f(z), which is (component-wise) bounded on the compact set, we have, by the definition of weak convergence, that almost surely, for all t ∈ E_m,

    ∫ f(z) d μ^β,t_τ - ∫ f(z) dζ' → 0   as τ→∞.
  
We have ∫ f(z) dζ' = _ζ [ f(Z_0) ] = f̅. For τ_n = nT, by the definition of μ^β,t_τ_n,

    ∫ f(z) d μ^β,t_τ_n = ∑_i=m_β(nT)^m^-_β(nT+t)β_i f(Z_i)/∑_i=m_β(nT)^m^-_β(nT+t) β_i.

Then, since for all t ≤ T, ∑_i=m_β(nT)^m^-_β(nT+t)β_i ≤ T, 
we obtain from (<ref>) that, almost surely, for all t ∈ E_m,

    (∑_i=m_β(nT)^m^-_β(nT+t)β_i ) ·∫ f(z) d μ^β,t_τ_n - f̅ → 0   as n →∞,

which is the same as 

    Δ_n, t = ∑_i=m_β(nT)^m^-_β(nT+t)β_i ( f(Z_i) - f̅) → 0   as n →∞.

Consequently, max_t ∈ E_mΔ_n, t→ 0 as n →∞, almost surely. This proves (<ref>), from which the desired relation (<ref>) follows, as shown earlier.


We can now apply Prop. <ref> to show that the algorithm satisfies the required “averaging conditions” (<ref>)  and (<ref>). We will do this below, inside the end part of our proof for Theorem <ref>.

Completion of the proof of Theorem <ref>


In the rest of this proof of Theorem <ref>, we discuss separately the two cases of setting λ: set λ according to (<ref>), and set λ according to the composite scheme (<ref>). This is because the first case is simpler and provides the basis for treating the second case, as the analysis in Section <ref> already showed.

Case I: history-dependent λ's set according to (<ref>).
In this case, for each (θ, x), applying Prop. <ref> with f(·)=k(θ, x, ·) and stepsizes {β_n}, we see that the algorithm satisfies the averaging condition (<ref>) for the fast time-scale. For each θ, applying the proposition with f(·) =g(θ, x̅(θ), ·) and stepsizes {α_n}, we see that the algorithm satisfies the averaging condition (<ref>) for the slow time-scale. 
All the conditions required for the fast time-scale have now been verified. So we can apply <cit.> to the equivalent form (<ref>) of the algorithm at the fast time-scale, and conclude that almost surely, (θ_n, x_n) converges to the limit set of the ODE (<ref>) in B_θ× B_x for the initial condition x(0) ∈{ϕ()}. This limit set is {(θ, x̅(θ) ) |θ∈ B_θ}, as shown by Lemma <ref> if B_x satisfies the stronger condition (<ref>), and by Lemma <ref> otherwise.

We now consider the θ-iterates (<ref>) at the slow time-scale. Among the conditions listed above, we still need to verify the condition (<ref>) in (iii), which we do now. Using the almost sure convergence of (θ_n, x_n) to {(θ, x̅(θ) ) |θ∈ B_θ} just proved, and using also the uniform continuity of x̅(θ) on B_θ, we have x_n - x̅(θ_n)  0. Then, since {Z_n} lie in a bounded set and since for all z in that set and θ∈ B_θ, g(θ, x, z) is Lipschitz continuous in x uniformly w.r.t. (θ, z), there is a constant c such that for all i,

    g(θ_i, x_i, Z_i) - g(θ_i, x̅(θ_i), Z_i) ≤ c  x_i - x̅(θ_i)  0.

Consequently, 

    sup_j ≥ nmax_0 ≤ t ≤ T∑_i=m_α(jT)^m^-_α(jT+t)α_i ( g(θ_i, x_i, Z_i)  - g(θ_i, x̅(θ_i), Z_i)  ) ≤ c   T ·sup_i ≥ m_α(nT) x_i - x̅(θ_i)  0.
  
The l.h.s. above is bounded by some constant because {Z_n} and {(θ_n, x_n)} are all bounded. So, using the same reasoning that led to (<ref>) in the proof of Prop. <ref>, (<ref>) follows from the Markov inequality, the almost sure convergence to zero in (<ref>), and the bounded convergence theorem.

Thus all the conditions required for the slow time-scale are met. We can now apply <cit.> to the θ-iterates (<ref>) to conclude that almost surely, θ_n converges to the limit set of the ODE (<ref>) in B_θ. This limit set is Θ_opt, as we showed already in Section <ref> and Section <ref> for the two conditions on B_x. 
So we obtain the conclusions of Theorem <ref> in the case of history-dependent λ's set according to (<ref>), for the GTDa algorithm, as well as for the GTDb algorithm (since its proof under the stronger condition (<ref>) on B_x is essentially the same as given above).

Case II: history-dependent λ's set according to the composite scheme (<ref>). We proceed to verify (<ref>) and (<ref>) in the conditions (ii) for both time-scales. Recall from Section <ref> that in this case we have ℓ concurrent state-trace processes, { (S_n, y_n, _n^(q) )}_n ≥ 0, q ≤ℓ, each of which has a unique invariant probability measure ζ^(q). As showed in the proof of Lemma <ref>, we can decompose k(·) and g(·) as the sum of ℓ functions and write, for each (θ, x),

    k(θ, x, Z_n) = ∑_q=1^ℓ k^o(θ, x, Z_n^(q)),      g(θ, x, Z_n) = ∑_q = 1^ℓ g(θ, x, Z_n^(q)),
  
where k^o(·) is a continuous function and Z_n^(q) = (S_n, y_n, _n^(q), S_n+1), and we also have that

    k̅(θ, x)     = ∑_q=1^ℓk̅_q(θ, x),    where  k̅_q(θ, x) = _ζ^(q)[   k^o(θ, x, Z_0^(q)) ], 
    g̅(θ)     = ∑_q=1^ℓg̅_q(θ),     where  g̅_q(θ) = _ζ^(q)[   g(θ, x̅(θ), Z_0^(q)) ].

We now apply Prop. <ref> to the ℓ state-trace processes individually. For each q ≤ℓ, applying Prop. <ref> to the q-th process with f(·)=k^o(θ, x, ·) and stepsizes {β_n}, we obtain that for each η, T > 0,

    lim_n →∞{sup_j ≥ nmax_0 ≤ t ≤ T∑_i=m_β(jT)^m^-_β(jT+t)β_i ( k^o(θ, x, Z^(q)_i) - k̅^o_q(θ, x) ) ≥η} = 0,     ∀  q ≤ℓ;

and applying Prop. <ref> to the q-th process with f(·) = g(θ, x̅(θ), ·) and stepsizes {α_n}, we obtain that for each η, T > 0,

    lim_n →∞{sup_j ≥ nmax_0 ≤ t ≤ T∑_i=m_α(jT)^m^-_α(jT+t)α_i ( g(θ, x̅(θ), Z^(q)_i) - g̅_q(θ) ) ≥η} = 0,     ∀  q ≤ℓ.
 
Clearly, that (<ref>) and (<ref>) hold for each η, T > 0 imply, respectively, that the desired relations (<ref>) and (<ref>) also hold for each η, T > 0. We have thus verified the “averaging condition” (ii) for both time-scales. From this point on, the rest of the proof is the same as that given above for the first case of history-dependent λ. 
This completes our proof of Theorem <ref>.



  §.§.§ Convergence proof for MD-GTD


The proof of Theorem <ref> for MD-GTD follows the same steps and uses the same arguments as the proof for GTDa we just gave. So we will only briefly explain the proof outline.
In particular, take MD-GTDa as an example. 
The associated mean ODEs for the fast and slow time-scales are the same ODEs (<ref>) and (<ref>) discussed in Section <ref>. 
As in the previous GTDa case, we apply <cit.> twice to analyze the iterates {(θ^*_n, x_n)}, first to the fast time-scale and then to the slow time-scale, to establish that the average dynamics of {(θ^*_n, x_n)} at the fast time-scale and the average dynamics of {θ^*_n} at the slow time-scale are both characterized by the respective mean ODEs. Within this proof step, we first combine the conclusions obtained about the fast-time-scale iterates with the characterization of the limit set of the fast-time-scale ODE (<ref>) under the assumed condition on B_x. This leads to the conclusion that x_n - x_θ_n 0, where θ_n = ∇ψ^*(θ^*_n). We then use this in the analysis of the slow-time-scale θ^*-iterates, and by applying <cit.> again, we obtain a convergence result about the θ^*-iterates—in particular, their almost sure convergence to the limit set of the slow-time-scale ODE (<ref>). 
Then, as in Section <ref>, we translate this convergence of the θ^*-iterates to the convergence of the θ-iterates by using the relation (<ref>).

To apply <cit.> to MD-GTDa, we need to verify two sets of conditions for the fast and slow time-scales, similar to those given in the previous subsection for GTDa. The conditions are now on the (θ^*, x)-space instead of the (θ, x)-space, but they are essentially the same as those for GTDa, after we make the change of variable θ = ∇ψ^*(θ^*), similar to the case with the weak convergence proof method (cf. Footnote <ref> in Section <ref>). We therefore omit the details about the verification of these conditions.




 §.§ Single-Time-Scale GTDa and Biased GTDa
 

We consider now the single-time-scale GTDa algorithm (<ref>)-(<ref>) and its biased variant discussed in Section <ref>. We analyze separately the case with history-dependent λ, where the trace iterates lie in a pre-determined bounded set, and the case with state-dependent λ, where the trace iterates can be unbounded.
In the former case, we have more satisfactory results, obtained with an analysis similar to the one given in the previous subsection. 

 
For the single-time-scale GTDa algorithm:



  (i) Condition <ref> holds.

  (ii) The λ-parameters are set according to either (<ref>) or (<ref>), where the memory states satisfy Condition <ref>, and the function λ(·) in (<ref>) or each function λ^(i)(·), i ≤ℓ in (<ref>) satisfies Condition <ref>.

  (iii) The stepsizes {α_n} satisfy the conditions in Assumption <ref> for the fast-time-scale stepsizes, as well as the square-summable condition, ∑_n ≥ 0α^2_n < ∞.

  (iv) The initial x_0, _0 ∈{ϕ()}.



In the convergence theorem below, the definitions of the saddle points D_θ×{x̅} and Θ_ opt×{ x_ opt} are as in Section <ref>. Recall that Θ_ opt = _θ∈ B_θ{ J(θ) + p(θ) } and x_ opt is the unique dual optimal solution in {ϕ()} of the minimax problem (<ref>) (which places no constraints on the maximizer).

 
Let {(θ_n, x_n)} be generated by the single-time-scale GTDa algorithm (<ref>)-(<ref>).
Then under Assumption <ref>, for each given initial condition, {(θ_n, x_n)} converges almost surely to the subset D_θ×{x̅} of saddle points of the minimax problem (<ref>). If the constraint set B_x contains x_ opt in its interior, then D_θ×{x̅}=Θ_ opt×{ x_ opt}.


For the case of state-dependent λ (or the case of the composite scheme discussed in Section <ref> that involves state-dependent λ), in general, the trace iterates can be unbounded. Our convergence result for this case is limited: it is only for stepsizes α_n = O(1/n). (This is also the reason why we do not have a convergence theorem for the two-time-scale GTDa algorithm with state-dependent λ—if we let the stepsize be O(1/n) for the fast time-scale, we do not have a choice of stepsize left for the slow time-scale.)

 
For the single-time-scale GTDa algorithm and its biased variant:



  (i) The conditions in Assumption <ref>(i) and (iv) hold.

  (ii) The stepsizes {α_n} satisfy α_n = O(1/n) and (α_n - α_n+1)/α_n = O(1/n).



In the convergence theorem below, the family of functions { h_K | K > 0} for the biased GTDa algorithm are the same as in Theorem <ref>.

 
Consider the case of state-dependent λ. Let Assumption <ref> hold, and let {(θ_n, x_n)} be generated by the single-time-scale GTDa algorithm (<ref>)-(<ref>) or by its biased variant (<ref>)-(<ref>) with h ∈{ h_K | K > 0}. Then for each given initial condition, the following hold:
 


  (i) In the case of GTDa, {(θ_n, x_n)} converges almost surely to the subset D_θ×{x̅} of saddle points of the minimax problem (<ref>). 

  (ii) In the case of biased GTDa, for each ϵ > 0, there exists K_ϵ > 0 such that if h=h_K with K ≥ K_ϵ, then {(θ_n, x_n)} converges almost surely to N_ϵ ( D_θ×{x̅}).

If the constraint set B_x contains x_ opt in its interior, then D_θ×{x̅}=Θ_ opt×{ x_ opt} in the above.





  §.§.§ Convergence proofs


To prove the preceding convergence theorems, we will again apply <cit.>. The mean ODEs associated with the single-time-scale GTDa and its biased variant are, respectively, the ODEs (<ref>) and (<ref>), which we analyzed in Section <ref>. We will use <cit.> to obtain the conclusion that the iterates produced by each algorithm converge almost surely to the limit set of its associated ODE, thereby proving the convergence theorems above.

Thus, all we need to do is to show that the algorithms satisfy the conditions of <cit.> under the assumptions of each theorem. For the single-time-scale GTDa, we need to verify the five conditions listed below, which correspond to the conditions (A.6.1.1), (A.6.1.3)-(A.6.1.4), and (A.6.1.6)-(A.6.1.7) in <cit.>. 
In the conditions, the functions g, k, g̅, k̅ are as defined in (<ref>) and (<ref>).
The “iteration pointers” m_α(·), m^-_α(·), etc. are defined w.r.t. the stepsizes {α_n} as in Section <ref>.

The conditions for the biased GTDa variant are the same, except that the functions g, k are replaced respectively by the functions g_h, k_h (which are similar to g, k but with h() in place of ), and the functions g̅, k̅ are replaced respectively by the functions g̅_h, k̅_h defined in (<ref>), and finally, in the conditions (i) and (iii), _n ω_n+1 is replaced by h(_n)ω_n+1.

Conditions for GTDa:



  (i) sup_n ≥ 0[  g(θ_n, x_n, Z_n) - ∇ p(θ_n) ] < ∞, sup_n ≥ 0[  k(θ_n, x_n, Z_n) + _n ω_n+1] < ∞.

  (ii) For each (θ, x) ∈ B_θ× B_x, η > 0 and T > 0,

    lim_n →∞{sup_j ≥ nmax_0 ≤ t ≤ T∑_i=m_α(jT)^m^-_α(jT+t)α_i ( g(θ, x, Z_i) - g̅(θ, x) ) ≥η} = 0,
 

    lim_n →∞{sup_j ≥ nmax_0 ≤ t ≤ T∑_i=m_α(jT)^m^-_α(jT+t)α_i ( k(θ, x, Z_i) - k̅(θ, x) ) ≥η} = 0.
 

  (iii) For each η > 0 and T > 0,

    lim_n →∞{sup_j ≥ nmax_0 ≤ t ≤ T∑_i=m_α(jT)^m^-_α(jT+t)α_i  _i ω_i+1≥η} = 0.


  (iv) There are nonnegative functions f_1(θ, x) and f_2(z) such that 

    max{g(θ, x, z) - ∇ p(θ),  k(θ, x, z)}≤ f_1(θ,x) f_2(z),
  
where
f_1 is bounded on B_θ× B_x, and for each η > 0,

    lim_t → 0lim_n →∞{sup_j ≥ n∑_i=m_α(jt)^m^-_α(jt+t)α_i f_2( Z_i) ≥η} = 0.
 

  (v) There are nonnegative functions f_3(θ, x) and f_4(z) such that 

    g(θ, x, z) - ∇ p(θ) - ( g(θ', x', z) - ∇ p(θ') ) ≤ f_3(θ - θ', x - x') f_4(z),


    k(θ, x, z) - k(θ', x', z) ≤ f_3(θ - θ', x - x') f_4(z),

where f_3 is bounded on B_θ× B_x and f_3(θ, x) → 0 as (θ, x) → 0, and 

    {lim sup_n →∞∑_i=n^m_α(n,t)α_i f_4( Z_i) < ∞} = 1,   for some t > 0.




Given the analysis in the previous section, Theorem <ref> is evident:


Assumption <ref>(ii) ensures that the trace iterates {_n} lie in a pre-determined bounded set. It is then evident that in the previous Section <ref>, we have already verified essentially all the conditions listed above, when analyzing the two-time-scale GTDa under the same choice of the λ-parameters. To avoid repetition, we will only discuss the condition (ii) here. Since g(θ, x, z) and k(θ, x, z) are continuous functions, applying Prop. <ref> to each of them (for fixed (θ, x)) with stepsizes {α_n}, we see that the condition (ii) is satisfied when we set the λ-parameters according to (<ref>). If the composite scheme with history-dependent λ is used and the λ-parameters are set according to (<ref>), then we proceed as in the analysis of Case II in the last part of the proof for Theorem <ref>. In particular, we first decompose g(·) and k(·) as the sums of ℓ functions and then apply Prop. <ref> to each of the functions, before we combine the results (cf. the reasoning given in (<ref>)-(<ref>) in Section <ref>).



We now proceed to prove Theorem <ref>. It concerns the case of state-dependent λ. In this case, without conditions to guarantee the boundedness of {_n}, a major difficulty for the algorithms is in satisfying the “averaging condition” (ii). We can only show that this condition is met for stepsizes satisfying α_n = O(1/n) and (α_n - α_n+1)/α_n = O(1/n). The proof is given below. It utilizes the ergodicity property of the state-trace process given in Theorem <ref>(ii), and it is similar to the proof given in <cit.> and <cit.> for the constrained off-policy TD and ETD algorithms respectively.


We give the proof for GTDa only, as the proof for the biased variant is essentially the same, except that it involves the functions g_h, k_h, g̅_h, g̅_k instead of the functions g, k, g̅, k̅, as mentioned earlier.

We need to show that the five conditions listed above are all met. 
Similar to the derivations (<ref>)-(<ref>) and (<ref>)-(<ref>) in Section <ref>, 
in the conditions (iv)-(v), 
we can take 

    f_1(θ, x) =  x  + θ + ∇ p(θ) + 1,      f_3(θ, x) =  x  + θ + u_p(θ),

and take f_2(z), f_4(z) to be c   (  + 1) for some constant c. (Note that as required, f_3(θ, x) → 0 as (θ, x) → 0.)

Then, since the iterates (θ_n, x_n) lie in the bounded set B_θ× B_x, using the bound (<ref>), we see that the two requirements in the condition (i) are implied by 
sup_n ≥ 0 [ _n ] < ∞ and sup_n ≥ 0 [ _n ω_n+1 ] < ∞. The latter inequality is itself implied by sup_n ≥ 0 [ _n  ] < ∞, because given the history Z_i , R_i, i ≤ n, the noise term ω_n+1 has zero mean and finite variance which depends only on the state transition (S_n, S_n+1).
Since Prop. <ref>(ii) ensures sup_n ≥ 0 [ _n ] < ∞, we see that the condition (i) is met.

We now focus on proving (<ref>)-(<ref>) and (<ref>)-(<ref>) in the conditions listed above. 
Consider first (<ref>)-(<ref>) in the conditions (iv)-(v). 
Based on the remarks in <cit.>, to prove (<ref>), it is sufficient to show that with f̅_2 = _ζ [ f_2(Z_0) ],

    lim_n →∞{sup_j ≥ nmax_0 ≤ t ≤ T∑_i=m_α(jT)^m^-_α(jT+t)α_i ( f_2( Z_i) - f̅_2 ) ≥μ} = 0;

and to prove (<ref>), it is sufficient to prove (<ref>) with f_4 in place of f_2 and f̅_4 = _ζ [ f_4(Z_0) ] in place of f̅_2. The above condition (<ref>) has the same form as (<ref>)-(<ref>) in the conditions (ii)-(iii). (They are known as the asymptotic rate of change conditions, or the Kushner-Clark conditions; cf. <cit.>) By the examples in <cit.>, one sufficient condition for this type of assumptions is that a strong law of large numbers holds. 
In particular, let ψ_i represent any of the following terms involved in the conditions (ii)-(v): g(θ, x, Z_i) - g̅(θ, x), k(θ, x, Z_i) - k̅(θ, x), _i ω_i+1, f_2( Z_i) - f̅_2, and f_4( Z_i) - f̅_4. Then for each of the conditions to hold, it is sufficient that for the respective ψ_i's, we have α_n ∑_i=0^n-1ψ_i  0 and (α_n - α_n+1)/α_n = O(α_n).  

Since the functions g(θ, x, ·), k(θ, x, ·), f_2(·) and f_4(·) are Lipschitz continuous in the trace variable, by Theorem <ref>(ii), for stepsizes α_n = O(1/n), we have α_n ∑_i=0^n-1ψ_i  0 for the corresponding ψ_i's. Consequently the relations (<ref>)-(<ref>) and  (<ref>)-(<ref>) hold.
For (<ref>) in the condition (iii), the proof that 1/n∑_i=0^n-1_i ω_i+1 0 was given in the context of ETD in <cit.>, and the same proof applies here. 
Thus all the required conditions are met, so we can apply <cit.>, and using its conclusions, together with Prop. <ref> and Lemma <ref>, we then obtain the conclusions of the theorem.




 §.§ Unconstrained Single-Time-Scale GTDa
 

Finally, we consider the unconstrained single-time-scale GTDa algorithm that uses the minimax approach to solve the problem inf_θ J_p(θ), for the regularized convex, differentiable objective function J_p(θ) = J(θ) + p(θ) as before. The algorithm is given by 

    θ_n+1    = θ_n + α_n  ρ_n (ϕ(S_n) - _n+1ϕ(S_n+1) ) ·_n^ x_n  - α_n ∇ p(θ_n),  
    
       x_n+1    =  x_n + α_n (_n δ_n(v_θ_n) - ϕ(S_n) ϕ(S_n)^ x_n).

For unconstrained algorithms, keeping the iterates bounded is a difficulty in theory as well as in practice. We will impose stronger conditions on the above algorithm than those 
on the constrained GTDa in the previous subsection. We shall consider only the case of history-dependent λ (including the composite scheme with history-dependent λ), where the traces {_n} lie in a pre-determined bounded set by the choice of λ. To obtain a convergence theorem, we will apply stability and convergence results from <cit.>, and to this end, we shall also impose stronger conditions on the function p(·) in the regularized objective function.

[Conditions on the objective function]  



  (a) The function p(·) is convex and differentiable, and its gradient mapping ∇ p(·) is Lipschitz continuous. Moreover, there is a convex differentiable function p_∞(·) such that for each θ, lim_a →∞ p(a θ)/a^2 = p_∞(θ). 

  (b) The set of optimal solutions to inf_θ J_p(θ) is nonempty.

  (c) The origin is the unique optimal solution of inf_θ{12Π_ξ ( - I) v_θ_ξ^2 + p_∞(θ) }. 

  (c') Alternative to (c), assume that ∇ p(·) is such that ∇ p(θ) ∈{ϕ()} for all θ∈{ϕ()}, and that the origin is the uniquq optimal solution of inf_θ{12Π_ξ ( - I) v_θ_ξ^2 + p_∞(θ) } in {ϕ()}. 



We remark that Assumption <ref>(a) is especially strong. The requirement on ∇ p(·) being Lipschitz continuous rules out regularizers such as p(θ) = ∑_j=1^d | θ_j|^c, c ∈ (1, 2), which can be useful if sparse solutions are preferred. The requirement on the convergence to p_∞ also seems strong, since if ∇ p(·) is Lipschitz continuous, then for all a > 0, the functions p(a θ)/a^2 are pointwise bounded and there always exists a sequence of them converging to a finite convex function. It may be possible to relax this requirement if one can make sure that the proof arguments in <cit.> still go through under weakened conditions.
We also note that it can be shown that Assumption <ref>(a) and (c) imply Assumption <ref>(b). However, the latter is a more natural assumption on the problem that the unconstrained algorithm aims to solve, since p_∞ is introduced only for the purpose of analysis.

The next two lemmas are two implications of the preceding assumptions. They concern the optimal solutions of the minimization problem inf_θ J_p(θ) and the saddle points of its equivalent minimax problem. We will need them to state our convergence result.

 
Under Assumption <ref>(a)-(c), the set of optimal solutions of inf_θ J_p(θ) is compact. If instead of Assumption <ref>(c), Assumption <ref>(c') holds, then the subset of those optimal solutions of  inf_θ J_p(θ) in {ϕ()} is nonempty and compact.



Proof by contradiction. Suppose that Assumption <ref>(a)-(c) hold and yet inf_θ J_p(θ) has an unbounded (closed) set Θ_ opt of optimal solutions. 
Let θ_i ∈Θ_ opt for i ≥ 0, with a_i:= θ_i _2 → + ∞ as i → +∞. 
Recall 

    J_p(θ) = 12Π_ξ ( v_θ - v_θ ) _ξ^2 + p(θ)  =  12Π_ξ (  - I) Φθ  + Π_ξ r_π^(λ)_ξ^2 + p(θ).

By the optimality of the θ_i's, J_p(θ_i) ≤ J_p(θ) for all θ∈^d. 
Therefore, with θ̃_i = θ_i/a_i = θ_i / θ_i_2, we have

    1/a_i^2 J_p(a_i θ̃_i)  =   1/a_i^2 J_p(θ_i) 
          =   12Π_ξ (  - I) Φθ̃_i  + 1a_iΠ_ξ r_π^(λ)_ξ^2 + 1/a_i^2 p(a_i θ̃_i) 
       ≤1/a_i^2  J_p(a_i θ)

for all θ∈^d.
By choosing a subsequence if necessary, we can suppose that the sequence {θ̃_i} converges to a point θ̃_∞ on the unit ball. 
Then, take i → +∞ in both sides of (<ref>), and we have, by Assumption <ref>(a), that 

    lim_i →∞1/a_i^2 J_p(a_i θ̃_i)      =  12Π_ξ (  - I) Φθ̃_∞_ξ^2 + p_∞(θ̃_∞)  
       ≤12Π_ξ (  - I) Φθ_ξ^2 + p_∞(θ)

for all θ∈^d, where, to derive the inequality (<ref>), we used the pointwise convergence of the convex functions 1/a_i^2 p(a_i  ·) to p_∞(·), and to derive the first equality, we used the fact that this pointwise convergence of 1/a_i^2 p(a_i  ·) to p_∞(·) implies that the convergence is uniform on a neighborhood of θ̃_∞ <cit.>. 
The inequality (<ref>) implies that the point θ̃_∞≠0 is an optimal solution of inf_θ{12Π_ξ ( - I) v_θ_ξ^2 + p_∞(θ) }. 
This contradicts Assumption <ref>(c), under which the latter optimization problem has the origin as its unique optimal solution. Therefore, the set of optimal solutions to inf_θ J_p(θ) must be compact. 

For the second part of the lemma, note that if ∇ p(θ) ∈{ϕ()} for all θ∈{ϕ()}, then ∇ J_p(θ) ∈{ϕ()} for all θ∈{ϕ()}. This implies that the subset Θ̃_ opt of those optimal solutions of inf_θ J_p(θ) in {ϕ()} is nonempty.
[To see this, let θ_opt be an optimal solution of inf_θ J_p(θ), and let θ̅ be the projection of  θ_opt on {ϕ()}. Then by the convexity of J_p, J_p(θ_opt) ≥ J_p(θ̅) + ⟨∇ J_p(θ̅), θ_opt - θ̅⟩ = J_p(θ̅), implying that θ̅ is also an optimal solution.] 

Now apply the preceding proof argument with the set Θ̃_ opt in place of Θ_ opt, and it follows that Θ̃_ opt must be compact.


Recall from the discussion in Section <ref> that the problem inf_θ J_p(θ) can be equivalently written as the minimax problem 

    inf_θsup_x ψ^o(θ, x) = inf_θsup_x ∈{ϕ() }ψ^o(θ, x)

for the convex-concave function ψ^o(θ, x)  = ⟨Φ x,  v_θ - v_θ⟩_ξ - 12Φ x ^2_ξ  + p(θ) (cf. (<ref>)),
which is constant over the subspace {ϕ()}^⊥ for each θ. Let Θ_ opt = _θ J_p(θ).


 
Under Assumption <ref>(b), the maximization problem sup_x {inf_θψ^o(θ, x)} has a unique optimal solution x_ opt in {ϕ()}, and the set of saddle points of inf_θsup_x ∈{ϕ() }ψ^o(θ, x) is Θ_ opt×{x_ opt}.



By the definition of a saddle point, (θ̅, x̅) is a saddle point of inf_θsup_x ψ^o(θ, x) if and only if ∇ψ^o(θ̅, x̅) = (0, 0).
The latter equation is the same as

    ( Φ^Ξ  (- I) Φ)^x̅ + ∇ p(θ̅) = 0,     Φ^Ξ Φx̅ = Φ^Ξ  ( v_θ̅ - v_θ̅).
 
Consider any θ̅∈Θ_ opt (which is nonempty by our assumption). The second equation in (<ref>) has a unique solution x_θ̅ in {ϕ()}, as discussed earlier (cf. (<ref>)-(<ref>)). 
By the optimality of θ̅ for inf_θ J_p(θ), ∇ J_p(θ̅) = ∇ J(θ̅) + ∇ p(θ̅) = 0, 
and then using the expression (<ref>) of ∇ J(θ̅), we see that (θ̅, x_θ̅) satisfies the first equation in (<ref>). Thus (θ̅, x_θ̅) is a saddle point. It then follows from <cit.> that inf_θsup_x ψ^o(θ, x) has a finite saddle-value and its saddle points are the points in Θ_ opt× X_ opt, where X_ opt = _x {inf_θψ^o(θ, x)} is the set of dual optimal solutions.
Obviously, since in the preceding proof the point x_θ̅∈{ϕ()}, X_ opt∩{ϕ()}≠∅. Suppose we can find in this set two different points x_1, x_2. Then for any θ̅∈Θ_ opt, since both (θ̅, x_1) and (θ̅, x_2) are saddle points, they must both satisfy (<ref>). But the second equation in (<ref>) has a unique solution in {ϕ()} for each θ̅, a contradiction. Therefore, X_ opt∩{ϕ()} must be a singleton.


Let Θ̃_ opt = _θ J_p(θ)  ∩ {ϕ()} if Assumption <ref>(a)-(b) and (c') hold; it is nonempty and compact by Lemma <ref>. By the preceding Lemma <ref>, 
Θ̃_ opt×{x_ opt} is the subset of those saddle points in {ϕ()}×{ϕ()} for the minimax problem inf_θsup_x ψ^o(θ, x).
 
Here is our convergence result for the unconstrained GTDa:

 
Let {(θ_n, x_n)} be generated by the unconstrained single-time-scale GTDa algorithm (<ref>)-(<ref>).
Then under Assumption <ref> and Assumption <ref>(a)-(c), for each given initial condition, {(θ_n, x_n)} converges almost surely to the set Θ_ opt×{ x_ opt}. If instead of Assumption <ref>(c), Assumption <ref>(c') holds, then with θ_0 ∈{ϕ()},  {(θ_n, x_n)} converges almost surely to the set Θ̃_ opt×{ x_ opt}.



  

A version of Theorem <ref> also holds for the unconstrained GTDa algorithm (<ref>)-(<ref>) or equivalently, (<ref>)-(<ref>), which uses a scaling parameter η > 0. In this case, the conditions are the same as stated by Theorem <ref>. In the conclusion, we only need to replace the minimax problem inf_θsup_x ψ^o(θ, x) by the minimax problem corresponding to η, which is 

    inf_θ∈^dsup_x̃∈^d{ a  ⟨Φx̃,   v_θ - v_θ⟩_ξ - a^2/2Φx̃^2_ξ  + p(θ) },   for  a = √(η),
 
and use its unique dual optimal solution x̃_opt in the subspace {ϕ()} in replace of the point x_opt. The convergence conclusion for (<ref>)-(<ref>) then translates to that for (<ref>)-(<ref>) via the correspondence x = √(η) x̃. The reasoning is essentially the same as that given in Section <ref> for the constrained GTDa, which shows why the case η > 0 can be treated by the same line of analysis for η = 1. In particular, the derivation of the preceding minimax problem (<ref>) is similar to that of (<ref>), and since the two minimax problems, (<ref>) and inf_θsup_x ψ^o(θ, x), have the same structure, the arguments we give in this subsection for proving Theorem <ref> carry over to the general case η > 0.
 



  §.§.§ Convergence proof


We apply the “natural averaging” argument given in <cit.> to prove Theorem <ref>.
Assumption <ref> is indeed introduced for this purpose. Let us derive more implications of it, which will be needed to show that the conditions required by the analysis in <cit.> are satisfied.

First, consider the desired mean ODE for the unconstrained GTDa algorithm (<ref>)-(<ref>),

    θ̇(t) = g̅ (θ(t), x(t)) - ∇ p(θ(t)),     ẋ(t) = k̅ (θ(t), x(t)),

where the functions g̅, k̅ are defined as in (<ref>):

    g̅(θ, x) = ( Φ^Ξ  (I - ) Φ)^ x,     ̨̅(θ, x) = Φ^Ξ  ( v_θ - v_θ) - Φ^Ξ Φ  x.

Note that, as before, from any initial x(0) ∈{ϕ()}, the x-component of the solution of (<ref>) lies entirely in {ϕ()}.
Note also that if, as assumed in Assumption <ref>(c'), ∇ p(θ) ∈{Φ()} for all θ∈{Φ()}, then from any initial θ(0) ∈{Φ()}, the θ-component of the solution of (<ref>) also lies entirely in {ϕ()}.
We can use the same analysis given in Section <ref> for the constrained GTDa to show that the above ODE has the following solution properties:

  

 

  (i) Under Assumption <ref>(a)-(c), consider the ODE (<ref>) on the space ^d ×{ϕ()}. Then w.r.t. the latter space, the set Θ_ opt×{ x_ opt} is globally asymptotically stable.

  (ii) Under Assumption <ref>(a)-(b) and (c'), consider the ODE (<ref>) on the space {ϕ()}×{ϕ()}. Then w.r.t. the latter space, the set Θ̃_ opt×{ x_ opt} is globally asymptotically stable.

 
 

Recall the solution properties of differential inclusion with a maximal monotone mapping  given by <cit.> and described in Section <ref>. We apply them here with  being the maximal monotone mapping (θ, x) ↦(∇_θψ^o(θ, x), - ∇_x ψ^o(θ, x) ). 
With y(t) = (θ(t), x(t)), the ODE (<ref>) is the same as ẏ(t) = - (y(t)). For any θ̅∈Θ_ opt and x̅ = x_ opt, y(·) ≡ (θ̅, x̅) is a solution of (<ref>). 
Then by the solution property (iii) for differential inclusion with  described in Section <ref>, for any ϵ > 0, starting from the ϵ-neighborhood of Θ_ opt×{ x_ opt}, the solutions of (<ref>) remain in that same neighborhood. 

Now consider an arbitrary solution y̅(·) of (<ref>) with the initial y̅(0) ∈^d ×{ϕ()}.
Define a set D ⊂^d ×{ϕ()} by

    D = { (θ, x)  |  dist( (θ, x),  Θ_ opt×{ x_ opt}) ≤dist( y̅(0),  Θ_ opt×{ x_ opt})  } ∩( ^d ×{ϕ()}).

Note that D is compact by Lemma <ref>, and any solution of (<ref>) starting from D remains in D. 
Then, as t → + ∞, y̅(t) must converge to the set ∩_t̅≥ 0 { y(t) | y(0) ∈ D, t ≥t̅ }, which is the limit set of the ODE (<ref>) for all initial conditions in D. 

We now apply the same proof arguments given in Section <ref> for analyzing the limit set of the ODE associated with the constrained GTDa. In particular, with the set D in place of the constraint set B_θ× B_x, Lemma <ref> and its proof remain valid for the case here, and Lemma <ref> and Cor. <ref> then also hold for the ODE (<ref>) here. In turn, the proof of Prop. <ref> (which uses Cor. <ref>, as we recall) also applies here, if we now take the sets B_θ and B_x appearing in that proof to be the entire space ^d, and let all the boundary reflection terms of the ODE be zero in that proof. The result of applying the proof of Prop. <ref> here is that the limit set ∩_t̅≥ 0 { y(t) | y(0) ∈ D, t ≥t̅ } = Θ_ opt×{ x_ opt}, analogous to Prop. <ref>. This shows that any solution y̅(t) must converge to Θ_ opt×{ x_ opt} as t → +∞. So w.r.t. the space ^d ×{ϕ()}, the set Θ_ opt×{ x_ opt} is globally asymptotically stable.

The second part of the lemma follows from the same argument, with {ϕ()}×{ϕ()} in place of the space ^d ×{ϕ()}.
 
 

Consider now the minimization problem inf_θ{12Π_ξ ( - I) v_θ_ξ^2 + p_∞(θ) } appearing in Assumption <ref>. 
The same reasoning given before (<ref>) in Section <ref> shows that it is equivalent to the minimax problem

    inf_θsup_x ψ_∞(θ, x),     where ψ_∞(θ, x) : = ⟨Φ x,  ( - I) v_θ⟩_ξ - 12Φ x ^2_ξ  + p_∞(θ).
 
Analogously to (<ref>), consider the following ODE for solving this minimax problem:

    θ̇(t)  = - ∇_θψ_∞(θ(t), x(t)),     ẋ(t) =  ∇_x ψ_∞(θ(t), x(t)),

where

    ∇_θψ_∞(θ, x) =  ( Φ^Ξ  ( - I) Φ)^ x + ∇ p_∞(θ),     ∇_x ψ_∞(θ, x) =   Φ^Ξ  ( - I) v_θ - Φ^Ξ Φ  x.


We need the following two implications of Assumption <ref>, which will be used in establishing the boundedness of the iterates of the unconstrained GTDa algorithm.

 
Under Assumption <ref>(a), for each (θ, x),

    lim_a → +∞1a g̅(aθ, a x) - 1a ∇ p(a θ) = - ∇_θψ_∞(θ, x),     lim_a → +∞1a k̅(aθ, a x) =  ∇_x ψ_∞(θ, x),
 
and the convergence is uniform on any compact set of (θ, x).



It is clear that 1a g̅(aθ, a x) = ( Φ^Ξ  (I - ) Φ)^ x for any a > 0, and 
lim_a → +∞1a k̅(aθ, a x) = Φ^Ξ  ( - I) v_θ - Φ^Ξ Φ  x = ∇_x ψ_∞(θ, x) and the convergence is uniform on any compact set of (θ, x). 
Now note that 1a ∇ p(a θ) is the gradient of the convex function 1a^2  p(a θ) at θ. 
By Assumption <ref>(a), as a → + ∞, the function 1a^2  p(a θ) converges pointwise to the convex differentiable function p_∞(θ).
Then by <cit.>, as a → + ∞, the gradients 1a ∇ p(a θ) of 1a^2  p(a θ) converge to ∇ p_∞(θ) uniformly on any compact set of θ.
The lemma then follows.


 
Let Assumption <ref>(a) hold in what follows.

 

  (i) Under Assumption <ref>(c) or (c'), the point x = 0 is the unique dual optimal solution in {ϕ()} for the minimax problem (<ref>), and the point (θ, x) = (0,0) is its saddle point. 

  (ii) If Assumption <ref>(c) (respectively, (c')) holds, then for the ODE (<ref>) on the space ^d ×{ϕ()} (respectively, the space {ϕ()}×{ϕ()}), the point (θ, x) = (0,0) is globally asymptotically stable.




The first part follows from the same proof of Lemma <ref> with the function ψ_∞ in place of ψ^o and with the point θ_ opt = 0 in place of the set Θ_ opt. 

For the second part, the case of Assumption <ref>(c) follows from the proof of Lemma <ref> with ψ_∞ in place of ψ^o and with the point (θ_ opt, x_ opt) = (0,0) in place of the set Θ_ opt×{ x_ opt}.
For the case of Assumption <ref>(c'), note first that 1a ∇ p(a θ) →∇ p_∞ (θ) as shown in the proof of Lemma <ref>. Then, since ∇ p(a θ) ∈{ϕ()} for all θ∈{ϕ()} under Assumption <ref>(c'), we have ∇ p_∞ (θ) ∈{ϕ()} for all θ∈{ϕ()}. 
The desired conclusion then follows from the first part of the proof of Lemma <ref>, with ψ_∞ in place of ψ^o, with the point (θ_ opt, x_ opt) = (0,0) in place of the set Θ_ opt×{ x_ opt}, and with the space 
{ϕ()}×{ϕ()} in place of the space ^d ×{ϕ()}.







Finally, we are ready to prove Theorem <ref>:


First, let us note the space we work with in the proof. In the case where Assumption <ref>(c) or (c') holds, we restrict attention to the space ^d ×{ϕ()} or the space {ϕ()}×{ϕ()}, respectively. Recall that in each of these two cases, it is ensured by our assumption on the initial condition x_0, _0 and θ_0 for the algorithm that all the iterates {(θ_n, x_n)} lie in the corresponding space ^d ×{ϕ()} or {ϕ()}×{ϕ()}.

Under Assumption <ref>, by Prop. <ref>(i), the process {Z_n} is an ergodic weak Feller Markov chain with a unique invariant probability measure. To prove the theorem, we will apply the “natural averaging” results of <cit.> for the “uncontrolled” case, which involves an ergodic weak Feller Markov chain whose evolution is not affected by the (θ, x)-iterates of the algorithm. Under our Assumption <ref>(ii), the traces {_n} lie in a pre-determined compact set by the choice of λ. With the trace component  of z restricted to this compact set, the continuous functions g(θ, x, z), k(θ, x, z) in our GTDa algorithm are Lipschitz continuous in (θ, x) uniformly w.r.t. z. By Assumption <ref>(i), the gradient mapping ∇ p(θ) is Lipschitz continuous. These two facts together fulfill the condition on the algorithm in <cit.>. 

With the boundedness of {_n} just mentioned, it also holds trivially that the set of random probability measures μ_τ^α, t, t ≥ 0 defined before Lemma <ref> in Section <ref> is tight for each τ≥ 0. This fulfills one major assumption in the analysis in <cit.> (cf. page 71 therein). The second major assumption is for proving that the iterates of the algorithm are almost surely bounded, and it is the condition in <cit.>. This condition is the same as the properties stated in our Lemma <ref> and Lemma <ref>(ii) above, and which is thus fulfilled by these lemmas. 

Then, applying <cit.>, we obtain that the iterates (θ_n, x_n) are almost surely bounded, and in turn, applying <cit.>, we obtain that { (θ_n, x_n) } converges almost surely to an internally chain transitive invariant set of the ODE (<ref>), for the space ^d ×{ϕ()} or the space {ϕ()}×{ϕ()} that we work with (which depends on whether Assumption <ref>(c) or (c') holds, as noted at the beginning of the proof). Finally, combining this conclusion with Lemma <ref>, we obtain the conclusions of Theorem <ref>.




§ DISCUSSION
 

We have presented a collection of convergence results for a number of gradient-based off-policy TD algorithms, including the two-time-scale GTD and mirror-descent GTD algorithms, the single-time-scale GTDa algorithm, as well as the biased variants of these algorithms. For slowly diminishing stepsize and constant stepsize, using the weak convergence method, we have shown that w.r.t. a sense of convergence that is weaker than the almost sure convergence, all these algorithms have comparable, satisfactory convergence properties, whether they employ state-dependent or history-dependent λ. 
For the standard type of diminishing stepsize, however, concerning the almost sure convergence of the algorithms, our results are more satisfactory  in the case of history-dependent λ where the traces are bounded, and more limited in the case of state-dependent λ (including constant λ) where the traces can be unbounded. We believe this is not entirely an artifact of the analysis: It could be that with general state-dependent (or constant) λ, in certain situations, the algorithms need not converge almost surely under the standard diminishing stepsize condition, whereas in some other situations, our current results can be strengthened. Further investigation is needed in order to clarify this issue.

In this paper we have focused on one case of history-dependent λ where the way λ(·) and memory states are defined ensures that the state-trace process has a unique invariant probability measure. Another worthwhile subject for future research, as mentioned in the introduction, is to use stochastic approximation theory for differential inclusions to study the asymptotic behavior of the TD algorithms when the λ-parameters are chosen in a history-dependent way that is more flexible than the one we considered and when the state-trace process can have multiple invariant probability measures.





§ ACKNOWLEDGEMENT

I thank Professor Richard Sutton for helpful discussion. This research was supported by a grant from Alberta Innovates—Technology Futures.


tocsectionReferences
apa

911 





