Optimal Impulse Control of Dynamical Systems
Alexey Piunovskiy Alexander Plakhov Delfim F. M. Torres Yi Zhang





Optimal Impulse  Control of Dynamical Systems
    
Alexey Piunovskiy 

Department of Mathematical Sciences, University of Liverpool, L69 7ZL, UK.
 
  

Alexander Plakhov

Center for R&D in Mathematics and Applications, Department of Mathematics,
 University of Aveiro, 3810-193, Portugal

and Institute for Information Transmission Problems, Moscow, 127051, Russia
 
  

Delfim F. M. Torres

Center for R&D in Mathematics and Applications, Department of Mathematics,
 University of Aveiro, 3810-193, Portugal
  
  

Yi Zhang

Department of Mathematical Sciences, University of Liverpool, L69 7ZL, UK.



    
=============================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================




Using the tools of the Markov Decision Processes, we justify the dynamic programming approach to the optimal impulse control of deterministic dynamical systems. We prove the equivalence of the integral and differential forms of the optimality equation.  The theory is illustrated by an example from mathematical epidemiology. The developed methods can be also useful for the study of piecewise deterministic Markov processes.



  K̄ēȳw̄ōr̄d̄s̄:̄
Dynamical System, Impulse Control, Total Cost, Discounted Cost, 
 Randomized Strategy, Piecewise Deterministic Markov Process

 AMS 2000 subject classification: P̄rimary 49N25; Secondary 90C40.






§ INTRODUCTION


Impulse control of various dynamical systems attracts attention of many researchers: <cit.>, to mention the most relevant and the most recent works. The underlying system can be described in terms of ordinary <cit.> or stochastic <cit.> differential equations; that was an abstract Markov process in <cit.>. In <cit.>, along with the given deterministic drift, there are spontaneous (or natural) Markov jumps of the state. Such models are called Piecewise Deterministic Markov Processes (PDMP); the drift is usually described by a fixed flow. On the other hand, if there is no drift and the trajectories are piecewise constant, the model is called  Continuous-Time Markov Decision Process (MDP) <cit.>. By the gradual control we mean that only the local characteristics of the underlying process are under control. In case of PDMP, it means that the deterministic drift and the rate of the spontaneous/natural jumps, as well as the post-jump distribution are under control. But the impulse control means the following: at particular discrete time moments, the decision maker decides to intervene by instantaneously moving the process to some new point in the state space; that new point may be also random. Then, restarting at this new point, the process runs until the next intervention and so on. Sometimes, such control is called `singular control' <cit.>. The goal is to minimize the total (expected) accumulated cost which may be discounted <cit.> or not <cit.>. The case of long-run average cost was also studied in e.g. <cit.>.

The most popular method of attack to such problems is Dynamic Programming <cit.>. In <cit.>, versions of the Pontryagin Maximum Principle is used. In <cit.>, the impulse control is firstly reformulated as the linear program on measures: impulses correspond to the singular, Dirac components. After that, the numerical approximate scheme is developed in the form of Linear Matrix Inequalities.

Impulse control theory is widely applied to different real-life problems: epidemiology <cit.>, Internet congestion control <cit.>, reliability <cit.>, economic and finance <cit.>, moving objects <cit.>, medicine <cit.>,  genetics and ecology <cit.> etc.

The distinguishing features of the current work are as follows.


  * We consider the purely deterministic positive model with the total cost. As is known and explained in the text, the discounted model is a special case, as well as the absorbing model.

  * The imposed conditions partially overlap with those introduced in other articles. Generally speaking, our conditions are weaker than the assumptions introduced in the cited literature.

  * For the model under study, we demonstrate the new method to obtain the optimality equation in the integral form and to develop the corresponding successive approximations. This method is based on the well known tools for Discrete-Time MDP.

  * Under mild conditions, we prove the equivalence of the optimality (Bellman) equation in the integral and differential form. The analytical proof is new. Moreover, as mentioned in Conclusion, this proof remains valid also for the more general case of PDMP. Note also that the differential form is slightly different from what appeared in other works.

  * We present the solution to the optimal impulse control of an epidemic model, which is of its own interest.


The paper is organized in the following way. After describing the problem statement, we demonstrate the MDP approach and provide the integral optimality equation in  Section <ref>. In Section <ref>, we prove the equivalence of the integral and differential forms of the optimality equation. The impulse control of SIR epidemic is developed in Section <ref>. In Conclusion, we briefly describe the ways for generalizing our results to PDMP.

The following notations are frequently used throughout this paper. ={1,2,…} is the set of natural numbers;
δ_x(·)
is the Dirac measure concentrated at x, we call such distributions degenerate; I{·} is the indicator function.
B(E) is the Borel σ-algebra of the Borel space E, P(E) is the Borel space of probability measures on E. (It is always clear which σ-algebra is fixed in E.) The Borel σ-algebra B( P(E)) comes from the weak convergence of measures, after we fix a proper topology in E. ℝ_+ (0,+∞), ℝ_+^0[0,+∞), ^0_+[0,+∞]; in _+ and _+^0, we consider the Borel σ-algebra, and Leb is the Lebesgue measure.
The abbreviation w.r.t. (resp. a.s.) stands for “with respect to” (resp. “almost surely”); for b∈[-∞,+∞], b^+max{b,0} and b^-min{b,0}. inf∅ +∞. Measures introduced in the current article can take infinite values.  If b=∞ then the integrals ∫_(a,b]f(u)du are taken over the open interval (a,∞).



§ PROBLEM STATEMENT



We will deal with a control model defined through the following elements.


  * X is the state space, a Borel subset of a complete separable metric space with metric ρ_X and the Borel σ-algebra.

  * ϕ(· , ·):  X×^0_+→ X is the flow possessing the semigroup property ϕ(x,t+s)=ϕ(ϕ(x,s),t) for all x∈ X and (t,s)∈(^0_+)^2; ϕ(x,0)=x for all x∈ X. Between the impulses, the state changes according to the flow.

  * A is the action space, again a Borel subset of a complete separable metric space with metric ρ_A and the Borel σ-algebra.

  * l(· , ·):  X× A→ X is the mapping describing the new state after the corresponding action/impulse is applied.

  * C^g(·):  X→ is the (gradual) cost rate.

  * C^I(· , ·):  X× A→ is the cost associated with the actions/impulses applied in the corresponding states.

All the mappings ϕ(·),l(·),C^g(·) and C^I(·) are assumed to be measurable.

Let X_Δ= X∪{Δ}, where Δ is an isolated artificial point describing the case that the controlled process is over and no future costs will appear. The dynamics (trajectory) of the system can be represented as one of the following sequences

    x_0→ (θ_1,a_1)→ x_1→ (θ_2,a_2)→…;    θ_i<+∞ i∈, 
          
          x_0→ (θ_1,a_1)→…→ x_n→ (+∞,a_n+1)→Δ→ (θ_n+2,a_n+2) →Δ→…,

where x_0∈ X is the initial state of the controlled process and θ_i<+∞ for all i=1,2,…, n. For the state x_i-1∈ X, i∈, the pair (θ_i,a_i)∈^0_+× A is the control at the step i: after θ_i time units, the impulsive action a_i will be applied leading to the new state

    x_i={[ l(ϕ(x_i-1,θ_i),a_i),              θ_i<+∞;;                   Δ,              θ_i=+∞. ].

The state Δ will appear forever, after it appeared for the first time, i.e., it is absorbing.

After each impulsive action, if θ_1,θ_2,…,θ_i-1<+∞, the decision maker has in hand complete information about the history, that is, the sequence

    x_0, (θ_1,a_1), x_1,…,  (θ_i-1,a_i-1),x_i-1.

The next control (θ_i,a_i) is based on this information and we allow the pair (θ_i,a_i) to be randomized.

The cost on the coming interval of the length θ_i equals

    ∫_(0,θ_i]  C^g(ϕ(x_i-1,u))du+I{θ_i<+∞} C^I(ϕ(x_i-1,θ_i),a_i),

the last term being absent if θ_i=+∞.
If the cost functions C^g(·) and C^I(·) can take positive and negative values, then one can calculate separately the expressions in (<ref>) for the positive and negative parts of the costs and accept the convention +∞-∞ +∞.
The next state x_i is given by formula (<ref>).

In the space of all the trajectories (<ref>)

    Ω   =   ⋃_n=1^∞[ X×((^0_+× A)× X)^n
    ×({+∞}× A)×{Δ}× ((^0_+× A)×{Δ})^∞]
          ⋃ [ X×((^0_+× A)× X)^∞],

we fix the natural σ-algebra F.
Finite sequences

    h_i=(x_0, (θ_1,a_1), x_1, (θ_2,a_2),…,x_i)

will be called (finite) histories; i=0,1,2,…, and the space of all such histories will be denoted as H_i. Capital letters X_i,T_i,Θ_i, A_i and H_i denote the corresponding functions of ω∈Ω, i.e., random elements.


A control strategy π={π_i}_i=1^∞ is a sequence of stochastic kernels π_i on ^0_+× A given H_i-1. A control strategy is called stationary deterministic and denoted as (φ_θ,φ_a), if, for all i=1,2,…, π_i(dθ× da|h_i-1)=δ_φ_θ(x_i-1)(dθ) δ_φ_a(x_i-1)(da), where φ_θ: X_Δ→^0_+ and φ_a: X_Δ→ A are measurable mappings.


If the initial state x_0∈ X and a strategy π are fixed, then there is a unique probability measure P^π_x_0(·) on Ω satisfying the following conditions:

    P^π_x_0(X_0∈Γ_X)=δ_x_0(Γ_X)Γ_X∈ B( X_Δ);

for all i∈, Γ∈ B(^0_+× A), Γ_X∈ B( X_Δ),

    P^π_x_0((Θ_i,A_i)∈Γ|H_i-1)   =   π_i(Γ |H_i-1);
    
    P^π_x_0(X_i∈Γ_X|H_i-1,(Θ_i,A_i))
       =   {[ δ_l(ϕ(X_i-1,Θ_i),A_i)(Γ_X),           X_i-1∈ X, Θ_i<+∞;;                    δ_Δ(Γ_X)                             ].

For details, see the Ionescu Tulcea Theorem <cit.>. The mathematical expectation w.r.t. P^π_x_0 is denoted as E^π_x_0.

The optimal control problem under study looks as follows.

    π      
     V(x_0,π)    =     E^π_x_0[∑_i=1^∞ I{X_i-1Δ}{∫_(0,Θ_i]  C^g(ϕ(X_i-1,u)) du..
          + I{Θ_i<+∞}.. C^I(ϕ(X_i-1,Θ_i),A_i)}].



A control strategy π^* is called uniformly optimal if, for all x∈ X_Δ, V(x,π^*)= V^*(x)inf_π V(x,π).





§ MDP APPROACH

In this section, we establish the optimality results for problem (<ref>) by referring to the known ones for its induced total undiscounted Markov Decision Process (MDP).

The MDP under study is given by the state and action spaces X_Δ and ^0_+× A, transition kernel

    Q(dy|x,(θ,a)){[ δ_l(ϕ(x,θ),a)(dy),           xΔ, θ+∞;;            δ_Δ(dy)                  , ].

and cost

    C̃(x,(θ,a)) I{xΔ}{∫_(0,θ]  C^g(ϕ(x,u)) du
    + I{θ<+∞} C^I(ϕ(x,θ),a)}.

Clearly, actions of the form (+∞,a) can be treated as stopping the control process with the terminal cost ∫_(0,+∞) C^g(ϕ(x,u))du. In this framework, we denote as "stop" the strategy which immediately chooses θ_1=+∞: V(x_0,"stop")= ∫_(0,+∞) C^g(ϕ(x,u))du.
The artificial state Δ means that MDP is stopped without any future cost.  If,  for all x∈ X,  inf_π V(x_0,π)<+∞, then the optimal control problem (<ref>) is not degenerate. This assumption holds if ∫_(0,∞) C^g(ϕ(x,u))du<+∞ or, e.g., in the following cases.



  *  Absorbing case: there is a specific measurable  "cemetery" subset Y⊂ X such that for all y∈ Y C^g(y)≡ 0,
 for all u∈^0_+ ϕ(y,u)∈ Y and, for each x∈ X∖ Y,

 – either inf_u∈^0_+{u: ϕ(x,u)∈ Y} t^*(x)<+∞, the function t^*(x) is measurable, and ∫_(0,t^*(x)] C^g(ϕ(x,u))du<+∞,

 – or there is an action â∈ A such that l(x,â)∈ Y. 

  * Discounted case: the state space X has the form X= Y×^0_+, where Y is a Borel subset of a complete separable metric space, the component s_0 of the initial state x_0=(y_0,s_0) is zero, the flow ϕ(·) satisfies ϕ((y,s),t)=(ϕ_ Y(y,t),s+t), where ϕ_ Y(·) is a flow in Y, and
    
    C^g((y,s))   =   e^-α sC^g_ Y(y);
    
        C^I((y,s),a)   =   e^-α s C^I_ Y(y,a);
    
        l((y,s),a)   =   (l_ Y(y,a),s).

    Here the functions and mapping C^g_ Y(·), C^I_ Y(·) and l_ Y(·) are for the component y∈ Y only, and, for all y∈ Y, ∫_(0,+∞) e^-α u C^g_ Y(ϕ_ Y(y,u))du<+∞. α>0 is the discount factor, component s of the state x=(y,s) plays the role of time, and in principle one can consider the non-homogeneous model with the functions and mappings C^g_ Y(·), C^I_ Y(·) and l_ Y(·) depending also on the component s.

  * Generalized discounting: the model is as in the previous item, but

    C^g((y,s))=h(s)C^g_ Y(y);     C^I((y,s),a)=h(s) C^I_ Y(y,a),

and the measurable function h(·) is such that ∀ y∈ Y ∫_(0,+∞) h(s) C^g_ Y(ϕ_ Y(y,u))du<+∞.



Throughout this section, the following condition is satisfied.

C^g and C^I are ℝ_+^0-valued, that is, we consider the so called positive model with the total expected cost.


This condition means that we deal with a positive model. In this case, the value function V^*(x_0)inf_π V(x_0,π) is the minimal ℝ̅_+^0-valued lower semianalytic solution to the following optimality (Bellman) equation:
 
    V(Δ)   =   0,
    
     V(x)   =   inf_(θ,a)∈ℝ̅_+^0×A{C̃(x,(θ,a))+∫_ X_Δ V(y) Q(dy|x,(θ,a))}
       =   inf_(θ,a)∈ℝ̅_+^0×A{∫_(0,θ]  C^g(ϕ(x,u)) du
    + I{θ<+∞}( C^I(ϕ(x,θ),a)+V(l(ϕ(x,θ),a)))}
              ∀ x∈X.

(See <cit.>.)

Recall that our model is the special case of PDMP when the spontaneous (natural) jumps intensity λ equals zero. In case of discounted cost, corresponding versions of equation (<ref>) appeared in the works <cit.> on PDMP.

Note that the case of a simultaneous sequence of impulses, when Θ_i=Θ_i+1=…=0, is not excluded. In such cases, the total cost (<ref>) is calculated over a finite time horizon, up to the accumulation of impulses.

Section <ref> is devoted to the study of equation (<ref>),  without requiring that Condition <ref> is satisfied.

In the framework of stopping MDP, the decision to stop (here that means θ=+∞, and all the actions of the form (+∞,a) can be merged to one point) is usually considered as an isolated point of the action space ^0_+× A. But in this  case the remainder (real) action space ^0_+× A would be not compact. To avoid this inconvenience, we accept the following conditions.




  (a) The space A is compact, and +∞ is the one-point compactification of the positive real line ^0_+, so that the action space ^0_+× A in the MDP is compact.

  (b) The mapping (x,a)∈X×A→ l(x,a) is continuous.

  (c) The mapping (x,θ)∈X×ℝ_+^0→ϕ(x,θ) is continuous.

  (d) The function (x,a)∈X×A→ C^I(x, a) is lower semicontinuous.

  (e) The function x∈X→ C^g(x) is lower semicontinuous.



Still under these conditions, the model is not semicontinuous because, if xΔ, θ_n∈^0_+ and θ_n→+∞ then the transition probabilities Q(dy|x,(θ_n,a)) do not converge to δ_Δ(dy). Nevertheless, the usual dynamic programming approach is fruitful.


Suppose Conditions <ref> and <ref> are satisfied. Then the following assertions hold.


  (1) The minimal ^0_+-valued solution V(x) to equation (<ref>) is lower semicontinuous, unique, and can be constructed by successive approximations starting from V_0(x)≡ 0, x∈ X_Δ:
    
    V_n+1(Δ)   =    V_n(Δ);
    
        V_n+1(x)   =   inf_(θ,a)∈ℝ̅_+^0×A{∫_(0,θ]  C^g(ϕ(x,u)) du
    + I{θ<+∞}( C^I(ϕ(x,θ),a)..
          ..+V_n(l(ϕ(x,θ),a)))}     ∀ x∈X.

The sequence {V_n}_n=1^∞ increases point-wise and V(x)=lim_n→∞ V_n(x)= V^*(x).

  (2) There exist measurable mappings φ^*_θ: X→^0_+ and φ^*_a: X→ A such that, for all x∈ X,

    V^*(x)   =   ∫_(0,φ^*_θ(x)]  C^g(ϕ(x,u)) du
          + I{φ^*_θ(x)<+∞}( C^I(ϕ(x,φ^*_θ(x)),φ^*_a(x))
    + V^*(l(ϕ(x,φ^*_θ(x)),φ^*_a(x)))).


  (3) A stationary deterministic strategy (φ^*_θ,φ^*_a) is uniformly optimal if and only if it satisfies equality (<ref>).



Proof. It is sufficient to consider only x∈ X, as Δ is the isolated point of X_Δ. Suppose W(·) is a lower semicontinuous ^0_+-valued function on X and show that function on X×^0_+× A

    ∫_(0,θ]  C^g(ϕ(x,u)) du
    + I{θ<+∞}( C^I(ϕ(x,θ),a)+W(l(ϕ(x,θ),a)))

is lower semicontinuous and ^0_+-valued.

Firstly, let us show that the non-negative function ∫_(0,θ] C^g(ϕ(x,u))du is lower semicontinuous on X×^0_+. By a well known result of Baire, see e.g., <cit.>, there exists an increasing sequence of bounded ℝ_+^0-valued continuous functions, say {c_m(·)}_m=1^∞, on X such that c_m(x)↑ C^g(x) for each x∈X_.
For every m=1,2,…, function (x,θ)∈ X×^0_+→∫_(0,θ] c_m(ϕ(x,u))e^-u/m du is bounded continuous. By the monotone convergence theorem and using the result of Baire again, we see that function

    (x,θ)∈ X×^0_+ →∫_(0,θ] C^g(ϕ(x,u))du    =   ∫_(0,θ]lim_m→∞ c_m(ϕ(x,u))e^-u/m du
       =   lim_m→∞∫_(0,θ] c_m(ϕ(x,u))e^-u/m du

is lower semicontinuous.

Secondly, let us show that the non-negative function

    F(x,θ,a) I{θ<+∞}( C^I(ϕ(x,θ),a)+W(l(ϕ(x,θ),a)))

is also lower semicontinuous on X×^0_+× A. Suppose (x_n,θ_n,a_n)→ (x,θ+∞,a) as n→∞. Since the flow ϕ(·) and the mapping l(·) are continuous, we deduce that the sequences y_nϕ(x_n,θ_n) and l_n l(ϕ(x_n,θ_n),a_n) converge to yϕ(x,θ) and l l(ϕ(x,θ),a) correspondingly. Therefore

    lim inf_n→∞ F(x_n,θ_n,a_n)≥lim inf_n→∞ C^I(y_n,a_n)+lim inf_n→∞W(l_n,a_n)≥ C^I(y,a)+W(l,a)= F(x,θ,a)
 because the both functions C^I(·) and W(·) are lower semicontinuous. (See <cit.>). In case (x_n,θ_n,a_n)→ (x,+∞,a) as n→∞, it is obvious that lim inf_n→∞ F(x_n,θ_n,a_n)≥ 0=F(x,+∞,a).

Therefore, function (<ref>) is lower semicontinuous and obviously ^0_+-valued.

(1) Clearly, V_1(x)≥ V_0(x), so that the sequence {V_n}_n=1^∞ increases point-wise and hence converges to some ^0_+-valued function V(·). The non-negative function V_0(·) is lower semicontinuous and, if V_n(·) is a non-negative lower semicontinuous function then so is  function V_n+1(·) by Proposition 7.32 <cit.>. Therefore, function V(·) is lower semicontinuous by the mentioned above Baire result.

For every n=0,1,2,…, function (<ref>), with W(·) being replaced with V_n(·), is lower semicontinuous. Therefore, the set

    U_n(x,λ)      {(θ,a)∈^0_+× A: ∫_(0,θ] C^g(ϕ(x,u))du+I{θ<+∞} (C^I(ϕ(x,θ),a)
          +V_n(l(ϕ(x,θ),a)))≤λ}

is closed and hence compact for all x∈ X, λ∈. (See Condition <ref>(a).) By Proposition 9.17 <cit.>, V(x)= V^*(x).

(2) The value function V^*(·) satisfies equation (<ref>) and is lower semicontinuous. By Proposition 7.33 <cit.>, there exists a measurable mapping φ^*:  X→^0_+× A which provides the infimum in (<ref>) for all x∈ X. Assertion (2) follows.

(3) This assertion follows directly from Proposition 9.12 <cit.>.

 


Suppose Conditions <ref> and <ref> are satisfied and (φ^*_θ,φ^*_a) is a uniformly optimal stationary deterministic strategy. Then the following assertions hold true.


  (1) For every x∈ X with φ^*_θ(x)<+∞, for yϕ(x,φ^*_θ(x)), equality

    inf_a∈ A{C^I(y,a)+V(l(y,a))}=C^I(y,φ^*_a(x))+V(l(y,φ^*_a(x)))=V(y)

is valid.

  (2) For every x∈ X with φ^*_θ(x)>0, for each t∈[0,φ^*_θ(x)), for yϕ(x,t), equality

    V(y)    =   ∫_(0,φ^*_θ(x)-t] C^g(ϕ(y,u))du+I{φ^*_θ(x)<+∞}[C^I(ϕ(y,φ^*_θ(x)-t),φ^*_a(x)) 
           +V(l(ϕ(y,φ^*_θ(x)-t),φ^*_a(x)))]


is valid and hence

    V(x)=∫_(0,t] C^g(ϕ(x,u))du+V(ϕ(x,t)).



Proof. (1) The left equality is obvious.

The case when V(y) is bigger than the expression of the left is excluded. (Consider θ=0 for y in (<ref>).) If V(y) is smaller then, the pair

    (φ̃_θ(x)φ^*_θ(x)+φ^*_θ(y), φ^*_a(y))

gives rise to the  expression

    ∫_(0,φ^*_θ(x)] C^g(ϕ(x,u))du+∫_(φ^*_θ(x),φ̃_θ(x)] C^g(ϕ(x,u))du
          +I{φ̃_θ(x)<+∞}[C^I(ϕ(x,φ̃_θ(x)),φ^*_a(y))+ V(l(ϕ(x,φ̃_θ(x)),φ^*_a(y)))]
       =   ∫_(0,φ^*_θ(x)] C^g(ϕ(x,u))du+ V(y),

that is, (φ̃_θ(x),φ^*_a(y)=φ^*_a(ϕ(x,φ^*_θ(x))))
provides the smaller value for

    ∫_(0,θ] C^g(ϕ(x,u))du+I{θ<+∞} [C^I(ϕ(x,θ),a)+V(l(ϕ(x,θ),a))]

than the pair (φ^*_θ,φ^*_a), which contradicts the definition of (φ^*_θ,φ^*_a).

(2) If φ^*_θ(x)<+∞, then φ^*_a(x) provides the infimum to

    C^I(ϕ(y,φ^*_θ(x)-t),a)+V(l(φ(y,φ^*_θ(x)-t),a))
    =C^I(ϕ(x,φ^*_θ(x)),a)+V(l(φ(x,φ^*_θ(x)),a)).

V(y) cannot be bigger than the expression on the left in (<ref>). (Consider θ=φ^*_θ(x)-t for y in (<ref>).) If V(y) is smaller then, like previously, the pair

    (φ̃_θ(x)=t+φ^*_θ(y), φ^*_a(y))

provides the smaller value for (<ref>) than the pair (φ^*_θ,φ^*_a), which contradicts the definition of (φ^*_θ,φ^*_a).

Equality (<ref>) follows from (<ref>) because

    V(x)   =   ∫_(0,t] C^g(ϕ(x,u))du+∫_(0,φ^*_θ(x)-t] C^g(ϕ(y,u))du
          +I{φ^*_θ(x)<+∞}[C^I(ϕ(y,φ^*_θ(x)-t),φ^*_a(x)) +V(l(ϕ(y,φ^*_θ(x)-t),φ^*_a(x)))].


 


If Conditions <ref> and <ref> are satisfied then, for each x∈ X, in case the pair (θ̂<+∞,φ^*_a(x)) provides the infimum in equation (<ref>), equality

    C^I(ϕ(x,θ̂),φ^*_a(x))+V(l(ϕ(x,θ̂),φ^*_a(x)))=V(ϕ(x,θ̂))

is valid. The proof coincides with the proof of Item (1) of Corollary <ref>.


Below, it will be assumed that the function V^*(·) is finite-valued. This requirement is obviously satisfied for positive models if, for each x∈ X, there exists a control strategy π such that V(x,π)<+∞. The latter assumption follows from the following condition.

 For all x ∈𝐗 the composite function C^g(ϕ(x,t)) is Lebesgue integrable on ℝ_+^0. This means that the integral

    ∫_(0,+∞) C^g(ϕ(x,t))  dt

exists and is finite.


In what follows,
we accept the following convention.
We say that a function g :  X →ℝ satisfies a certain property (is continuous, absolutely continuous, measurable, Lebesgue integrable, etc.) along the flow ϕ, if for all x ∈ X the composite function t ↦ g(ϕ(x,t)) from ℝ_+^0 to ℝ satisfies this property. In view of this convention, Condition <ref> asserts that the function C^g is Lebesgue integrable along the flow.

The following proposition states that for each x∈ X the set of values θ providing the infimum in (<ref>)  is closed in ^0_+, and hence, contains its minimal value denoted as θ^*(x).

Under Conditions <ref> and <ref>, for the minimal non-negative solution V to equation (<ref>),
we introduce the function G:  X×^0_+× A→^0_+  by

    G(x,θ,a)∫_(0,θ] C^g(ϕ(x,u))du+I{θ<+∞}[C^I(ϕ(x,θ),a)+V(l(ϕ(x,θ),a))]

and the sets  Θ(x) by

    Θ(x) {θ∈^0_+: inf_a∈ A G(x,θ,a) =V(x)}.

For a fixed x∈ X, the set Θ(x) contains all time moments θ such that the pair (θ,â) provides the infimum in (<ref>). Here, for θ∈Θ(x)∩^0_+, â∈ A provides the infimum in (<ref>); such â exist because the function G is lower semicontinuous in a, if Conditions <ref> and <ref> are satisfied.


Suppose Conditions <ref> and <ref> are satisfied. Then for all x ∈𝐗 the set Θ(x) is non-empty and closed in ^0_+, and therefore contains the value θ^*(x) infΘ(x), the mapping θ^*(·) being measurable.


The proofs of all propositions are postponed to the Appendix.

Recall that, under Conditions <ref> and <ref>, there is a measurable mapping φ̂^*_a:  X×^0_+→ A providing

    inf_a∈ A{C^I(ϕ(x,θ),a)+V(l(ϕ(x,θ),a))}.

(See Proposition 7.33 of <cit.>.)
The mapping
θ^*(x) = infΘ(x) from Proposition <ref> is measurable, so that the pair (φ^*_θ,φ^*_a) with φ^*_θ(x)θ^*(x) and
φ^*_a(x)φ̂^*_a(x,θ^*(x)) satisfies Item (2) of Theorem <ref>.

Further, we will need the following two conditions strengthening Conditions  <ref> and <ref> respectively.
 There is K ∈ℝ_+ such that for all x∈ X, ∫_(0,+∞) |C^g(ϕ(x,u))|du≤ K.


 The function C^g is ℝ_+^0-valued and C^I≥δ>0.


Condition <ref>  guarantees that, for reasonable strategies π, Θ_i is finite only a finite number of times (P^π_x_0-a.s.): otherwise V(x_0,π)=+∞.

Under Conditions <ref> and <ref>, starting from any initial state x_0∈ X, for the optimal strategy the MDP must be stopped at a finite time moment

    T_stop=min{i: Θ_i+1=+∞}

and, for any reasonable strategy π,

    E^π_x_0 [T_stop]≤K/δ:

otherwise, the total cost

    V(x_0,π)=E^π_x_0[∑_i=0^∞C̃(X_i,(Θ_i,A_i))]>K

is bigger than that coming from stopping the MDP immediately:

    V(x_0,"stop")=∫_(0,+∞) C^g(ϕ(x,u))du≤ K.

If we restrict ourselves to such control strategies, then we are in the framework of absorbing MDP <cit.>, and the following proposition can be proved similarly to Theorem 9.6.10(c) <cit.>.


Suppose Conditions <ref>, <ref> and <ref> are satisfied. Then the Bellman equation (<ref>) has a unique bounded lower semicontinuous solution.


  
According to the proof of Proposition <ref> (see Appendix),
the Bellman equation (<ref>) has a unique bounded lower semicontinuous solution also in the case when Conditions <ref> and <ref> are satisfied, the function V^* is bounded and, for all strategies π, for all x_0∈ X, lim_i→∞ E^π_x_0[V(X_i)]=0, where V is a bounded lower semicontinuous function satisfying equation (<ref>). In this case V = 𝒱^*.
 





§ DIFFERENTIAL FORM OF THE OPTIMALITY EQUATION


In this section, we establish the equivalence of the integral and differential formulations of the optimality equation using minimal assumptions about the system. We do not assume any structure of the sets 𝐗 and 𝐀 and of the maps C^I :  X ×𝐀→ℝ and l :  X ×𝐀→ X; we only require Condition <ref> to be satisfied.  Firstly, we justify the differential form of the optimality equation for the general model studied in Section <ref>. After that, we briefly  discuss the discounted model.



 §.§ Total Cost Model



A point x is said to be a singular point of the flow ϕ, if it is not an intermediate point of a trajectory, that is, the equation x = ϕ(x̃, s) has no solutions for all x̃∈ X and s > 0. Note that if the flow possesses the group property, i.e., ϕ(x,t+s)=ϕ(ϕ(x,s),t) for all x∈ X and (t,s)∈^2, then there are no singular points.


Let V : 𝐗→ℝ be a certain function. We denote

    ℱ_+^V(x) lim_t → 0^+[ V(ϕ(x,t)) - V(x)/t + 1/t∫_(0,t] C^g(ϕ(x,u))  du ],

provided that the limit in the right hand side exists.

Further, if x is a nonsingular point of the flow, we define the number set

    F_-^V(x)      {lim_t → 0^+[ V(x) - V(ϕ(x̃, s-t))/t.
    
          . + 1/t∫_(-t,0] C^g(ϕ(x̃, s+u))  du ] :  (x̃,s)∈ X×_+,  ϕ(x̃,s)=x}⊂ℝ∪{±∞}.

With some abuse of notation, if F_-^V(x) is a singleton (e.g. if the flow possesses the group property), then we identify it with its element.
If, otherwise, x is a singular point, then we set F_-^V(x) = ∅.


If X'⊂𝐗⊂^d is a smooth open manifold, the flow is given by the differential equation ẋ = f(x), satisfying the standard conditions on the existence of a unique local solution in X' (for positive and negative t), for each initial condition from X', and C^g(x) is continuous along the flow in X' and V(x) is continuously differentiable along the flow in X', then F_-(x) is a singleton for all x∈ X' and

    ℱ_+^V(x) = F_-^V(x) = C^g(x) + ∇ V(x) · f(x).


Consider the optimality equation (<ref>) on X, that is,
the following integral equation:

    V(x) = inf_θ∈R̅_+^0{∫_(0,θ] C^g(ϕ(x,u))  du + I{θ < +∞} inf_a ∈𝐀{ C^I(ϕ(x,θ), a) + V(l(ϕ(x,θ), a)) }},   x∈ X.

Everywhere further, we assume that the function V(x) is finite-valued. For example, under Conditions <ref>, <ref> and <ref> the value function V^*(x)=V(x), studied in Section <ref>, is finite and satisfies the equation (<ref>) by Theorem <ref>.

  For each x the former infimum in the right hand side of (<ref>) is attained on a nonempty set Θ(x) ⊂ℝ̅_+^0, and Θ(x) contains its infimum.


We emphasise that Condition <ref> is satisfied under Conditions <ref> and <ref> if X is a Borel space: see Proposition <ref>.

We also consider the so called Bellman equation in the differential form:

    


for all x∈𝐗,

    either  (a)    ℱ_+^V(x) = 0
     
    and     inf_a∈𝐀[ C^I(x, a) + V(l(x, a)) - V(x) ] > 0,
    
    or   (b)    F_-^V(x) ⊂ℝ̅_+^0
    
    and     inf_a∈𝐀[ C^I(x, a) + V(l(x, a)) - V(x) ] = 0.



In the case (a) it is assumed that the right limit exists and equals 0. If it does not exist then the case (b) should take place.


Recall that our model is the special case of PDMP when the spontaneous (natural) jumps intensity λ equals zero. In case of discounted cost, corresponding versions of equation (<ref>) appeared in the works <cit.> on PDMP, see also the paper <cit.> on the purely deterministic system. In <cit.>, the undiscounted case was also investigated.
More about connection of the current work with existing results at the end of Subsection <ref>. Here, we only emphasize that the differential form in the shape of inclusion F_-^V(x) ⊂ℝ̅_+^0 did not appear in the cited literature. Remember, F_-^V(x) is a singleton in case the flow possesses the group property.

Define the set

    ℒ{ x ∈ X : inf_a∈𝐀[ C^I(x, a) + V(l(x, a)) - V(x) ] = 0 }.

If V is  a solution to equation (<ref>), the set L
can be understood as the set of the states, where actions/impulses must be applied. Suppose Conditions <ref> and <ref> are satisfied and function V is the minimal ^0_+-valued solution to equation (<ref>). For each x∈ X, if θ̂∈Θ(x)∩^0_+ (see (<ref>)), then θ̂ provides the infimum to inf_a∈ A G(x,θ,a) and hence the pair (θ̂,φ^*_a(x)), where C^I(ϕ(x,θ̂),φ^*_a(x))+V(l(ϕ(x,θ̂),φ^*_a(x)))=V(ϕ(x,θ̂)), provides the infimum in (<ref>). According to Remark <ref>, ϕ(x,θ̂)∈ L, that is, Θ(x)∩^0_+⊂{t∈^0_+: ϕ(x,t)∈ L}. Usually, this inclusion is strict, and Θ(x)∩^0_+ is a singleton coinciding with the infimum of {t∈^0_+: ϕ(x,t)∈ L}.

We consider the following conditions on the function V(x) satisfying equation (<ref>).



 For all x the set { t ∈ℝ_+^0 : ϕ(x,t) ∈ℒ} is either empty, or contains its infimum.


  The function V(x) is right lower semicontinuous and left upper semicontinuous along the flow. That is, first, for all x we have

    lim_t → 0^+ V(ϕ(x,t)) ≥ V(x).

Second, for all x and all (x̃,s)∈ X×_+ such that ϕ(x̃, s) = x we have

    lim_t → 0^+ V(ϕ(x̃, s-t)) ≤ V(x).

(If x is singular, the inequality is satisfied by default.)


  If, for some x ∈𝐗 and s > 0 and for all 0 ≤ t < s the states ϕ(x,t) are not in ℒ,
then lim_t → s^- V(ϕ(x,t)) = V(ϕ(x,s)). In other words, if the relative interior points of the flow trajectory between x and ϕ(x,s) are not contained in ℒ then V(x) is left continuous along the flow at ϕ(x,s).


In the following theorem, we establish the equivalence of the integral and differential forms of the optimality equation. To the best of our knowledge, such an analytical proof never appeared in the existing literature. Note that we do not assume that the flow possesses the group property. 

 Suppose Condition <ref> is satisfied.
Let the  function V : 𝐗→ℝ be measurable along the flow, and additionally, the integral ∫_(0,+∞) V(ϕ(x,t))  dt be finite for all x∈ X. Then the following statements are equivalent.


  (1) V(x) satisfies equation (<ref>) and Condition <ref>;

  (2) V(x) satisfies equation (<ref>) and Conditions <ref>-<ref>.



Proof. Below, we use the notation

    ℐV(x) inf_a ∈𝐀{ C^I(x, a) + V(l(x, a)) }.


1. Suppose that assertion (1) is valid and prove assertion (2).

Let x∈ X be fixed. For any t > 0 we can write down

    V(x) ≤inf_θ∈ [t, +∞]{∫_(0,θ] C^g(ϕ(x,u))  du + I{θ < +∞} ℐV(ϕ(x, θ)) }.

Changing the variables u - t = v,  θ - t = s, using the semigroup property ϕ(x, v+t) = ϕ(ϕ(x,t), v) and denoting for brevity x' = ϕ(x,t), we get

    V(x) ≤∫_(0,t] C^g(ϕ(x,u))  du + inf_s∈ℝ̅_+^0{∫_(0,s] C^g(ϕ(x', v))  dv
    
    
    + I{ s < +∞} ℐV(ϕ(x', s)) },

and thus,

    V(x) ≤∫_(0,t] C^g(ϕ(x,u))  du + V(ϕ(x,t)).

One can easily see that the integral in the right hand side goes to 0 as t → 0^+, and, taking the lower limit of the both parts in this inequality, one obtains

    V(x) ≤lim_t → 0^+ V(ϕ(x,t)).

That is, V is right lower semicontinuous along the flow.

If x is not singular, let x = ϕ(x̃, s) with s > 0. For all t ∈ [0,  s] we have

    V(ϕ(x̃, s-t)) ≤inf_θ∈ [t, +∞]{∫_(0,θ] C^g(ϕ(x̃, s-t+u))  du
    + I{θ < +∞} ℐV(ϕ(x̃, s-t+θ)) }.

Changing the variables u - t = v and θ - t = τ, we get

    V(ϕ(x̃, s-t)) ≤∫_(-t,0] C^g(ϕ(x̃, s+v))  dv


    +inf_τ∈ℝ̅_+^0{∫_(0,τ] C^g(ϕ(x̃, s+v))  dv
    + I{τ < +∞} ℐV(ϕ(x̃, s+τ)) }.

Taking into account that ϕ(x̃, s) = x and using (<ref>) we obtain

    V(ϕ(x̃, s-t)) ≤∫_(-t,0] C^g(ϕ(x̃, s+v))  dv + V(x).

Now taking the upper limit of the both parts in this relation and using that the integral in the right hand side goes to 0 as t → 0^+, one gets

    lim_t → 0^+ V(ϕ(x̃, s-t)) ≤ V(x).

That is, V is left upper semicontinuous along the flow, which means that Condition <ref> is satisfied.

Recall that Θ(x) is the (nonempty) set of values θ minimizing (<ref>). Let θ^*(x) = infΘ(x). By Condition <ref> we have θ^*(x) ∈Θ(x). Consider two cases: θ^*(x) > 0 and θ^*(x) = 0, and prove equation (<ref>).
 


(α) θ^*(x) > 0.

Take a finite t ∈ [0, θ^*(x)]. Then (<ref>) remains valid if the infimum is taken over θ∈ [t,  +∞]; as a consequence, we have the equality instead of "≤" in relations (<ref>) and (<ref>). Moreover, the infimum in (<ref>) is attained at s∈Θ(x) - t. Hence

    V(x) = ∫_(0,t] C^g(ϕ(x,u))  du + V(ϕ(x,t))

and Θ(ϕ(x,t)) = Θ(x) - t, and therefore,

    θ^*(ϕ(x,t)) = θ^*(x) - t.


 It follows from (<ref>) that
 
    V(ϕ(x,t)) - V(x)/t + 1/t∫_(0,t] C^g(ϕ(x,u))  du = 0,

and therefore,

    ℱ_+^V(x) = 0.



Now using that the infimum in (<ref>) is not attained at θ = 0, we have V(x) < ℐV(x), and therefore,

    inf_a∈𝐀[ C^I(x, a) + V(l(x, a)) - V(x) ] > 0.

Thus, relations (<ref>a) are valid, and therefore x ∉ℒ.



(β) θ^*(x) = 0.

If x is a singular point of the flow then the first relation in (<ref>b) is transformed into the valid formula ∅⊂ℝ̅_+^0.

Suppose x is a nonsingular point, that is, x = ϕ(x̃, s) with s > 0.
Rewrite inequality (<ref>) as follows

    V(ϕ(x̃, s-t)) -  V(x)/t≤1/t∫_(-t,0] C^g(ϕ(x̃, s+v))  dv;

hence

    lim_t → 0^+[  V(x) - V(ϕ(x̃, s-t))/t + 1/t∫_(-t,0] C^g(ϕ(x̃, s+v))  dv ] ≥ 0.

It follows that

    F_-^V(x) ⊂ℝ̅_+^0.


Further, since 0 ∈Θ(x) and therefore the infimum in (<ref>) is attained at θ = 0, we have V(x) = ℐV(x) = inf_a ∈𝐀{ C^I(x, a) + V(l(x, a)). It follows that

    inf_a∈𝐀[ C^I(x, a) + V(l(x, a)) - V(x) ] = 0.

Thus, relations (<ref>b) are valid, and therefore x ∈ℒ.



Let us check Condition <ref>. Suppose s > 0 and ϕ(x,t) ∉ℒ for all 0 ≤ t < s. This means that θ^*(ϕ(x,t)) > 0, and using (<ref>) we conclude that θ^*(x)>t for all t∈[0,s); hence θ^*(x) ≥ s. Substituting  s for t in formula (<ref>) we obtain V(x) = ∫_(0,s] C^g(ϕ(x,u))  du + V(ϕ(x,s)). Subtracting (<ref>) from this formula, one obtains

    0 = ∫_(t,s] C^g(ϕ(x,u))  du + V(ϕ(x,s)) - V(ϕ(x,t)),

and therefore,

    V(ϕ(x,t)) =  ∫_(t,s] C^g(ϕ(x,u))  du+V(ϕ(x,s)).

It follows that lim_t → s^- V(ϕ(x,t)) = V(ϕ(x,s)), and so, Condition <ref> is satisfied.

It remains to check Condition <ref>.
From (α) and (β) we conclude that if θ^*(x) > 0 then x ∉ℒ, and if θ^*(x) = 0 then x ∈ℒ. By formula (<ref>), if 0 ≤ t < θ^*(x) then θ^*(ϕ(x,t)) = θ^*(x) - t > 0 and therefore ϕ(x,t) ∉ℒ, and  if t = θ^*(x) then θ^*(ϕ(x,t)) = 0 and so, ϕ(x,t) ∈ℒ. It follows that if θ^*(x) = +∞ then the set

    { t ⊂ℝ_+^0: ϕ(x,t) ∈ℒ}

is empty, and if θ^*(x) < +∞ then θ^*(x) is contained in this set and is its infimum. Thus, Condition <ref> is satisfied.



2. Suppose that assertion (2) is valid and prove assertion (1). In the proof below we use the following statements, which will be proved in Appendix.


Let t ∈ℝ̅_+^0. If h is defined on [0,  t] ∩ℝ_+^0 and for all s ∈ (0,  t)

    h'_-(s)lim inf_t→ 0^+h(s)-h(s-t)/t≥ 0     h'_+(s)lim inf_t→ 0_+h(s+t)-h(s)/t≥ 0,

and additionally, h is right lower semicontinuous on [0,  t) and left upper semicontinuous on (0,  t] ∩ℝ_+^0, then h is monotone nondecreasing.



Let t ∈ℝ̅_+^0. If h is defined on [0,  t] ∩ℝ_+^0 and is left continuous on (0,  t] ∩ℝ_+^0, and h'_+(s) = 0 for s ∈ [0,  t) then h is constant.


Fix arbitrary x ∈𝐗 and t ∈ℝ̅_+^0 and define the function h by

    h(s) = V(ϕ(x,s)) -  ∫_(s,t] C^g(ϕ(x, v))  dv - I{ t < +∞} ℐ V(ϕ(x,t)),

with s ∈ [0,  t] ∩ℝ_+^0.

First, we show that h is (monotone) nondecreasing.
For s∈[0,t), consider h(s+τ)-h(s)/τ under τ∈(0,t-s). This ratio equals

    V(ϕ(x, s+τ)) - V(ϕ(x, s))/τ +  1/τ∫_(s,s+τ] C^g(ϕ(x,v))  dv.

Denoting x' = ϕ(x,s) and using the semigroup property of the flow and that the integral in the right hand side goes to zero as τ→ 0^+, we get

    h'_+(s) = lim_τ→ 0^+{V(ϕ(x', τ)) - V(x')/τ + 1/τ∫_(0,τ] C^g(ϕ(x', u))  du }= F^V_+(ϕ(x,s)),

if h'_+(s) and F^V_+(ϕ(x,s)) exist. We emphasize that F^V_+(ϕ(x,s)) and h'_+(s) exist (or do not exist) simultaneously.

In a similar way one calculates the lower left derivative of h and concludes that

    h'_-(s) ∈F_-^V(ϕ(x,s))

 for all s∈(0,t]∩ and also for s=0, provided x is not a singular point.

According to equation (<ref>), for x ∉ℒ the derivative ℱ_+^V(x) exists and equals zero, and for x ∈ℒ we have F_-^V(x) ⊂ℝ̅_+^0. By (<ref>) and (<ref>) we conclude that if ϕ(x, s) ∉ℒ then h'_+(s) exists and equals zero, and,  if ϕ(x, s) ∈ℒ then h'_-(s) ≥ 0. Taking into account Condition <ref>, we conclude that the function h is also right lower  semicontinuous in [0,  t) and left upper semicontinuous  in (0,  t] ∩ℝ_+^0. Therefore h satisfies all conditions of Proposition <ref> and hence is nondecreasing.

If t < +∞, we have

    h(t) = V(ϕ(x,t)) - ℐ V(ϕ(x,t)),

and by virtue of (<ref>), h(t) ≤ 0. Thus,

    h(0) = V(x) -  ∫_(0,t] C^g(ϕ(x, v))  dv - ℐ V(ϕ(x,t)) ≤ h(t) ≤ 0.


If, otherwise, t = +∞, we have for all s∈^0_+

    h(0) ≤ h(s) = V(ϕ(x,s)) - ∫_(s,+∞) C^g(ϕ(x, v))  dv.

According to Condition <ref>, the integral in the right hand side of this relation is finite and goes to 0 as s → +∞. The function s ↦ V(ϕ(x,s)) has a finite Lebesgue integral on ℝ_+^0. Therefore  h(s) ≤ 0 for all s∈^0_+ and in particular,

    h(0) = V(x) -  ∫_(0,+∞) C^g(ϕ(x,u))  du ≤ 0.


This proves that

    V(x) ≤inf_t∈ℝ̅_+^0{∫_(0,t]∩ C^g(ϕ(x,u))  du + I{ t < +∞} ℐ V(ϕ(x,t)) }.




Now for each x∈ X take the value

    t^*(x) := inf{ t ∈ℝ_+^0 : ϕ(x,t) ∈ℒ}∈ℝ̅_+^0

and set t = t^*(x) in the definition of the function h. We intend to show that in this case h(s)≡ 0.

For 0 ≤ s < t^*(x) we have ϕ(x,s) ∉ℒ and therefore h'_+(s) = 0.
According to Condition <ref>, the function s ↦ V(ϕ(x, s)), and therefore also the function h, are left continuous for finite 0 < s ≤ t^*(x). Hence by Proposition <ref>  h is constant on [0,  t^*(x)] ∩ℝ_+^0.

Let t^*(x) be finite. Condition <ref> states that ϕ(x, t^*(x)) ∈ℒ, and therefore,

    h(t^*(x)) = V(ϕ(x, t^*(x))) - ℐ V(ϕ(x, t^*(x))) = 0.

It follows that h(s) = 0 for all 0 ≤ s ≤ t^*(x), and in particular,

    h(0) = V(x) - ∫_(0,t^*(x)] C^g(ϕ(x,u))  du - ℐ V(ϕ(x,t^*(x))) = 0.


If, otherwise, t^*(x) = +∞, the constant function h(s) is the sum of a function that has a finite Lebesgue integral on ℝ_+^0 and a function going to 0 as s → +∞; therefore it is the null function, and again,

    h(0) = V(x) -  ∫_(0,+∞) C^g(ϕ(x,u))  du = 0.

This proves that

    V(x) =  ∫_(0,t^*(x)] C^g(ϕ(x,u))  du + I{ t^*(x) < +∞} ℐ V(ϕ(x,t^*(x)))


    ≥inf_t∈ℝ̅_+^0{∫_(0,t] C^g(ϕ(x,u))  du + I{ t < +∞} ℐ V(ϕ(x,t)) }.

As a consequence of (<ref>) and (<ref>) we obtain that equation (<ref>) is true and t^*(x) ∈Θ(x).

It remains to show that Condition <ref> is satisfied. Set an arbitrary 0 ≤ t < t^*(x) in the definition of h. Since ϕ(x,t) ∉ℒ, by (<ref>) and (<ref>a) we have

    h(t) =  V(ϕ(x,t)) - ℐ V(ϕ(x,t)) < 0,

and taking into account that h is nondecreasing, we conclude that

    h(0) = V(x) -  ∫_(0,t] C^g(ϕ(x,u))  du - ℐ V(ϕ(x,t)) ≤ h(t) < 0.

As a result we have

    V(x) < ∫_(0,t] C^g(ϕ(x,u))  du + ℐ V(ϕ(x,t));

that is, t ∉Θ(x) for 0 ≤ t < t^*(x). This implies that t^*(x) = infΘ(x), and so, Condition <ref> is satisfied.



Obviously, for all x∈ X, t∈_+, V^*(ϕ(x,t))≤∫_(t,+∞) C^g(ϕ(x,u))du. Therefore, if, under Condition <ref>, the following stronger version of Condition <ref>:

    ∀ x∈ X     ∫_(0,+∞)(∫_(t,+∞) C^g(ϕ(x,u))du) dt<+∞

is satisfied, then the integral ∫_(0,+∞) V^*(ϕ(x,t))dt is finite,  provided the function V^*:  X→ is Lebesgue-measurable along the flow. By Theorem <ref>, under Conditions <ref> and <ref> the latter requirement is satisfied and V^*=V is the minimal non-negative solution to equation (<ref>).






 §.§ Discounted Model


Note that for the validity of Theorem <ref>, only Condition <ref> is needed. In the case of the discounted model described in Section <ref>, it takes the following form.

The function C^g_ Y : 𝐘→ℝ is measurable along the flow and the integral

    ∫_(0,+∞) e^-α t C^g_ Y(ϕ_ Y(y,t))  dt

is finite.

The key  notations of F^V_+(x) and F^V_-(x) transform to

    ℱ_+^V(y) lim_t → 0^+[ e^-α t V(ϕ_ Y(y,t)) - V(y)/t + 1/t∫_(0,t] e^-α u C^g_ Y(ϕ_ Y(y,u))  du ];


    ℱ_-^V(y) {lim_t → 0^+[ V(y) - e^α t V(ϕ_ Y(ỹ, s-t))/t + 1/t∫_(-t,0]e^-α u C^g_ Y(ϕ_ Y(ỹ, s+u))  du ]:


    (ỹ,s)⊂ Y×_+, ϕ_ Y(ỹ,s)=y }.


If 𝐘' is a smooth open manifold, the flow is given by the differential equation ẏ = f(y), satisfying the standard conditions on the existence of a unique local solution in Y' (for positive and negative t) for each initial condition from Y',
and C^g_ Y(y) is continuous along the flow in Y' and V is continuously differentiable along the flow in Y' then ℱ^V_-(y) is a singleton for all y∈ Y' and

    ℱ_+^V(y) = ℱ_-^V(y) = -α V(y) + C^g_ Y(y) + ∇ V(y) · f(y).


The integral Bellman equation (<ref>) takes the form

    V(y) = inf_θ∈ℝ̅_+^0{∫_(0,θ] e^-α u C^g_ Y(ϕ_ Y(y,u))  du
    + I{θ < +∞}  e^-αθℐV(ϕ_ Y(y, θ)) },


    where ℐV is as before, ℐV(y) =  inf_a ∈𝐀{ C^I_ Y(y, a) + V(l_ Y(y, a)) }.

To be more precise, one had to denote the above Bellman function as V_ Y. In the framework of the extended state space X= Y×^0_+, the Bellman function is V((y,s))=e^-α sV_ Y(y).

The Bellman equation in the differential form (<ref>) remains as it was with the obvious changes x→ y, C^I(x,a)→ C^I_ Y(y,a) etc.

Finally, Theorem <ref> remains valid, provided that the integral ∫_(0,+∞) e^-α t V(ϕ_ Y(y,t))  dt is finite.
The latter holds true if C^g_ Y(y)≤ K<∞: Condition <ref> is satisfied because C^g((y,s))= e^-α s C^g_ Y(y) and, according to Theorem <ref>, the minimal positive solution to equation (<ref>) (i.e., to equation (<ref>)) has the form V((y,s))=e^-α sV_ Y(y), where 0≤ V_ Y(y)≤K/α.

As was mentioned after Remark <ref>, our model is the special case of PDMP when λ=0. In this connection, it is worth comparing equation ℱ_+^V(y) =0, which comes to the stage only if the right limit exists, and the corresponding differential forms obtained in <cit.>. After adding and subtracting e^-α t V(y) in the formula for ℱ_+^V(y), we obtain

    ℱ_+^V(y) = -α V(y) + lim_t → 0^+[ e^-α t V(ϕ_ Y(y,t)) - e^-α t V(y)+ ∫_(0,t] e^-α u C^g_ Y(ϕ_ Y(y,u))  du/t].

After denoting

    𝒳V(y) lim_t → 0^+[ e^-α t V(ϕ_ Y(y,t)) - e^-α t V(y)+ ∫_(0,t] e^-α u C^g_ Y(ϕ_ Y(y,u))  du/t]-C^g_ Y(y),

equality ℱ_+^V(y) =0 takes the form

    𝒳V(y)-α V(y)+C^g_ Y(y)=0,

which appears in <cit.> for the case X⊆^d. Moreover, in the smooth case, if the flow comes from the differential equation ẏ=f(y), as was mentioned in Remark <ref>, X V(y)=∇ V(y) · f(y). (See <cit.>.)

Connection between the integral and differential forms of the optimality equation was underlined in <cit.>. But it seems that the formal rigorous equivalence of such representations, as established in Theorem <ref>, is proved here for the first time for a general Borel space X and both for discounted and undiscounted cases. As explained in Conclusion, one can easily generalize Theorem <ref> for PDMP.



§ IMPULSE CONTROL OF SIR EPIDEMIC


In this section, we illustrate all the theoretical issues on a meaningful example having its own interest.

In the following proposition, we enlist all the conditions, which appeared in the previous sections, needed for the study of the model stated below.


Suppose Conditions <ref>, <ref> and <ref> are satisfied. Assume that a lower semicontinuous bounded function V:  X→^0_+ is such that


  * equation (<ref>) is valid;

  * Conditions <ref>, <ref> and <ref> are satisfied;

  * inequality ∫_(0,∞) V(ϕ(x,t))dt<∞ holds true for all x∈ X;

  * lim_i→∞ E^π_x_0[V(X_i)]=0 for all strategies π and for all x_0∈ X.

Then


  * V= V^* and

  * the strategy (φ^*_θ,φ^*_a), such that φ^*_θ(x)=inf{θ: ϕ(x,θ)∈ L}, where the set L is defined in (<ref>), and φ^*_a satisfies equality C^I(x,φ^*_a(x))+V(l(x,φ^*_a(x)))-V(x)=0, is uniformly optimal, provided that the both maps φ^*_θ and φ^*_a are measurable.




To formulate our Susceptible–Infected–Recovered (SIR) model
of epidemics, we use functions t ↦ x^1(t),
t ↦ x^2(t) and t ↦ x^3(t),
where x^1 : ℝ^0_+ →ℝ^0_+
denotes the dynamics of the susceptible population,
x^2 : ℝ^0_+ →ℝ^0_+
the dynamics of the infective population  and
x^3 : ℝ^0_+ →ℝ^0_+
the dynamics of the removed population (recovered or dead).
Following <cit.>, the progress
of infection is described by the following initial value problem:

    ẋ^1(t) = -βx^1(t) x^2(t)/x^1(t)+x^2(t),
    ẋ^2(t) = βx^1(t) x^2(t)/x^1(t)+x^2(t) - γ x^2(t),
    ẋ^3(t) = γ x^2(t),
    
    
    x^1(0) = x^1_0,    x^2(0) = x^2_0,    x^3(0) = 0,

for some given constant parameters β, γ∈ℝ_+
and initial data x^1_0, x^2_0 ∈ℝ^0_+. If x^1_0=x^2_0=0 then x^1(t)≡ x^2(t)≡ 0.
Problem (<ref>) has a unique solution, obtained in the closed
form in <cit.>. Explicit (non-impulse) optimal control policies
for (<ref>) are available in the literature: optimal isolation/treatment
of infective individuals has been studied in <cit.> while the case
of immunization/vaccination is investigated in <cit.>. Here we
formulate and explicitly solve an optimal control problem with
isolation/treatment impulses.





 §.§ Optimal Control Problem



Suppose there are no impulses.

We begin by noting that in (<ref>) one has
ẋ^1(t) + ẋ^2(t) + ẋ^3(t) = 0 for any t,
so that the total population is constant along time:
x^1(t) + x^2(t) + x^3(t) is a fixed constant x^1_0 + x^2_0 ∈ℝ_+.
For this reason, x^3(t) = x^1_0+x^2_0 - x^1(t) - x^2(t) and it is sufficient to
restrict ourselves to differential equations

    ẋ^1(t) = -βx^1(t) x^2(t)/x^1(t)+x^2(t),
    ẋ^2(t) = βx^1(t) x^2(t)/x^1(t)+x^2(t) - γ x^2(t),
    
    
    x^1(0) = x^1_0,    x^2(0) = x^2_0,

which define the flow ϕ.

Since there is no immigration (and no births) and isolation leads to the decrease of x^2, the whole state space is the triangle

    X={x^1≥ 0, x^2≥ 0,  x^1+x^2 < N }

with the topology induced from ^2;
it is convenient to take N>x^1_0+x^2_0, so that there are no singular points in X.
The gradual cost rate is the infection rate

    C^g(x^1,x^2) = βx^1 x^2/x^1+x^2,   C^g(0,0)=0,

which, after integration along the flow, results in the total number of new infectives. Here and below, usually the brackets of the argument of a function of x=(x^1,x^2) are omitted.

At any moment, the decision maker can isolate all infectives, so that A={1} is a singleton. The cost of an impulse equals

    C^I(x^1,x^2,a) = c x^2,

where c > 0 is a given constant. The new state after the impulse equals

    l(x^1,x^2,a) = (x^1, 0).

If x^2_0=0 then
 there are no individuals who can cause infection
and, therefore, the susceptible population will remain constant forever:
x^1(t) ≡ x^1_0. Thus

    Y={(x^1,x^2)∈ X: x^2=0}

is the "cemetery" subset.


Quite formally, in the states (x^1,x^2)∈ Y, one can still apply impulses leading to no cost and no change of the state. But actually, the controlled process is finished as soon as the state belongs to Y.


Note also that

    ∫_(0,+∞) C^g(ϕ((x^1_0,x^2_0),u))du ≤ x^1_0<  N < +∞,

meaning that the Bellman function V^* is bounded on the bounded domain X.

One can easily check that  Conditions <ref>,<ref>,<ref> and <ref> are satisfied.

To solve the optimal control problem, we investigate the differential form of the Bellman equation (<ref>) which is equivalent to (<ref>) by Theorem <ref>. As was mentioned in Remark <ref>, under certain conditions  which are satisfied in our example,

    F^V_+(x^1,x^2) =  F^V_-(x^1,x^2)= βx^1x^2/x^1+x^2 +∂ V/∂ x^1(-βx^1x^2/x^1+x^2)+∂ V/∂ x^2(βx^1x^2/x^1+x^2-γ x^2).


In the future, we will need the following explicit expressions defining the flow ϕ(x^1,x^2) at x^1,x^2 > 0 coming from the differential equation (<ref>) (see <cit.>):

    .[                                                                             βγ,                                                                                ;                                                                                               x^1(t)=x^1_0(1+x^2_0/x^1_0)^β/β-γ/(1+x^2_0/x^1_0 e^(β-γ)t)^β/β-γ;;                                                                                      x^2(t)=x^2_0(1+x^2_0/x^1_0)^β/β-γ e^(β-γ)t/(1+x^2_0/x^1_0 e^(β-γ)t)^β/β-γ;;                                                                            β=γ,                                                                                ;                                                                                 x^1(t)=x^1_0 e^-β x^2_0 t/x^1_0+x^2_0;   x^2(t)=x^2_0 e^-β x^2_0 t/x^1_0+x^2_0. ]}

From these expressions, it is clear that

    x^2(t)/x^1(t)=x^2_0/x^1_0 e^(β-γ)t

and, for all t≥ 0, x^1(t),x^2(t)>0, if x^1_0,x^2_0>0.

 Below, we summarise general properties of the epidemic model under study.


  * The Bellman function V^* is bounded and Condition <ref> is satisfied because of inequality (<ref>).

  * Conditions <ref> and <ref> are satisfied.

  * If V is a bounded lower semicontinuous function satisfying equation (<ref>), then V(0,x^2)=V(x^1,0)=0 for all x^1,x^2∈^0_+. As a result, for all strategies π, for all (x^1_0,x^2_0)∈ X, E^π_(x^1_0,x^2_0)[V(X_1)]=0 because either X^2_1=0 (if Θ_1<∞) and all the further values of the second component equal zero, or the next state is X_1=Δ (if Θ_1=∞). Hence, lim_i→∞ E^π_x_0[V(X_i)]=0 for all strategies π and for all x_0∈ X.

  * According to Remark <ref>, the Bellman equation (<ref>) has a unique bounded lower semicontinuous solution V= V^*.

  * If function V:  X→ is such that V(x^1,0)=V(0,x^2)=0 then, for x^1=0 or x^2=0, equalities

    F^V_+(x^1,x^2)= F^V_-(x^1,x^2)=0

and

    inf_a∈ A[C^I(x^1,x^2,a)+V(l(x^1,x^2,a))-V(x^1,x^2)]=cx^2

are valid. Thus, equations (<ref>) hold: case (a) is x^2>0 and x^1=0 and case (b) if x^2=0. All the conditions <ref>, <ref> and <ref> are trivially satisfied if x^1=0 or x^2=0.


The points of the form  (x^1,0) belong to L. Formally speaking, if x^2=0 one has to apply the simultaneous infinite sequence of impulses, each of them having no effect: see Remark <ref>. At the states (0,x^2>0), no impulses are needed: the number of infectives x^2 decreases to zero resulting in no cost.

We have seen that the flow ϕ(x,t) is continuous. We will see in all the three cases investigated in the further subsections that the function V to be defined below is continuous on X∩(_+)^2 and the corresponding set L is closed. It follows that conditions <ref>, <ref> and <ref> are satisfied.




 §.§ Solution in the Case β≥γ



In this subsection, we show that the  continuous function

    V(x^1,x^2)={[                            cx^2, x^2≤x^1/c,   x^1 ≥ 0,   x^2 ≥ 0;;                             x^1,      x^2>x^1/c,   x^1 ≥ 0, x^2>0 ].

satisfies all the requirements of  Proposition <ref>.

Firstly, let us show that the integral ∫_(0,+∞) V(ϕ(x^1_0,x^2_0,t))dt is finite for all x^1_0,x^2_0>0.

If x^2_0 > x^1_0/c, then

    x^2(t) > x^1(t)/ct > 0

because of (<ref>).
Therefore, for such initial values (x^1_0,x^2_0),

    ∫_(0,+∞) V(ϕ(x^1_0,x^2_0,t))dt=∫_(0,+∞) x^1(t)dt.

According to (<ref>),

    x^1(t)=x^2(t)x^1_0/x^2_0 e^-(β-γ)t.

Since function x^2(t) is uniformly bounded, in case β>γ, the integral ∫_(0,+∞) V(ϕ(x^1_0,x^2_0,t))dt is finite. If β=γ, its finiteness follows directly from (<ref>).

In case x^2_0 ≤x^1_0/c and β>γ, again using (<ref>), we see that x^2(t^*)=x^1(t^*)/c at t^*=ln(x^1_0)-ln(x^2_0 c)/β-γ<∞ and

    ∫_(0,+∞) V(ϕ(x^1_0,x^2_0,t))dt=∫_(0,t^*] c x^2(t) dt+∫_(t^*,+∞) x^1(t)dt<∞.

If x^2_0 ≤x^1_0/c and β=γ, then x^2(t)≤x^1(t)/c for all t≥ 0 and ∫_(0,+∞) V(ϕ(x^1_0,x^2_0,t))dt<∞ by (<ref>).

The  closed set L defined in (<ref>) has the form

    L={(x^1,x^2)∈ X∩(^0_+)^2: x^2≤x^1/c}.


Now show that equation (<ref>) is valid.

If (x^1,x^2)∈ L then

    F^V_-(x^1,x^2)   =   βx^1 x^2/x^1+x^2+∂ V/∂ x^2[βx^1 x^2/x^1+x^2-γ x^2]
    = x^2/x^1+x^2[ β(1+c)x^1-γ c(x^1+x^2)].

On the boundary x^2=x^1/c, the expression ∂ V/∂ x^2 means the left derivative, and the right derivative ∂ V/∂ x^1=0.
Since x^2≤x^1/c, we conclude that

    F^V_-(x^1,x^2)≥x^1 x^2/x^1+x^2[β(1+c)-γ(c+1)]=x^1 x^2/x^1+x^2 (β-γ)(c+1)≥ 0,

so that equality (<ref>), case (b), is satisfied.

The cases x^1=0 or x^2=0 were considered in Subsection <ref>.

If (x^1,x^2)∉ L and (x^1,x^2)∈(_+)^2 then

    inf_a∈ A [C^I(x^1,x^2,a)+V(l(x^1,x^2,a))-V(x^1,x^2)]>0

and

    F^V_+(x^1,x^2)=βx^1 x^2/x^1+x^2 +∂ V/∂ x^1[-βx^1 x^2/x^1+x^2]=0,

so that equality (<ref>), case (a), is satisfied.

According to  Proposition <ref>, the stationary strategy

    φ^*_θ(x^1,x^2) = {[            ∞,  (x^1,x^2)∉ L  ⟺ x^2>x^1/c;;            0,  (x^1,x^2)∈ L ⟺  x^2≤x^1/c; ].   φ^*_a(x^1,x^2)=1

is uniformly optimal. The straight line x^2=x^1/c is a dispersal line.

For (x^1,x^2)∈(_+)^2, it is reasonable to rewrite expression (x^1,x^2)∈ L as

    {(x^1,x^2)∈(_+)^2:  x^1≥ c x^2},

to understand better the meaning of the optimal strategy. The goal of the control is to save susceptibles  from being infected, but the cost of isolation is c x^2. Thus, isolation is reasonable only when there are many susceptibles to be saved: x^1≥ c x^2 because otherwise the cost of isolation, c x^2, is bigger than the profit for saving susceptibles (i.e., x^1).

The optimal strategy is shown in Figures <ref> and <ref>. If the initial state (x_0^1,x_0^2) is below the line x^2=x^1/c (shown in bold) then the impulse should be applied (dashed line).  If, otherwise, the initial state is above the line x^2=x^1/c then no impulse is needed and the system evolves according to equations (<ref>) (solid curves). If β=γ, then the critical line x^2=x^1/c is the trajectory of the dynamical system (<ref>). It is equally optimal to move along this line or to apply the impulse immediately or at any further time.







 §.§ Solution in the Case β<γ




  §.§.§ Case c≥β/γ-β
  In this subsection, we show that the function

    V(x^1,x^2)=x^1-x^1(1+x^2/x^1)^-β/γ-β,   (x^1,x^2)∈(_+)^2

satisfies all the requirements of  Proposition <ref>. According to Subsection <ref>, V(x^1,x^2)=0 if x^1=0 or x^2=0.
Firstly, let us show that the integral ∫_(0,+∞) V(ϕ(x^1_0,x^2_0,t))dt is finite for all x^1_0,x^2_0>0. According to (<ref>) and keeping in mind that x^1(t)≤ x^1_0, it is sufficient to prove that the integral

    I=∫_(0,+∞)[1-(1+x^2_0/x^1_0 e^-(γ-β)t)^-β/γ-β] dt

is finite.  This is a simple consequence of the fact that the integrand  is O(e^-(γ-β)t) as t→∞.

In the case under consideration, L= Y={(x^1,x^2): x^2=0}  is closed. The value x^2=0 is not reachable in finite time from initial conditions (x^1,x^2>0).

It remains to check equation (<ref>) for the presented function V. Namely, we will show that the version (a) is valid. The cases x^1=0 or x^2=0 were considered in Subsection <ref>. For (x^1,x^2)∈(_+)^2, according to (<ref>), equality F^V_+(x^1,x^2)=0 can be checked straightforwardly. Finally, for (x^1,x^2)∈(_+)^2,

    inf_a∈ A [C^I(x^1,x^2,a)+V(l(x^1,x^2,a))-V(x^1,x^2)]=cx^2-x^1+x^1(1+x^2/x^1)^-β/γ-β


    ≥ x^1 [ (1+x^2/x^1)^-β/γ-β - ( 1 - β/γ-βx^2/x^1)] >0.



According to  Proposition <ref>, the stationary strategy

    φ^*_θ(x^1,x^2) = {[           ∞, (x^1,x^2)∉ L     ⟺ x^2>0;;           0, (x^1,x^2)∈ L    ⟺  x^2=0; ].   φ^*_a(x^1,x^2)=1

is uniformly optimal.

When x^2>0, isolation is not reasonable as its cost c≥γ/γ-β is too high. When x^2=0, the epidemic is actually terminated, although the formal solution prescribes isolation of zero infectives for zero cost, without any real effect.

The optimal strategy in this case for the values c = 5, β = 3 and γ = 4 is shown in Figure <ref>. No impulses are needed here, and the system evolves according to equations (<ref>).







  §.§.§ Case c<β/γ-β



In this subsection, we show that the continuous function

    V(x^1,x^2)={[                                             c x^2,                             x^2≤β+β c-γ c/γ c x^1;; x^1[ 1-(γ c(1+x^2/x^1)/β+β c)^-β/γ-β
    (1+c) γ-β/γ],                              x^2>β+β c-γ c/γ c x^1 ].

satisfies all the requirements of  Proposition <ref>.

Firstly, let us show that the integral ∫_(0,+∞) V(ϕ(x^1_0,x^2_0,t))dt is finite for all x^1_0,x^2_0>0. Indeed, if x^2_0/x^1_0≤β+β c-γ c/γ c, then, by (<ref>),

    ∫_(0,+∞) V(ϕ(x^1_0,x^2_0,t))dt=c∫_(0,+∞) x^1(t)(x^2_0/x^1_0 e^-(γ-β)t) dt<∞

because the function x^1(t)≤ x^1_0 is bounded. If x^2_0/x^1_0>β+β c-γ c/γ c, then x^2(t^*)/x^1(t^*)=β+β c-γ c/γ c at the finite time moment

    t^*=θ^*(x^1,x^2)=1/γ-βlnγ c x^2_0/x^1_0(β+β c-γ c)>0,

the integral ∫_(0,t^*] V(ϕ(x^1_0,x^2_0,t))dt is finite, and on the interval (t^*,+∞) the previous reasoning applies.

 Let us check that the set L defined in (<ref>) has the form

    L={(x^1,x^2)∈ X∩(^0_+)^2: x^2≤β+β c-γ c/γ c· x^1},

 and therefore is closed.

The case when x^1=0 or x^2=0 was considered in Subsection <ref>.

If 0<x^2≤β+β c-γ c/γ c· x^1, then

    C^I(x^1,x^2,a)+V(l(x^1,x^2,a))-V(x^1,x^2)= cx^2+V(x^1,0)-cx^2=0,

so that (x^1,x^2)∈ L. Remember, a=1∈ A is the unique action.

If x^2 > β+β c-γ c/γ c· x^1>0, then

    C^I(x^1,x^2,a)+V(l(x^1,x^2,a))-V(x^1,x^2)= x^1Υ(x^2/x^1),

where  function

    Υ(w)=cw-1+(γ c(1+w)/β+β c)^-β/γ-β
     (1+c) γ-β/β

 is strictly convex. When w=β+β c-γ c/γ c,

    Υ(β+β c-γ c/γ c)=dΥ/dw(β+β c-γ c/γ c)=0.

Thus, Υ (w) > 0 for  w>β+β c-γ c/γ c, and
therefore,

    C^I(x^1,x^2,a)+V(l(x^1,x^2,a))-V(x^1,x^2)>0

if x^2>β+β c-γ c/γ c· x^1≥ 0, and (x^1,x^2)∉ L: the case x^1=0 is not excluded, as well.


Now show that equation (<ref>) is valid.

If (x^1,x^2)∉ L and (x^1,x^2)∈(_+)^2 then we already know that

    C^I(x^1,x^2,a)+V(l(x^1,x^2,a))-V(x^1,x^2)>0.

Equality

    F^V_+(x^1,x^2)=βx^1 x^2/x^1+x^2 +∂ V/∂ x^1[-βx^1 x^2/x^1+x^2]
    +∂ V/∂ x^2[βx^1 x^2/x^1+x^2-γ x^2]=0

in the area x^2>β+β c-γ c/γ c· x^1>0 can be checked by the direct substitution.
Equation (<ref>), case (a), is valid.

The cases x^1=0 or x^2=0 were considered in Subsection <ref>.

If x^1>γ c/β+β c-γ c x^2>0, then

    F^V_-(x^1,x^2)   =   βx^1 x^2/x^1+x^2+∂ V/∂ x^2[βx^1 x^2/x^1+x^2-γ x^2]
    = x^2/x^1+x^2[ β(1+c)x^1-γ c(x^1+x^2)]
       ≥   x^2/x^1+x^2[(β+β c-γ c)x^2γ c/β+β c-γ c-γ c x^2 ]=0.

On the boundary x^2=β+β c-γ c/γ c· x^1, we need to consider the  left derivative ∂ V/∂ x^1 and the right derivative ∂ V/∂ x^2.  As  a result, F^V_-(x^1,x^2)=0 similarly to (<ref>).
Equation (<ref>), case (b), is valid.

According to Proposition <ref>, the stationary strategy

    φ^*_θ(x^1,x^2) = {[                         ∞,               (x^1,x^2)∉ L  ⟺ x^2>β+β c-γ c/γ c· x^1;;                         0,               (x^1,x^2)∈ L ⟺  x^2≤β+β c-γ c/γ c· x^1; ].   φ^*_a(x^1,x^2)=1

is uniformly optimal. The straight line x^2=β+β c-γ c/γ c· x^1 is a switching line.

Like in the case β≥γ, isolation of infectives is reasonable only when there are sufficiently many susceptibles to be saved: x^1≥γ c/β+β c-γ c  x^2.

In this case we take c = 3/2, β = 3 and γ = 4; see Figure <ref>. If the initial state (x_0^1,x_0^2) lies below the line x^2=β+β c-γ c/γ c· x^1 (shown in bold) then the impulse should be applied (dashed line).  If the initial state lies above this line then initially no action is needed and the system evolves according to equations (<ref>) (solid curves)
up to the moment when x^2(t)=β+β c-γ c/γ c· x^1(t) when the impulse should be applied.







 §.§ Discussion

The threshold nature of the optimal isolation strategy for other epidemic models with similar cost functions was established in <cit.>: intervene only if the current number of infectives is below a certain value. Moreover, it was shown that the intervention must be global, i.e., it is better to isolate all infectives at once.

It is interesting to compare the impulse control problem from Subsection <ref> with its gradual control analogue investigated in <cit.>. Instead of impulses, dynamic control u(t)∈[0,U] appears in the second equation of (<ref>):

    ẋ^2(t) = βx^1(t) x^2(t)/x^1(t)+x^2(t) - γ x^2(t)-u(t)x^2(t).

Objective functional in <cit.>

    V(x_0^1,x_0^2,u)=∫_(0,∞)(βx^1(t) x^2(t)/x^1(t)+x^2(t)+cu(t) x^2(t))dt→inf_u

has the same meaning as in the current paper: combination of the total number of the new infectives and the total cost of isolation with the weight coefficient c>0. Intuitively, the impulse isolation at time moment t means that u(t)→∞. Thus, look at the optimal strategy obtained in <cit.> when U→∞.


  * If β≥γ then one has to apply the maximal rate of isolation U as soon as  x^2≤ζ(U) x^1, where

    ζ(U)=(γ+U+cU/cU)^γ+U-β/γ+U-1,

and the straight line x^2=ζ(U) x^1 is a dispersal line. When U→∞,

    lim_U→∞ζ(U)=1+c/c-1=1/c,

and we finish with exactly the optimal impulse strategy presented in Subsection <ref>.

  * If β<γ and c≥β/γ-β then, both in <cit.> and in Subsubsection  <ref>, it is optimal not to immunize at all.

  * If β<γ and c<β/γ-β then one has to apply the maximal rate of isolation U as soon as x^2≤ξ(U) x^1, where
    
    ξ(U)=(β(γ+U+cU)/cγ(γ+U-β))^γ+U-β/γ+U-1,

and the straight line x^2=ξ(U) x^1 is a switching line. When U→∞,

    lim_U→∞ξ(U)=β(1+c)/cγ-1=β+β c-γ c/γ c,

and we finish with exactly the optimal impulse strategy presented in Subsection <ref>.


There are many other sensible optimal control problems in mathematical epidemiology. For example, one can consider immunization of susceptibles. Such problem for the model (<ref>) was solved in <cit.>, but again in the framework of gradual dynamic control, where the term -u(t)x^1(t) appears in the first equation of (<ref>). No doubt, the impulse version of immunization can also be tackled using the methods developed in the current paper.



§ CONCLUSION


Application of the MDP methods to the purely deterministic optimal impulse control problem results in the integral optimality equation. After that, a formal analytical proof shows that the integral and differential forms are equivalent. All the theory is illustrated by a meaningful example on the SIR epidemic.

Note that Theorem <ref> remains also valid in the case when the underlying process is a Piecewise Deterministic Markov Process. To be specific, consider the discounted version of the positive model with the state space Y=^d, the uncontrolled flow, and the uncontrolled fixed jumps intensity λ. Under the mild  relevant conditions, the integral equation (<ref>) was obtained in <cit.>; it has the form

    V_ Y(y)   =   inf_θ∈^0_+{∫_(0,θ] e^-α u[ C^g_ Y(ϕ_ Y(y,u))
    +λ[∫_ Y V_ Y(z)Q(dz|ϕ_ Y(y,u))-V_ Y(ϕ_ Y(y,u))]].du
          +.  I{θ<+∞}e^-αθinf_a∈ A{C^I_ Y(ϕ_ Y(y,θ),a)+V_ Y(l_ Y(ϕ_ Y(y,θ),θ))}},

where Q is the stochastic kernel describing the distribution after the spontaneous (natural) jumps with intensity λ>0. Here, we follow the notations introduced for the discounted model in Section <ref>, which also appeared in Subsection <ref>.

Suppose a measurable along the flow function V_ Y: Y→  is the minimal positive solution to equation (<ref>) and satisfies the corresponding discounted version of Condition <ref>. One can show that, if C^g_ Y(y)≤ K<∞, then the integral ∫_(0,+∞) e^-α t V_ Y(ϕ_ Y(y,t)) dt is finite for all y∈ Y. Denote

    D(y)λ[∫_ Y V_ Y(z) Q(dz|y)-V_ Y(y)].

Then, by Theorem <ref>, V_ Y(y) satisfies the discounted version of the differential equation (<ref>) with C^g_ Y(y) being replaced by C^g_ Y(y)+ D(y). Similarly, one can show that the differential equation (<ref>) and Conditions <ref>-<ref> imply the integral equation (<ref>) and Condition <ref>.

To summarize, the current paper can be a starting point for the rigorous investigation of different types of the optimality equation for impulsively controlled PDMP.



§ ACKNOWLEDGEMENTS


This research was supported by the Royal Society International Exchanges award IE160503, by FCT and CIDMA within project UID/MAT/04106/2013, and  by TOCCATA FCT project PTDC/EEI-AUT/2933/2014.



§ APPENDIX
 

Proof of Proposition <ref>.

The function V(·) is lower semicontinuous by Theorem <ref>. Then the function G(·) is lower semicontinuous, as seen in the proof of Theorem <ref>. By Proposition 7.32 of <cit.>, for each x∈X, inf_a∈ A G(x,θ,a) defines a lower semicontinuous function on ^0_+, and thus Θ(x) is closed and thus compact in ^0_+. The nonemptyness of Θ(x) is by Theorem <ref>.

By Proposition D.5 of <cit.>, inf_a∈ A G(x,θ,a) defines a measurable function on X×^0_+.  Then the graph of the multifunction Θ(·), given by {(x,θ)∈X×^0_+: inf_a∈ A G(x,θ,a) =V(x)}, is measurable and hence the multifunction Θ(·) is Borel-measurable by Proposition D.4 of <cit.>.
By Proposition D.5 of <cit.>, θ^∗(x)=inf_θ∈Θ(x)θ defines a measurable function on X.  


Proof of Proposition <ref>.

According to Theorem <ref> and inequalities 0≤ V(x)≤ K, it is sufficient to show only the uniqueness, namely, we will show that if V is a bounded lower semicontinuous solution to (<ref>), then V= V.

Consider the obvious formula

    V(x_0,π)    =    V(x_0)+lim_N→∞∑_i=1^N E^π_x_0[C̃(X_i-1,(Θ_i,A_i)). 
          .+∫_ X V(y) Q(dy|X_i-1,(Θ_i,A_i))-V(X_i-1)]-lim_N→∞ E^π_x_0[V(X_N)]

valid for each strategy π such that

    V(x_0,π)≤ K.

Note that, for such strategies,

    E^π_x_0[T_stop]=∑_i=1^∞ P^π_x_0(T_stop≥ i)≤K/δ,

so that lim_N→∞ P^π_x_0(T_stop≥ N)=0
and

    lim_N→∞ E^π_x_0[V(X_N)]≤sup_x∈ X V(x)lim_N→∞ P^π_x_0(T_stop≥ N)=0.

Now the stationary deterministic strategy (φ^*_θ,φ^*_a), providing the infimum in (<ref>), is uniformly optimal and

    V^*(x_0)=inf_π V(x_0,π)= V(x_0,(φ^*_θ,φ^*_a))=V(x_0)

because all the other strategies except for those satisfying (<ref>) cannot give smaller value for V(x_0,π).



Proof of Proposition <ref>.

It suffices to prove that for all c>0 the function g(s)  h(s) +cs is nondecreasing on [0,t]∩^0_+.

Note that for any s ∈ (0,  t) there exists ε =  ε_s such that



(i) either g(s) < g(σ) for all σ∈ (s,  s + ε),



(ii) or g(σ) < g(s) for all σ∈ (s - ε,  s).



Suppose that g(s_1) > g(s_2) for some 0 ≤ s_1 < s_2 ≤ t. Our aim is to come to a contradiction.

Take y ∈ (g(s_2),  g(s_1)) and take s_* = inf A, where A = { s ∈ [s_1,  s_2] : g(s) < y }. Note that A contains s_2, and therefore is nonempty.

If s_* = s_1, then on each interval [s_1,s_1+δ] there are points from A, so that

    lim_s → s_1^+  g(s) ≤ y < g(s_1),

in contradiction with the right lower semicontinuity of g at s_1.

If s_* = s_2, we have

    g(s_2) < y ≤lim_s → s_2^-  g(s) ≤lim_s → s_2^-  g(s),

in contradiction with the left upper semicontinuity of g at s_2.

It follows that s_1 < s_* < s_2.



For all s ∈ [s_1,  s_*) we have g(s) ≥ y, therefore

    y ≤lim_s → s_*^-  g(s) ≤lim_s → s_*^-  g(s) ≤ g(s_*)

because g is left upper semicontinuous at s_*.
On the other hand, there exists a sequence { s_k }⊂ A converging to s_*. If at least one term of this sequence coincides with s_* then g(s_*) < y. If, otherwise, no terms of the sequence coincide with s_* then all s_k > s_* and, for all k, g(s_k)<y. Since g is right lower semicontinuous at s_*,

    g(s_*) ≤lim_s → s_*^+  g(s) ≤lim_k →∞  g(s_k) ≤ y.

It follows from (<ref>) and (<ref>) that g(s_*) = y, and both conditions (i), (ii) are violated for s_*: for all s∈[s_1,s_*], g(s)≥ g(s_*) and in each right neighbourhood of s there are points s∈ A such that g(s_*)>g(s). 


Proof of Proposition <ref>.

One easily sees that h is continuous on [0,  t]. Both h and -h satisfy the conditions of Proposition <ref>, therefore both h and -h are nondecreasing. It follows that h is constant. 

 Proof of Proposition <ref>.
Condition <ref> follows from Condition <ref>. According to Theorem <ref>, the bounded lower semicontinuous non-negative function V satisfies the Bellman equation (<ref>). The function V^* is bounded  because of Conditions <ref>  and <ref>. According to Remark <ref>, the Bellman equation cannot have another bounded lower semicontinuous solution. Therefore, V= V^* is the minimal ^0_+-valued solution to equation (<ref>) and the strategy (φ^*_θ,φ^*_a) is uniformly optimal by Theorem <ref>.


99

b14 Abakuks, A., Optimal isolation policy for an epidemic, J. Appl. Prob., 10 (1973), 247–262.

b1 Avrachenkov, K., Habachi, O., Piunovskiy, A.  and   Zhang, Y., 	Infinite horizon optimal impulsive control with applications to Internet congestion control, Intern. J. of Control, 88  (2015), 703–716.

MR0452809
Ball, F.G. and  O'Neill, P.D., A modification of the general stochasic epidemic motivated by AIDS modelling, Advances in Applied Probability, 25 (1993), 39–62.

Bertsekas:1978 Bertsekas, D. and Shreve, S. (1978). Stochastic Optimal Control. Academic Press, New York.

b2 Claeys, M., Arzelier, D., Henrion, D.  and  Lasserre, J-B., Measures and LMIs for impulsive nonlinear optimal control, IEEE Trans, on Autom. Control, 59 (2014), no. 5, 1374–1379.

MR2199469
Clancy, D and Piunovskiy, A.B.,
An explicit optimal isolation policy for a deterministic epidemic model,
Appl. Math. Comput., 163 (2005), no. 3, 1109–1121.

b3 Costa, O.L.V.  and  Raymundo, C.A.B., Impulse and continuous control of piecewise deterministic Markov processes, Stochastics Stochastics Rep., 70 (2000), (1-2) 75–107.

b4 de Saporta, B., Dufour, F.  and  Geeraert, A., Optimal strategies for impulsive control of piecewise deterministic Markov processes, Automatica, 77 (2017), 219–229.

b5 Dempster, M.A.H.  and  Ye, J.J., Impulse control of pieciwise deterministic Markov processes, Ann. Appl. Probab., 5 (1995), no. 2, 399–423.

b6 Dufour, F.  and  Piunovskiy, A., 	Impulsive control for continuous-time Markov decision processes, Adv. Appl. Prob., 47 (2015) 106–127.

b7 Dufour, F., Horiguchi, M.  and  Piunovskiy, A., Optimal impulsive control of piecewise deterministic Markov processes, Stochastics, 88 (2016), no. 7, 1073–1098.

b10 Dykhta, V. and Samsonyuk, O.N. (2000). Optimal Impulsive Control with Applications, Fizmatlit "Nauka", Moscow (Russian).

MR0945530
Gleissner, W.,
The spread of epidemics,
Appl. Math. Comput., 27 (1988), no. 2, 167–171.

las96 Hernández-Lerma, O. and Lasserre, J. (1996). Discrete-Time Markov Control Processes, Springer-Verlag, New York.

las99 Hernández-Lerma, O. and Lasserre, J. (1999). Further Topics in Discrete-Time Markov Control Processes, Springer-Verlag, New York.

b11 Hou, S.H. and Wong, K.H., Optimal impulsive control problem with application to human immunodeficiency virus treatment, J. Optim. Theory Appl., 151 (2011), 385–401.

b15 Jack, A., Johnson, T.C. and Zervos, M., A singular control model with application to the goodwill problem, Stochastic Process. Appl., 118 (2008), no. 11, 2098-–2124

b13 Kyriakidis, E.G. and Pavitsos, A., On the optimal control of a multidimensional simple epidemic process, Math. Scientist, 32 (2007), 118–126.

b8 Leander, R., Lenhart, S.  and  Protopopescu, V., Optimal control of continuous systems with impulse controls, Optim. Control Appl. Meth., 36 (2015), 535–549.

b16 Menaldi, J.L. and Robin, M., On some impulse control problems with constraint, SIAM J. Control Optim., 55 (2017), no. 5, 3204–3225.

MR2478635
Piunovskiy, A.B. and Clancy, D.,
An explicit optimal intervention policy for a deterministic epidemic model,
Optimal Control Appl. Methods, 29 (2008), no. 6, 413–428.

b12  Wei, J., Yang, H. and Wang, R., Classical and impulse control for the optimization of dividend and proportional reinsurance policies with regime switching, J. Optim. Theory Appl., 147 (2011), 358–377.

b9 Yushkevich, A.A., Verification theorems for Markov decision processes with controlled deterministic drift and gradual and impulsive controls, Teor. Probab. Appl., 34 (1989), no. 3, 474–496.



