conditions,(for detail check Corollary 3.14 of Continuous-Time Markov Decision Processes), if our optimal value function is independent of state , we will have the following inequality: If there exists a function , then will be the smallest satisfying the above equation. In order to find , we could use the following linear programming model: Primal linear program(P-LP) Dual linear program(D-LP) is a feasible solution to the D-LP if is nonnative and satisfied the constraints in the D-LP problem. A feasible solution to the D-LP is said to be an optimal solution if for all feasible solution to the D-LP. Once we have found the optimal solution , we can use it to establish the optimal policies. Hamilton–Jacobi–Bellman equation In continuous-time MDP, if the state space and action space are continuous, the optimal criterion could be found by solving Hamilton–Jacobi–Bellman (HJB) partial differential equation. In order to discuss the HJB equation, we need to reformulate our problem is the terminal reward function, is the system state vector, is the system control vector we try to find. shows how the state vector changes over time. The Hamilton–Jacobi–Bellman equation is as follows: We could solve the equation to find the optimal control , which could give us the optimal value function Application Continuous-time Markov decision processes have applications in queueing systems, epidemic processes, and population processes. Alternative notations The terminology and notation for MDPs are not entirely settled. There are two main streams — one focuses on maximization problems from contexts like economics, using the terms action, reward, value, and calling the discount factor or , while the other focuses on minimization problems from engineering and navigation, using the terms control, cost, cost-to-go, and calling the discount factor . In addition, the notation for the transition probability varies. In addition, transition probability is sometimes written , or, rarely, Constrained Markov decision processes Constrained Markov decision processes (CMDPs) are extensions to Markov decision process (MDPs). There are three fundamental differences between MDPs and CMDPs. There are multiple costs incurred after applying an action instead of one. CMDPs are solved with linear programs only, and dynamic programming does not work. The final policy depends on the starting state. There are a number of applications for CMDPs. It has recently been used in motion planning scenarios in robotics. See also Probabilistic automata Odds algorithm Quantum finite automata Partially observable Markov decision process Dynamic programming Bellman equation