# Encyclopedia of Systems and Control

2015 Edition
| Editors: John Baillieul, Tariq Samad

# Numerical Methods for Continuous-Time Stochastic Control Problems

• George Yin
Reference work entry
DOI: https://doi.org/10.1007/978-1-4471-5058-9_232

## Abstract

This expository article provides a brief review of numerical methods for stochastic control in continuous time. It concentrates on the methods of Markov chain approximation for controlled diffusions. Leaving most of the technical details out with the broad general audience in mind, it aims to serve as an introductory reference or a user’s guide for researchers, practitioners, and students who wish to know something about numerical methods for stochastic control.

### Keywords

Markov chain approximation Numerical methods Stochastic control
This is a preview of subscription content, log in to check access

## Notes

### Acknowledgements

The research of this author was supported in part by the Army Research Office under grant W911NF-12-1-0223.

### Bibliography

1. Barles G, Souganidis P (1991) Convergence of approximation schemes for fully nonlinear second order equations. J Asymptot Anal 4:271–283
2. Bertsekas DP, Castanon DA (1989) Adaptive aggregation methods for infinite horizon dynamic programming. IEEE Trans Autom Control 34:589–598
3. Chancelier P, Gomez C, Quadrat J-P, Sulem A, Blankenship GL, La Vigna A, MaCenary DC, Yan I (1986) An expert system for control and signal processing with automatic FORTRAN program generation. In: Mathematical systems symposium, Stockholm. Royal Institute of Technology, StockholmGoogle Scholar
4. Chancelier P, Gomez C, Quadrat J-P, Sulem A (1987) Automatic study in stochastic control. In: Fleming W, Lions PL (eds) IMA volume in mathematics and its applications, vol 10. Springer, BerlinGoogle Scholar
5. Crandall MG, Lions PL (1983) Viscosity solutions of Hamilton-Jacobi equations. Trans Am Math Soc 277:1–42
6. Crandall MG, Ishii H, Lions PL (1992) User’s guide to viscosity solutions of second order partial differential equations. Bull Am Math Soc 27:1–67
7. Fleming WH, Rishel RW (1975) Deterministic and stochastic optimal control. Springer, New YorkGoogle Scholar
8. Fleming WH, Soner HM (1992) Controlled Markov processes and viscosity solutions. Springer, New YorkGoogle Scholar
9. Jin Z, Wang Y, Yin G (2011) Numerical solutions of quantile hedging for guaranteed minimum death benefits under a regime-switching-jump-diffusion formulation. J Comput Appl Math 235:2842–2860
10. Jin Z, Yin G, Zhu C (2012) Numerical solutions of optimal risk control and dividend optimization policies under a generalized singular control formulation. Automatica 48:1489–1501
11. Jin Z, Yang HL, Yin G (2013) Numerical methods for optimal dividend payment and investment strategies of regime-switching jump diffusion models with capital injections. Automatica 49:2317–2329
12. Krylov VN (2000) On the rate of convergence of finite-difference approximations for Bellman’s equations with variable coefficients. Probab Theory Relat Fields 117:1–16Google Scholar
13. Kushner HJ (1977) Probability methods for approximation in stochastic control and for elliptic equations. Academic, New YorkGoogle Scholar
14. Kushner HJ (1990a) Weak convergence methods and singularly perturbed stochastic control and filtering problems. Birkhäuser, BostonGoogle Scholar
15. Kushner HJ (1990b) Numiercal methods for stochastic control problems in continuous time. SIAM J Control Optim 28:999–1048
16. Kushner HJ (2000) Consistency issues for numerical methods for variance control with applications to optimization in finance. IEEE Trans Autom Control 44:2283–2296
17. Kushner HJ (2002) Numerical approximations for stochastic differential games. SIAM J Control Optim 40:457–486
18. Kushner HJ (2008) Numerical methods for controlled stochastic delay systems. Birkhäuser, BostonGoogle Scholar
19. Kushner HJ, Dupuis PG (1992) Numerical methods for stochastic control problems in continuous time. Springer, New YorkGoogle Scholar
20. Kushner HJ, Kleinman AJ (1968) Numerical methods for the solution of the degenerate nonlinear elliptic equations arising in optimal stochastic control theory. IEEE Trans Autom Control AC-13:344–353
21. Kushner HJ, Yin G (2003) Stochastic approximation and recursive algorithms and applications, 2nd edn. Springer, New YorkGoogle Scholar
22. Menaldi J (1989) Some estimates for finite difference approximations. SIAM J Control Optim 27:579–607
23. Peng S (1990) A general stochastic maximum principle for optimal control problems. SIAM J Control Optim 28:966–979
24. Sethi SP, Zhang Q (1994) Hierarchical decision making in stochastic manufacturing systems. Birkhäuser, BostonGoogle Scholar
25. Song QS, Yin G (2009) Rates of convergence of numerical methods for controlled regime-switching diffusions with stopping times in the costs. SIAM J Control Optim 48:1831–1857
26. Song QS, Yin G, Zhang Z (2006) Numerical method for controlled regime-switching diffusions and regime-switching jump diffusions. Automatica 42:1147–1157
27. Song QS, Yin G, Zhang Z (2008) Numerical solutions for stochastic differential games with regime switching. IEEE Trans Autom Control 53:509–521
28. Warga J (1962) Relaxed variational problems. J Math Anal Appl 4:111–128
29. Yan HM, Yin G, Lou SXC (1994) Using stochastic optimization to determine threshold values for control of unreliable manufacturing systems. J Optim Theory Appl 83:511–539
30. Yin G, Zhang Q (2013) Continuous-time Markov chains and applications: a two-time-scale approach, 2nd edn. Springer, New YorkGoogle Scholar
31. Yin G, Zhu C (2010) Hybrid switching diffusions: properties and applications. Springer, New YorkGoogle Scholar
32. Yin G, Liu RH, Zhang Q (2002) Recursive algorithms for stock liquidation: a stochastic optimization approach. SIAM J Optim 13:240–263
33. Yin G, Jin H, Jin Z (2009) Numerical methods for portfolio selection with bounded constraints. J Comput Appl Math 233:564–581
34. Yong J, Zhou XY (1999) Stochastic controls: Hamiltonian systems and HJB equations. Springer, New YorkGoogle Scholar
35. Zhang J (2006) Rate of convergence of finite difference approximations for degenerate ODEs. Math Comput 75:1755–1778Google Scholar
{"extraction_info": {"found_math": false, "script_math_tex": 0, "script_math_asciimath": 0, "math_annotations": 0, "math_alttext": 0, "mathml": 0, "mathjax_tag": 0, "mathjax_inline_tex": 0, "mathjax_display_tex": 0, "mathjax_asciimath": 0, "img_math": 0, "codecogs_latex": 0, "wp_latex": 0, "mimetex.cgi": 0, "/images/math/codecogs": 0, "mathtex.cgi": 0, "katex": 0, "math-container": 0, "wp-katex-eq": 0, "align": 0, "equation": 0, "x-ck12": 0, "texerror": 0, "math_score": 0.8541165590286255, "perplexity": 11447.030014563847}, "config": {"markdown_headings": true, "markdown_code": true, "boilerplate_config": {"ratio_threshold": 0.18, "absolute_threshold": 10, "end_threshold": 15, "enable": true}, "remove_buttons": true, "remove_image_figures": true, "remove_link_clusters": true, "table_config": {"min_rows": 2, "min_cols": 3, "format": "plain"}, "remove_chinese": true, "remove_edit_buttons": true, "extract_latex": true}, "warc_path": "s3://commoncrawl/crawl-data/CC-MAIN-2018-09/segments/1518891815951.96/warc/CC-MAIN-20180224211727-20180224231727-00235.warc.gz"}