Martin L. Puterman is Professor Emeritus at the Sauder School of Business, University of British Columbia. He received the INFORMS Lanchester Prize for his widely cited 1994 book Markov Decision Processes. He is an INFORMS Fellow and has received the CORS Award of Merit, the CORS Practice Prize and the INFORMS Case Competition Award. Timothy C. Y. Chan is Associate Vice-President and Vice-Provost, Strategic Initiatives and Professor of Industrial Engineering at the University of Toronto. He is an award-winning teacher, having been recognized with the INFORMS Prize for Teaching of OR/MS Practice, the INFORMS Case Competition Award, and the University of Toronto President's Teaching Award.

'This book invites a new generation of students, researchers, and practitioners into the thrilling world of sequential decision making. The mathematical exposition of both the foundations and cutting-edge ideas is both intuitive and rigorous.' Mykel Kochenderfer, Stanford University

• Preface; 1. Introduction; Part I. Fundamentals: 2. Markov decision process fundamentals; 3. Examples and applications; Part II. Classical Markov Decision Process Models: 4. Finite horizon models; 5. Infinite horizon models: expected discounted reward; 6. Infinite horizon models: expected total reward; 7. Infinite horizon models: long-run average reward; 8. Partially observable Markov decision processes; Part III. Reinforcement Learning: 9. Value function approximation; 10. Simulation in tabular models; 11. Simulation with function approximation; Appendix A. Notation and conventions; Appendix B. Markov chains; Appendix C. Linear programming; Bibliography; Index.

Martin L. Puterman is Professor Emeritus at the Sauder School of Business, University of British Columbia. He received the INFORMS Lanchester Prize for his widely cited 1994 book Markov Decision Processes. He is an INFORMS Fellow and has received the CORS Award of Merit, the CORS Practice Prize and the INFORMS Case Competition Award. Timothy C. Y. Chan is Associate Vice-President and Vice-Provost, Strategic Initiatives and Professor of Industrial Engineering at the University of Toronto. He is an award-winning teacher, having been recognized with the INFORMS Prize for Teaching of OR/MS Practice, the INFORMS Case Competition Award, and the University of Toronto President's Teaching Award.

'This book invites a new generation of students, researchers, and practitioners into the thrilling world of sequential decision making. The mathematical exposition of both the foundations and cutting-edge ideas is both intuitive and rigorous.' Mykel Kochenderfer, Stanford University

• Preface; 1. Introduction; Part I. Fundamentals: 2. Markov decision process fundamentals; 3. Examples and applications; Part II. Classical Markov Decision Process Models: 4. Finite horizon models; 5. Infinite horizon models: expected discounted reward; 6. Infinite horizon models: expected total reward; 7. Infinite horizon models: long-run average reward; 8. Partially observable Markov decision processes; Part III. Reinforcement Learning: 9. Value function approximation; 10. Simulation in tabular models; 11. Simulation with function approximation; Appendix A. Notation and conventions; Appendix B. Markov chains; Appendix C. Linear programming; Bibliography; Index.