Nathalie Peyrard is Senior Scientist at INRAE, Toulouse, France. Her research includes computational statistics in models with latent variables, with applications in ecology.Benoîte de Saporta is Professor of Applied Mathematics at the University of Montpellier, France. Her research includes applied probability (Markov processes, optimal stochastic control) and statistics (inference for partially hidden processes).

• Introduction xiBenoîte DE SAPORTA, Jean-Baptiste DURAND, Alain FRANC and Nathalie PEYRARDChapter 1 Monochain HSMM 1Jean-Baptiste DURAND, Alain FRANC, Nathalie PEYRARD, Nicolas VERGNE and Irene VOTSI1.1. Introduction 11.2. HSMM framework 21.2.1. Intuitive presentation with the Squirrel toy example 21.2.2. General HSMM framework 41.2.3. Standard HSMM 71.2.4. Explicit duration HMM 101.2.5. Hmm 111.3. Inferential topics for HSMMs 111.3.1. Likelihood evaluation 121.3.2. Asymptotic properties of the MLE 151.3.3. EM algorithm 161.3.4. Smoothing and filtering probabilities 191.3.5. State restoration 201.4. Two toy examples reappearing throughout the book 221.4.1. Squirrel toy example 221.4.2. Deer toy example 231.5. Reliability 241.5.1. Rate of occurrence of failures 251.5.2. Mean time to failure 261.6. Introducing mixed effects into HSMMs 271.6.1. Mixed HSMMs explained with the Squirrel example 281.6.2. Mixed effects for real-valued observations with the Deer example 321.6.3. Dynamic covariates: toward an alternative representation of HSMMs 331.6.4. Model selection issues for fixed and random effects 351.6.5. Mixed models in the HMM/HSMM literature 351.7. Conclusion/discussion 381.8. Notations 391.9. Acknowledgments 401.10. Appendix: EM algorithm for a monochain HMM 401.10.1. E step 411.10.2. M step 421.11. References 43Chapter 2 Review of HSMM R and Python Softwares 47Caroline BÉRARD, Marie-Josée CROS, Jean-Baptiste DURAND, Corentin LOTHODÉ, Sandra PLANCADE, Ronan TRÉPOS and Nicolas VERGNE2.1. Introduction 472.2. Software around HSMMs: state of the art 482.2.1. R packages 492.2.2. Python packages 542.2.3. Other relevant software 572.3. Comparative overview: R and Python packages for HSMM 622.3.1. General comparison 622.3.2. Sojourn durations 632.3.3. Observations 652.4. Illustration of the use of two packages for the toy examples 662.4.1. Docker image 662.4.2. Python package edhsmm on toy model Squirrel 672.4.3. R package hhsmm on deers 712.5. Conclusion 752.6. References 75Chapter 3 Multichain HMM 79Hanna BACAVE, Jean-Baptiste DURAND, Alain FRANC, Nathalie PEYRARD, Sandra PLANCADE and Régis SABBADIN3.1. Introduction 793.2. Different concepts of MHMM 813.2.1. General MHMM 813.2.2. MHMM with conditional independencies 833.2.3. Case 1 of MHMM-CI: 1to1-MHMM-CI 853.2.4. Case 2 of MHMM-CI: FHMM 883.3. Examples of models of class 1to1-MHMM-CI 903.3.1. Structures obtained by coupling 913.3.2. Applications 933.4. Metapopulation dynamics and MHMM 963.5. Parameter inference in MHMMs with the EM algorithm 983.5.1. Case of general MHMMs 1003.5.2. Case of 1to1-MHMM-CI 1003.5.3. Case of FHMM 1053.5.4. MHMM parameterization for continuous observations 1053.6. Approximate inference in MHMMs 1063.6.1. State of the art of approximate inference for CHMMs 1083.6.2. State of the art of approximate inference for FHMMs 1103.7. Discussion and conclusion 1113.8. Notations 1133.9. References 114Chapter 4 Multichain HSMM 117Jean-Baptiste DURAND, Nathalie PEYRARD, Sandra PLANCADE and Régis SABBADIN4.1. Multichain HSMM in literature 1174.2. Formalization of an explicit duration coupled semi-Markov model with interaction at jump events 1184.2.1. Definition based on literal hypotheses 1194.2.2. Generative definition using a time indexed representation 1194.2.3. Graphical representation 1214.3. Definition of coupled SMM classes based on a time-indexed representation 1224.3.1. Limitations of the (Z, E, R) representation 1224.3.2. Hazard rate representation 1234.3.3. Definition and formalization of a class of coupled standard SMMs 1244.3.4. Extension to general semi-Markov property 1324.3.5. Other uses of time-indexed representation 1324.4. Extension of some MHMM classes to semi-Markov framework 1334.4.1. Generative definition of MHSMM classes 1334.4.2. Graphical representation of MHSMM classes 1344.4.3. About inference 1364.5. Discussion and conclusion 1364.6. Notations 1364.7. Appendix: proof of proposition 1 1384.8. References 142Chapter 5 The Forward-backward Algorithm with Matrix Calculus 143Alain FRANC5.1. Introduction 1445.2. UHMDs, with elimination and marginalization algorithms 1455.2.1. Un-normalized heterogeneous Markov-based distribution 1455.2.2. Elimination algorithm 1465.2.3. Marginalization algorithm 1485.3. Complements on the complexity of elimination and marginalization algorithms for an UHMD 1505.3.1. Multichain UHMD 1515.3.2. Sparsity 1525.3.3. Independence between chains in an UHMD 1535.4. Hidden Markov model 1545.4.1. Computing the probability of the observations 1555.4.2. Smoothing and EM algorithm 1565.4.3. Presentation of the general approach 1575.4.4. Sparsity of the transition matrix 1585.5. Multichain hidden Markov models 1595.5.1. General MHMM 1605.5.2. A hierarchy of models 1625.5.3. Correspondence between hidden and observed variables: 1to1-MHMM-CI 1635.6. Hidden semi-Markov models 1665.6.1. General SMM as an MM in calendar time 1675.6.2. General HSMM in calendar time 1685.6.3. Computing the probability of the observations 1695.6.4. Particular cases of HSMM 1705.6.5. Explicit duration hidden Markov model 1705.7. Multichain HSMM 1725.7.1. 1to1-J-MHSMM-CI 1725.7.2. Multichain ED-HMM with conditional independence 1755.7.3. Different geometries of coupling 1765.8. Conclusions and perspectives 1765.9. Notations 1785.10. Acknowledgments 1795.11. Appendix: Viterbi algorithm and most likely state 1795.11.1. UHMD in a commutative semi-ring 1805.11.2. Setting the problem 1815.11.3. Computing the probability of the most likely state 1825.11.4. Recovering the most likely state 1835.12. References 184Chapter 6 Controlled Hidden Semi-Markov Models 185Alice CLEYNEN, Benoîte DE SAPORTA, Orlane ROSSINI, Régis SABBADIN and Amélie VERNAY6.1. Introduction 1856.2. Markov decision processes 1866.2.1. MDP definition 1876.2.2. Control for MDPs 1886.2.3. Partially observed Markov decision processes 1946.2.4. Solution algorithms for MDPs and POMDPs 1996.3. Piecewise deterministic Markov processes 2006.3.1. PDMP definition 2006.3.2. Impulse control for PDMPs 2106.4. Controlled PDMPs as members of the MDP family 2156.4.1. Controlled PDMPs as MDPs 2166.4.2. Partially observed controlled PDMPs as POMDPs 2206.5. Concluding remarks and open questions 2226.5.1. Open questions in impulse control of PDMPs that might be tackled from the MDP perspective 2226.5.2. Interesting questions in MDPs arising from converted PDMPs 2236.6. Notations 2236.7. Acknowledgments 2256.8. References 226List of Authors 231Index 233