Federico Girosi is a senior policy researcher at the RAND Corporation. Gary King is the David Florence Professor of Government, and director of the Institute for Quantitative Social Science, at Harvard University.

"A substantial contribution in terms of ideas and methods. This book introduces a new methodology for forecasting mortality that takes into account important predictors, formalizes the use of prior knowledge such as expert opinion, and produces estimates of forecasting uncertainty, using a Bayesian statistical framework. It also provides convincing evidence that the new methods can outperform some of the best alternatives available. The techniques have wide applicability."—Germán Rodríguez, Princeton University

• List of Figures xi List of Tables xiii Preface xv Acknowledgments xvii Chapter 1: Qualitative Overview 1 1.1 Introduction 1 1.2 Forecasting Mortality 3 1.2.1 The Data 3 1.2.2 The Patterns 5 1.2.3 Scientific versus Optimistic Forecasting Goals 8 1.3 Statistical Modeling 11 1.4 Implications for the Bayesian Modeling Literature 15 1.5 Incorporating Area Studies in Cross-National Comparative Research 16 1.6 Summary 18 Part I Existing Methods for Forecasting Mortality 19 Chapter 2: Methods without Covariates 21 2.1 Patterns in Mortality Age Profiles 22 2.2 A Unified Statistical Framework 24 2.3 Population Extrapolation Approaches 25 2.4 Parametric Approaches 26 2.5 A Nonparametric Approach: Principal Components 28 2.5.1 Introduction 28 2.5.2 Estimation 32 2.6 The Lee-Carter Approach 34 2.6.1 The Model 34 2.6.2 Estimation 36 2.6.3 Forecasting 36 2.6.4 Properties 38 2.7 Summary 42 Chapter 3: Methods with Covariates 43 3.1 Equation-by-Equation Maximum Likelihood 43 3.1.1 Poisson Regression 43 3.1.2 Least Squares 44 3.1.3 Computing Forecasts 46 3.1.4 Summary Evaluation 47 3.2 Time-Series, Cross-Sectional Pooling 48 3.2.1 The Model 48 3.2.2 Postestimation Intercept Correction 49 3.2.3 Summary Evaluation 49 3.3 Partially Pooling Cross Sections via Disturbance Correlations 50 3.4 Cause-Specific Methods with Microlevel Information 51 3.4.1 Direct Decomposition Methods 51 Modeling 51 3.4.2 Microsimulation Methods 52 3.4.3 Interpretation 53 3.5 Summary 53 Part II Statistical Modeling 55 Chapter 4: The Model 57 4.1 Overview 57 4.2 Priors on Coefficients 59 4.3 Problems with Priors on Coefficients 60 4.3.1 Little Direct Prior Knowledge Exists about Coefficients 61 4.3.2 Normalization Factors Cannot Be Estimated 62 4.3.3 We Know about the Dependent Variable, Not the Coefficients 64 4.3.4 Difficulties with Incomparable Covariates 65 4.4 Priors on the Expected Value of the Dependent Variable 65 4.4.1 Step 1: Specify a Prior for the Dependent Variable 66 4.4.2 Step 2: Translate to a Prior on the Coefficients 67 4.4.3 Interpretation 68 4.5 A Basic Prior for Smoothing over Age Groups 69 4.5.1 Step 1: A Prior for 69 4.5.2 Step 2: From the Prior on to the Prior on ss 71 4.5.3 Interpretation 71 4.6 Concluding Remark 73 Chapter 5: Priors over Grouped Continuous Variables 74 5.1 Definition and Analysis of Prior Indifference 74 5.1.1 A Simple Special Case 76 5.1.2 General Expressions for Prior Indifference 76 5.1.3 Interpretation 77 5.2 Step 1: A Prior for 80 5.2.1 Measuring Smoothness 81 5.2.2 Varying the Degree of Smoothness over Age Groups 83 5.2.3 Null Space and Prior Indifference 83 5.2.4 Nonzero Mean Smoothness Functional 85 5.2.5 Discretizing: From Age to Age Groups 85 5.2.6 Interpretation 86 5.3 Step 2: From the Prior on to the Prior on ss 92 5.3.1 Analysis 92 5.3.2 Interpretation 92 Chapter 6: Model Selection 94 6.1 Choosing the Smoothness Functional 94 6.2 Choosing a Prior for the Smoothing Parameter 97 6.2.1 Smoothness Parameter for a Nonparametric Prior 98 6.2.2 Smoothness Parameter for the Prior over the Coefficients 100 6.3 Choosing Where to Smooth 104 6.4 Choosing Covariates 108 6.4.1 Size of the Null Space 109 6.4.2 Content of the Null Space 110 6.5 Choosing a Likelihood and Variance Function 112 6.5.1 Deriving the Normal Specification 112 6.5.2 Accuracy of the Log-Normal Approximation to the Poisson 114 6.5.3 Variance Specification 120 Chapter 7: Adding Priors over Time and Space 124 7.1 Smoothing over Time 124 7.1.1 Prior Indifference and the Null Space 125 7.2 Smoothing over Countries 127 7.2.1 Null Space and Prior Indifference 128 7.2.2 Interpretation 130 7.3 Smoothing Simultaneously over Age, Country, and Time 131 7.4 Smoothing Time Trend Interactions 132 7.4.1 Smoothing Trends over Age Groups 133 7.4.2 Smoothing Trends over Countries 133 7.5 Smoothing with General Interactions 134 7.6 Choosing a Prior for Multiple Smoothing Parameters 136 7.6.1 Example 139 7.6.2 Estimating the Expected Value of the Summary Measures 141 7.7 Summary 144 Chapter 8: Comparisons and Extensions 145 8.1 Priors on Coefficients versus Dependent Variables 145 8.1.1 Defining Distances 145 8.1.2 Conditional Densities 147 8.1.3 Connections to "Virtual Examples" in Pattern Recognition 147 8.2 Extensions to Hierarchical Models and Empirical Bayes 148 8.2.1 The Advantages of Empirical Bayes without Empirical Bayes 149 8.2.2 Hierarchical Models as Special Cases of Spatial Models 151 8.3 Smoothing Data without Forecasting 151 8.4 Priors When the Dependent Variable Changes Meaning 153 Part III Estimation 159 Chapter 9: Markov Chain Monte Carlo Estimation 161 9.1 Complete Model Summary 161 9.1.1 Likelihood 162 9.1.2 Prior for ss 162 9.1.3 Prior for si 162 9.1.4 Prior for T 163 9.1.5 The Posterior Density 164 9.2 The Gibbs Sampling Algorithm 164 9.2.1 Sampling s 165 The Conditional Density 165 Interpretation 165 9.2.2 Sampling T 166 The Conditional Density 166 Interpretation 166 9.2.3 Sampling ss 167 The Conditional Density 167 Interpretation 168 9.2.4 Uncertainty Estimates 169 9.3 Summary 169 Chapter 10: Fast Estimation without Markov Chains 170 10.1 Maximum A Posteriori Estimator 170 10.2 Marginal Maximum A Posteriori Estimator 171 10.3 Conditional Maximum A Posteriori Estimator 172 10.4 Summary 173 Part IV Empirical Evidence 175 Chapter 11: Illustrative Analyses 177 11.1 Forecasts without Covariates: Linear Trends 178 11.1.1 Smoothing over Age Groups Only 178 11.1.2 Smoothing over Age and Time 181 11.2 Forecasts without Covariates: Nonlinear Trends 182 11.3 Forecasts with Covariates: Smoothing over Age and Time 187 11.4 Smoothing over Countries 189 Chapter 12: Comparative Analyses 196 12.1 All Causes in Males 197 12.2 Lung Disease in Males 200 12.2.1 Comparison with Least Squares 202 12.2.2 Country-by-Country Analysis 203 12.3 Breast Cancer in Females 205 12.3.1 Comparison with Least Squares 205 12.3.2 Country-by-country Analysis 205 12.4 Comparison on OECD Countries 206 12.4.1 Transportation Accidents in Males 208 12.4.2 Cardiovascular Disease in Males 210 Chapter 13: Concluding Remarks 211 Appendixes 213 A Notation 215 A.1 Principles 215 A.2 Glossary 216 B Mathematical Refresher 219 B.1 Real Analysis 219 B.1.1 Vector Space 219 B.1.2 Metric Space 220 B.1.3 Normed Space 221 B.1.4 Scalar Product Space 222 B.1.5 Functions, Mappings, and Operators 223 B.1.6 Functional 224 B.1.7 Span 224 B.1.8 Basis and Dimension 224 B.1.9 Orthonormality 225 B.1.10 Subspace 225 B.1.11 Orthogonal Complement 226 B.1.12 Direct Sum 226 B.1.13 Projection Operators 227 B.2 Linear Algebra 229 B.2.1 Range, Null Space, Rank, and Nullity 229 B.2.2 Eigenvalues and Eigenvectors for Symmetric Matrices 232 B.2.3 Definiteness 234 B.2.4 Singular Values Decomposition 234 Definition 234 For Approximation 235 B.2.5 Generalized Inverse 236 B.2.6 Quadratic Form Identity 238 B.3 Probability Densities 239 B.3.1 The Normal Distribution 239 B.3.2 The Gamma Distribution 239 B.3.3 The Log-Normal Distribution 240 C Improper Normal Priors 241 C.1 Definitions 241 C.2 An Intuitive Special Case 242 C.3 The General Case 243 C.4 Drawing Random Samples 246 D Discretization of the Derivative Operator 247 E Smoothness over Graphs 249 Bibliography 251 Index 259

Federico Girosi is a senior policy researcher at the RAND Corporation. Gary King is the David Florence Professor of Government, and director of the Institute for Quantitative Social Science, at Harvard University.

"A substantial contribution in terms of ideas and methods. This book introduces a new methodology for forecasting mortality that takes into account important predictors, formalizes the use of prior knowledge such as expert opinion, and produces estimates of forecasting uncertainty, using a Bayesian statistical framework. It also provides convincing evidence that the new methods can outperform some of the best alternatives available. The techniques have wide applicability."—Germán Rodríguez, Princeton University

• List of Figures xi List of Tables xiii Preface xv Acknowledgments xvii Chapter 1: Qualitative Overview 1 1.1 Introduction 1 1.2 Forecasting Mortality 3 1.2.1 The Data 3 1.2.2 The Patterns 5 1.2.3 Scientific versus Optimistic Forecasting Goals 8 1.3 Statistical Modeling 11 1.4 Implications for the Bayesian Modeling Literature 15 1.5 Incorporating Area Studies in Cross-National Comparative Research 16 1.6 Summary 18 Part I Existing Methods for Forecasting Mortality 19 Chapter 2: Methods without Covariates 21 2.1 Patterns in Mortality Age Profiles 22 2.2 A Unified Statistical Framework 24 2.3 Population Extrapolation Approaches 25 2.4 Parametric Approaches 26 2.5 A Nonparametric Approach: Principal Components 28 2.5.1 Introduction 28 2.5.2 Estimation 32 2.6 The Lee-Carter Approach 34 2.6.1 The Model 34 2.6.2 Estimation 36 2.6.3 Forecasting 36 2.6.4 Properties 38 2.7 Summary 42 Chapter 3: Methods with Covariates 43 3.1 Equation-by-Equation Maximum Likelihood 43 3.1.1 Poisson Regression 43 3.1.2 Least Squares 44 3.1.3 Computing Forecasts 46 3.1.4 Summary Evaluation 47 3.2 Time-Series, Cross-Sectional Pooling 48 3.2.1 The Model 48 3.2.2 Postestimation Intercept Correction 49 3.2.3 Summary Evaluation 49 3.3 Partially Pooling Cross Sections via Disturbance Correlations 50 3.4 Cause-Specific Methods with Microlevel Information 51 3.4.1 Direct Decomposition Methods 51 Modeling 51 3.4.2 Microsimulation Methods 52 3.4.3 Interpretation 53 3.5 Summary 53 Part II Statistical Modeling 55 Chapter 4: The Model 57 4.1 Overview 57 4.2 Priors on Coefficients 59 4.3 Problems with Priors on Coefficients 60 4.3.1 Little Direct Prior Knowledge Exists about Coefficients 61 4.3.2 Normalization Factors Cannot Be Estimated 62 4.3.3 We Know about the Dependent Variable, Not the Coefficients 64 4.3.4 Difficulties with Incomparable Covariates 65 4.4 Priors on the Expected Value of the Dependent Variable 65 4.4.1 Step 1: Specify a Prior for the Dependent Variable 66 4.4.2 Step 2: Translate to a Prior on the Coefficients 67 4.4.3 Interpretation 68 4.5 A Basic Prior for Smoothing over Age Groups 69 4.5.1 Step 1: A Prior for 69 4.5.2 Step 2: From the Prior on to the Prior on ss 71 4.5.3 Interpretation 71 4.6 Concluding Remark 73 Chapter 5: Priors over Grouped Continuous Variables 74 5.1 Definition and Analysis of Prior Indifference 74 5.1.1 A Simple Special Case 76 5.1.2 General Expressions for Prior Indifference 76 5.1.3 Interpretation 77 5.2 Step 1: A Prior for 80 5.2.1 Measuring Smoothness 81 5.2.2 Varying the Degree of Smoothness over Age Groups 83 5.2.3 Null Space and Prior Indifference 83 5.2.4 Nonzero Mean Smoothness Functional 85 5.2.5 Discretizing: From Age to Age Groups 85 5.2.6 Interpretation 86 5.3 Step 2: From the Prior on to the Prior on ss 92 5.3.1 Analysis 92 5.3.2 Interpretation 92 Chapter 6: Model Selection 94 6.1 Choosing the Smoothness Functional 94 6.2 Choosing a Prior for the Smoothing Parameter 97 6.2.1 Smoothness Parameter for a Nonparametric Prior 98 6.2.2 Smoothness Parameter for the Prior over the Coefficients 100 6.3 Choosing Where to Smooth 104 6.4 Choosing Covariates 108 6.4.1 Size of the Null Space 109 6.4.2 Content of the Null Space 110 6.5 Choosing a Likelihood and Variance Function 112 6.5.1 Deriving the Normal Specification 112 6.5.2 Accuracy of the Log-Normal Approximation to the Poisson 114 6.5.3 Variance Specification 120 Chapter 7: Adding Priors over Time and Space 124 7.1 Smoothing over Time 124 7.1.1 Prior Indifference and the Null Space 125 7.2 Smoothing over Countries 127 7.2.1 Null Space and Prior Indifference 128 7.2.2 Interpretation 130 7.3 Smoothing Simultaneously over Age, Country, and Time 131 7.4 Smoothing Time Trend Interactions 132 7.4.1 Smoothing Trends over Age Groups 133 7.4.2 Smoothing Trends over Countries 133 7.5 Smoothing with General Interactions 134 7.6 Choosing a Prior for Multiple Smoothing Parameters 136 7.6.1 Example 139 7.6.2 Estimating the Expected Value of the Summary Measures 141 7.7 Summary 144 Chapter 8: Comparisons and Extensions 145 8.1 Priors on Coefficients versus Dependent Variables 145 8.1.1 Defining Distances 145 8.1.2 Conditional Densities 147 8.1.3 Connections to "Virtual Examples" in Pattern Recognition 147 8.2 Extensions to Hierarchical Models and Empirical Bayes 148 8.2.1 The Advantages of Empirical Bayes without Empirical Bayes 149 8.2.2 Hierarchical Models as Special Cases of Spatial Models 151 8.3 Smoothing Data without Forecasting 151 8.4 Priors When the Dependent Variable Changes Meaning 153 Part III Estimation 159 Chapter 9: Markov Chain Monte Carlo Estimation 161 9.1 Complete Model Summary 161 9.1.1 Likelihood 162 9.1.2 Prior for ss 162 9.1.3 Prior for si 162 9.1.4 Prior for T 163 9.1.5 The Posterior Density 164 9.2 The Gibbs Sampling Algorithm 164 9.2.1 Sampling s 165 The Conditional Density 165 Interpretation 165 9.2.2 Sampling T 166 The Conditional Density 166 Interpretation 166 9.2.3 Sampling ss 167 The Conditional Density 167 Interpretation 168 9.2.4 Uncertainty Estimates 169 9.3 Summary 169 Chapter 10: Fast Estimation without Markov Chains 170 10.1 Maximum A Posteriori Estimator 170 10.2 Marginal Maximum A Posteriori Estimator 171 10.3 Conditional Maximum A Posteriori Estimator 172 10.4 Summary 173 Part IV Empirical Evidence 175 Chapter 11: Illustrative Analyses 177 11.1 Forecasts without Covariates: Linear Trends 178 11.1.1 Smoothing over Age Groups Only 178 11.1.2 Smoothing over Age and Time 181 11.2 Forecasts without Covariates: Nonlinear Trends 182 11.3 Forecasts with Covariates: Smoothing over Age and Time 187 11.4 Smoothing over Countries 189 Chapter 12: Comparative Analyses 196 12.1 All Causes in Males 197 12.2 Lung Disease in Males 200 12.2.1 Comparison with Least Squares 202 12.2.2 Country-by-Country Analysis 203 12.3 Breast Cancer in Females 205 12.3.1 Comparison with Least Squares 205 12.3.2 Country-by-country Analysis 205 12.4 Comparison on OECD Countries 206 12.4.1 Transportation Accidents in Males 208 12.4.2 Cardiovascular Disease in Males 210 Chapter 13: Concluding Remarks 211 Appendixes 213 A Notation 215 A.1 Principles 215 A.2 Glossary 216 B Mathematical Refresher 219 B.1 Real Analysis 219 B.1.1 Vector Space 219 B.1.2 Metric Space 220 B.1.3 Normed Space 221 B.1.4 Scalar Product Space 222 B.1.5 Functions, Mappings, and Operators 223 B.1.6 Functional 224 B.1.7 Span 224 B.1.8 Basis and Dimension 224 B.1.9 Orthonormality 225 B.1.10 Subspace 225 B.1.11 Orthogonal Complement 226 B.1.12 Direct Sum 226 B.1.13 Projection Operators 227 B.2 Linear Algebra 229 B.2.1 Range, Null Space, Rank, and Nullity 229 B.2.2 Eigenvalues and Eigenvectors for Symmetric Matrices 232 B.2.3 Definiteness 234 B.2.4 Singular Values Decomposition 234 Definition 234 For Approximation 235 B.2.5 Generalized Inverse 236 B.2.6 Quadratic Form Identity 238 B.3 Probability Densities 239 B.3.1 The Normal Distribution 239 B.3.2 The Gamma Distribution 239 B.3.3 The Log-Normal Distribution 240 C Improper Normal Priors 241 C.1 Definitions 241 C.2 An Intuitive Special Case 242 C.3 The General Case 243 C.4 Drawing Random Samples 246 D Discretization of the Derivative Operator 247 E Smoothness over Graphs 249 Bibliography 251 Index 259