Computational Statistics in Data Science

WALTER W. PIEGORSCH is Professor of Mathematics at the University of Arizona and Director of Statistical Research & Education at the University’s BIO5 Institute. He is also a former Chair of the UArizona Interdisciplinary Program in Statistics, and a past editor of the Journal of the American Statistical Association (Theory & Methods Section). He is a fellow of the American Statistical Association and an elected member of the International Statistical Institute. RICHARD A. LEVINE is Professor of Statistics at San Diego State University and Faculty Advisor overseeing the Statistical Modeling Group in SDSU Analytic Studies and Institutional Research. He is former Chair of the SDSU Department of Mathematics and Statistics and past Editor of the Journal of Computational and Graphical Statistics. He is Associate Editor for Statistics of the Notices of the American Mathematical Society and is a fellow of the American Statistical Association. HAO HELEN ZHANG is Professor of Mathematics at the University of Arizona and Chair of the UArizona Interdisciplinary Program in Statistics. She is Editor-in-Chief of STAT (the ISI journal) and Associate Editor of the Journal of the American Statistical Association and the Journal of the Royal Statistical Society. She is a fellow of the American Statistical Association, the Institute of Mathematical Statistics, and an elected member of the International Statistical Institute. THOMAS C. M. LEE is Professor of Statistics and Associate Dean of the Faculty in Mathematical and Physical Sciences at the University of California, Davis. He is a former Chair of the Department of Statistics at the same institution and a past editor of the Journal of Computational and Graphical Statistics. He is an elected fellow of the American Association for the Advancement of Science, the American Statistical Association, and the Institute of Mathematical Statistics.

• List of Contributors xxiiiPreface xxix Part I Computational Statistics and Data Science 11 Computational Statistics and Data Science in the Twenty-first Century 3Andrew J. Holbrook, Akihiko Nishimura, Xiang Ji, and Marc A. Suchard1 Introduction 32 Core Challenges 1–3 53 Model-Specific Advances 84 Core Challenges 4 and 5 125 Rise of Data Science 16  2 Statistical Software 23Alfred G. Schissler and Alexander D. Knudson1 User Development Environments 232 Popular Statistical Software 263 Noteworthy Statistical Software and Related Tools 304 Promising and Emerging Statistical Software 365 The Future of Statistical Computing 386 Concluding Remarks 39 3 An Introduction to Deep Learning Methods 43Yao Li, Justin Wang and Thomas C.M. Lee1 Introduction 432 Machine Learning: An Overview 433 Feedforward Neural Networks 454 Convolutional Neural Networks 485 Autoencoders 526 Recurrent Neural Networks 547 Conclusion 57  4 Streaming Data and Data Streams 59Taiwo Kolajo, Olawande Daramola, and Ayodele Adebiyi1 Introduction 592 Data Stream Computing 613 Issues in Data Stream Mining 614 Streaming Data Tools and Technologies 645 Streaming Data Pre-Processing: Concept and Implementation 656 Streaming Data Algorithms 657 Strategies for Processing Data Streams 688 Best Practices for Managing Data Streams 699 Conclusion and theWay Forward 70 Part II Simulation-Based Methods 795 Monte Carlo Simulation: Are We There Yet? 81Dootika Vats, James M. Flegal, and Galin L. Jones1 Introduction 812 Estimation 833 Sampling Distribution 844 Estimating Σ 875 Stopping Rules 886 Workflow 897 Examples 90  6 Sequential Monte Carlo: Particle Filters and Beyond 99Adam M. Johansen1 Introduction 992 Sequential Importance Sampling and Resampling 993 SMC in Statistical Contexts 1064 Selected Recent Developments 1127 Markov Chain Monte Carlo Methods, A Survey with Some Frequent Misunderstandings 119Christian P. Robert and Wu Changye1 Introduction 1192 Monte Carlo Methods 1213 Markov Chain Monte Carlo Methods 1284 Approximate Bayesian Computation 1415 Further Reading 145 8 Bayesian Inference with Adaptive Markov Chain Monte Carlo 151Matti Vihola1 Introduction 1512 Random-Walk Metropolis Algorithm 1513 Adaptation of Random-Walk Metropolis 1524 Multimodal Targets with Parallel Tempering 1565 Dynamic Models with Particle Filters 1576 Discussion 159 9 Advances in Importance Sampling 165Víctor Elvira and Luca Martino1 Introduction and Problem Statement 1652 Importance Sampling 1673 Multiple Importance Sampling (MIS) 1714 Adaptive Importance Sampling (AIS) 174 Part III Statistical Learning 183  10 Supervised Learning 185Weibin Mo and Yufeng Liu1 Introduction 1852 Penalized Empirical Risk Minimization 1863 Linear Regression 1904 Classification 1935 Extensions for Complex Data 2006 Discussion 203 11 Unsupervised and Semisupervised Learning 209Jia Li and Vincent A. Pisztora1 Introduction 2092 Unsupervised Learning 2103 Semisupervised Learning 2194 Conclusions 224 12 Random Forest 231Peter Calhoun, Xiaogang Su, Kelly M. Spoon, Richard A. Levine, and Juanjuan Fan1 Introduction 2312 Random Forest (RF) 2323 Random Forest Extensions 2354 Random Forests of Interaction Trees (RFIT) 2395 Random Forest of Interaction Trees for Observational Studies 2436 Discussion 249 13 Network Analysis 253Rong Ma and Hongzhe Li1 Introduction 2532 Gaussian Graphical Models for Mixed Partial Compositional Data 2553 Theoretical Properties 2574 Graphical Model Selection 2605 Analysis of a Microbiome–Metabolomics Data 2606 Discussion 261 14 Tensors in Modern Statistical Learning 269Will Wei Sun, Botao Hao, and Lexin Li1 Introduction 2692 Background2703 Tensor Supervised Learning 2724 Tensor Unsupervised Learning 2765 Tensor Reinforcement Learning 2826 Tensor Deep Learning 286 15 Computational Approaches to Bayesian Additive Regression Trees 297Hugh Chipman, Edward George, Richard Hahn, Robert McCulloch, Matthew Pratola, and Rodney Sparapani1 Introduction 2972 Bayesian CART 2983 TreeMCMC3024 The BART Model 3085 BART Example: Boston Housing Values and Air Pollution 3106 BARTMCMC3117 BART Extentions 3138 Conclusion 320  Part IV High-Dimensional Data Analysis 32316 Penalized Regression 325Seung Jun Shin and Yichao Wu1 Introduction 3252 Penalization for Smoothness 3263 Penalization for Sparsity 3284 Tuning Parameter Selection 330 17 Model Selection in High-Dimensional Regression 333Hao H. Zhang1 Model Selection Problem 3332 Model Selection in High-Dimensional Linear Regression 3353 Interaction-Effect Selection for High-Dimensional Data 3394 Model Selection in High-Dimensional Nonparametric Models 3425 Concluding Remarks 34918 Sampling Local Scale Parameters in High-Dimensional Regression Models 355Anirban Bhattacharya and James E. Johndrow1 Introduction 3552 A Blocked Gibbs Sampler for the Horseshoe 3563 Sampling (𝜉, 𝜎2, 𝛽) 3594 Sampling 𝜂 3605 Appendix: A. Newton–Raphson Steps for the Inverse-cdf Sampler for 𝜂 367 19 Factor Modeling for High-Dimensional Time Series 371Chun Yip Yau1 Introduction 3712 Identifiability 3723 Estimation of High-Dimensional Factor Model 3734 Determining the Number of Factors 383  Part V Quantitative Visualization 387  20 Visual Communication of Data: It Is Not a Programming Problem, It Is Viewer Perception 389Edward Mulrow and Nola du Toit1 Introduction 3892 Case Studies Part 1 3913 Let StAR Be Your Guide 3934 Case Studies Part 2: Using StAR Principles to Develop Better Graphics 3945 Ask Colleagues Their Opinion 3976 Case Studies: Part 3 3987 Iterate 4018 Final Thoughts 402 21 Uncertainty Visualization 405Lace Padilla, Matthew Kay, and Jessica Hullman1 Introduction 4052 Uncertainty Visualization Theories 408 3 General Discussion 42022 Big Data Visualization 427Leland Wilkinson1 Introduction 4272 Architecture for Big Data Analytics 4283 Filtering4304 Aggregating 4305 Analyzing 436 6 Big Data Graphics 4367 Conclusion 440 23 Visualization-Assisted Statistical Learning 443Catherine B. Hurley and Katarina Domijan1 Introduction 4432 Better Visualizations with Seriation 4443 Visualizing Machine Learning Fits 4454 Condvis2 Case Studies 4475 Discussion 453 24 Functional Data Visualization 457Marc G. Genton and Ying Sun1 Introduction 4572 Univariate Functional Data Visualization 4583 Multivariate Functional Data Visualization 4614 Conclusions 465 Part VI Numerical Approximation and Optimization 46925 Gradient-Based Optimizers for Statistics and Machine Learning 471Cho-Jui Hsieh1 Introduction 4712 Convex Versus Nonconvex Optimization 4723 Gradient Descent 4734 Proximal Gradient Descent: Handling Nondifferentiable Regularization 4755 Stochastic Gradient Descent 476 26 Alternating Minimization Algorithms 481David R. Hunter1 Introduction 4812 Coordinate Descent 4823 EM as Alternating Minimization 4843.1 Finite Mixture Models 4854 Matrix Approximation Algorithms 4865 Conclusion 489 27 A Gentle Introduction to Alternating Direction Method of Multipliers (ADMM) for Statistical Problems 493  Shiqian Ma and Mingyi Hong1 Introduction 4932 Two Perfect Examples of ADMM 4943 Variable Splitting and Linearized ADMM 4964 Multiblock ADMM 4995 Nonconvex Problems 5016 Stopping Criteria 5027 Convergence Results of ADMM 502 28 Nonconvex Optimization via MM Algorithms: Convergence Theory 509Kenneth Lange, Joong-Ho Won, Alfonso Landeros, and Hua Zhou1 Background5092 Convergence Theorems 5103 Paracontraction 5214 Bregman Majorization 523 Part VII High-Performance Computing 535 29 Massive Parallelization 537Robert B. Gramacy1 Introduction 5372 Gaussian Process Regression and Surrogate Modeling 5393 Divide-and-Conquer GP Regression 5424 Empirical Results 5485 Conclusion 552 30 Divide-and-Conquer Methods for Big Data Analysis 559Xueying Chen, Jerry Q. Cheng, and Min-ge Xie1 Introduction 5592 Linear Regression Model 5603 Parametric Models 5614 Nonparametric and Semiparametric Models 5675 Online Sequential Updating 5686 Splitting the Number of Covariates 5697 Bayesian Divide-and-Conquer and Median-Based Combining 5708 Real-World Applications 5719 Discussion 572 31 Bayesian Aggregation 577Yuling Yao1 From Model Selection to Model Combination 5772 From Bayesian Model Averaging to Bayesian Stacking 5803 Asymptotic Theories of Stacking 5844 Stacking in Practice 5865 Discussion 588 32 Asynchronous Parallel Computing 593Ming Yan1 Introduction 5932 Asynchronous Parallel Coordinate Update 5973 Asynchronous Parallel Stochastic Approaches 6024 Doubly Stochastic Coordinate Optimization with Variance Reduction 6045 Concluding Remarks 605