Multivariate Density Estimation

David W. Scott, PhD, is Noah Harding Professor in the Department of Statistics at Rice University. The author of over 100 published articles, papers, and book chapters, Dr. Scott is also Fellow of the American Statistical Association (ASA) and the Institute of Mathematical Statistics. He is recipient of the ASA Founder’s Award and the Army Wilks Award. His research interests include computational statistics, data visualization, and density estimation. Dr. Scott is also coeditor of Wiley Interdisciplinary Reviews: Computational Statistics and previous Editor of the Journal of Computational and Graphical Statistics.

"The book is an ideal reference for theoretical and applied statisticians, practicing engineers, as well as readers interested in the theoretical aspects of nonparametric estimation and the application of these methods to multivariate data. The second edition is also useful as a textbook for introductory courses in kernel statistics, smoothing, advanced computational statistics, and general forms of statistical distributions."  (Zentralblatt MATH, 1 June 2015)

• PREFACE TO SECOND EDITION xvPREFACE TO FIRST EDITION xvii1 Representation and Geometry of Multivariate Data 11.1 Introduction 11.2 Historical Perspective 41.3 Graphical Display of Multivariate Data Points 51.3.1 Multivariate Scatter Diagrams 51.3.2 Chernoff Faces 111.3.3 Andrews’ Curves and Parallel Coordinate Curves 121.3.4 Limitations 141.4 Graphical Display of Multivariate Functionals 161.4.1 Scatterplot Smoothing by Density Function 161.4.2 Scatterplot Smoothing by Regression Function 181.4.3 Visualization of Multivariate Functions 191.4.3.1 Visualizing Multivariate Regression Functions 241.4.4 Overview of Contouring and Surface Display 261.5 Geometry of Higher Dimensions 281.5.1 Polar Coordinates in d Dimensions 281.5.2 Content of Hypersphere 291.5.3 Some Interesting Consequences 301.5.3.1 Sphere Inscribed in Hypercube 301.5.3.2 Hypervolume of a Thin Shell 301.5.3.3 Tail Probabilities of Multivariate Normal 311.5.3.4 Diagonals in Hyperspace 311.5.3.5 Data Aggregate Around Shell 321.5.3.6 Nearest Neighbor Distances 32Problems 332 Nonparametric Estimation Criteria 362.1 Estimation of the Cumulative Distribution Function 372.2 Direct Nonparametric Estimation of the Density 392.3 Error Criteria for Density Estimates 402.3.1 MISE for Parametric Estimators 422.3.1.1 Uniform Density Example 422.3.1.2 General Parametric MISE Method with Gaussian Application 432.3.2 The L1 Criterion 442.3.2.1 L1 versus L2 442.3.2.2 Three Useful Properties of the L1 Criterion 442.3.3 Data-Based Parametric Estimation Criteria 462.4 Nonparametric Families of Distributions 482.4.1 Pearson Family of Distributions 482.4.2 When Is an Estimator Nonparametric? 49Problems 503 Histograms: Theory and Practice 513.1 Sturges’ Rule for Histogram Bin-Width Selection 513.2 The L2 Theory of Univariate Histograms 533.2.1 Pointwise Mean Squared Error and Consistency 533.2.2 Global L2 Histogram Error 563.2.3 Normal Density Reference Rule 593.2.3.1 Comparison of Bandwidth Rules 593.2.3.2 Adjustments for Skewness and Kurtosis 603.2.4 Equivalent Sample Sizes 623.2.5 Sensitivity of MISE to Bin Width 633.2.5.1 Asymptotic Case 633.2.5.2 Large-Sample and Small-Sample Simulations 643.2.6 Exact MISE versus Asymptotic MISE 653.2.6.1 Normal Density 663.2.6.2 Lognormal Density 683.2.7 Influence of Bin Edge Location on MISE 693.2.7.1 General Case 693.2.7.2 Boundary Discontinuities in the Density 693.2.8 Optimally Adaptive Histogram Meshes 703.2.8.1 Bounds on MISE Improvement for Adaptive Histograms 713.2.8.2 Some Optimal Meshes 723.2.8.3 Null Space of Adaptive Densities 723.2.8.4 Percentile Meshes or Adaptive Histograms with Equal Bin Counts 733.2.8.5 Using Adaptive Meshes versus Transformation 743.2.8.6 Remarks 753.3 Practical Data-Based Bin Width Rules 763.3.1 Oversmoothed Bin Widths 763.3.1.1 Lower Bounds on the Number of Bins 763.3.1.2 Upper Bounds on Bin Widths 783.3.2 Biased and Unbiased CV 793.3.2.1 Biased CV 793.3.2.2 Unbiased CV 803.3.2.3 End Problems with BCV and UCV 813.3.2.4 Applications 813.4 L2 Theory for Multivariate Histograms 833.4.1 Curse of Dimensionality 853.4.2 A Special Case: d = 2 with Nonzero Correlation 873.4.3 Optimal Regular Bivariate Meshes 883.5 Modes and Bumps in a Histogram 893.5.1 Properties of Histogram “Modes” 913.5.2 Noise in Optimal Histograms 923.5.3 Optimal Histogram Bandwidths for Modes 933.5.4 A Useful Bimodal Mixture Density 953.6 Other Error Criteria: L1,L4,L6,L8, and L∞ 963.6.1 Optimal L1 Histograms 963.6.2 Other LP Criteria 97Problems 974 Frequency Polygons 1004.1 Univariate Frequency Polygons 1014.1.1 Mean Integrated Squared Error 1014.1.2 Practical FP Bin Width Rules 1044.1.3 Optimally Adaptive Meshes 1074.1.4 Modes and Bumps in a Frequency Polygon 1094.2 Multivariate Frequency Polygons 1104.3 Bin Edge Problems 1134.4 Other Modifications of Histograms 1144.4.1 Bin Count Adjustments 1144.4.1.1 Linear Binning 1144.4.1.2 Adjusting FP Bin Counts to Match Histogram Areas 1174.4.2 Polynomial Histograms 1174.4.3 How Much Information Is There in a Few Bins? 120Problems 1225 Averaged Shifted Histograms 1255.1 Construction 1265.2 Asymptotic Properties 1285.3 The Limiting ASH as a Kernel Estimator 133Problems 1356 Kernel Density Estimators 1376.1 Motivation for Kernel Estimators 1386.1.1 Numerical Analysis and Finite Differences 1386.1.2 Smoothing by Convolution 1396.1.3 Orthogonal Series Approximations 1406.2 Theoretical Properties: Univariate Case 1426.2.1 MISE Analysis 1426.2.2 Estimation of Derivatives 1446.2.3 Choice of Kernel 1456.2.3.1 Higher Order Kernels 1456.2.3.2 Optimal Kernels 1516.2.3.3 Equivalent Kernels 1536.2.3.4 Higher Order Kernels and Kernel Design 1556.2.3.5 Boundary Kernels 1576.3 Theoretical Properties: Multivariate Case 1616.3.1 Product Kernels 1626.3.2 General Multivariate Kernel MISE 1646.3.3 Boundary Kernels for Irregular Regions 1676.4 Generality of the Kernel Method 1676.4.1 Delta Methods 1676.4.2 General Kernel Theorem 1686.4.2.1 Proof of General Kernel Result 1686.4.2.2 Characterization of a Nonparametric Estimator 1696.4.2.3 Equivalent Kernels of Parametric Estimators 1716.5 Cross-Validation 1726.5.1 Univariate Data 1726.5.1.1 Early Efforts in Bandwidth Selection 1736.5.1.2 Oversmoothing 1766.5.1.3 Unbiased and Biased Cross-Validation 1776.5.1.4 Bootstrapping Cross-Validation 1816.5.1.5 Faster Rates and PI Cross-Validation 1846.5.1.6 Constrained Oversmoothing 1876.5.2 Multivariate Data 1906.5.2.1 Multivariate Cross-Validation 1906.5.2.2 Multivariate Oversmoothing Bandwidths 1916.5.2.3 Asymptotics of Multivariate Cross-Validation 1926.6 Adaptive Smoothing 1936.6.1 Variable Kernel Introduction 1936.6.2 Univariate Adaptive Smoothing 1956.6.2.1 Bounds on Improvement 1956.6.2.2 Nearest-Neighbor Estimators 1976.6.2.3 Sample-Point Adaptive Estimators 1986.6.2.4 Data Sharpening 2006.6.3 Multivariate Adaptive Procedures 2026.6.3.1 Pointwise Adapting 2026.6.3.2 Global Adapting 2036.6.4 Practical Adaptive Algorithms 2046.6.4.1 Zero-Bias Bandwidths for Tail Estimation 2046.6.4.2 UCV for Adaptive Estimators 2086.7 Aspects of Computation 2096.7.1 Finite Kernel Support and Rounding of Data 2106.7.2 Convolution and Fourier Transforms 2106.7.2.1 Application to Kernel Density Estimators 2116.7.2.2 FFTs 2126.7.2.3 Discussion 2126.8 Summary 213Problems 2137 The Curse of Dimensionality and Dimension Reduction 2177.1 Introduction 2177.2 Curse of Dimensionality 2207.2.1 Equivalent Sample Sizes 2207.2.2 Multivariate L1 Kernel Error 2227.2.3 Examples and Discussion 2247.3 Dimension Reduction 2297.3.1 Principal Components 2297.3.2 Projection Pursuit 2317.3.3 Informative Components Analysis 2347.3.4 Model-Based Nonlinear Projection 239Problems 2408 Nonparametric Regression and Additive Models 2418.1 Nonparametric Kernel Regression 2428.1.1 The Nadaraya–Watson Estimator 2428.1.2 Local Least-Squares Polynomial Estimators 2438.1.2.1 Local Constant Fitting 2438.1.2.2 Local Polynomial Fitting 2448.1.3 Pointwise Mean Squared Error 2448.1.4 Bandwidth Selection 2478.1.5 Adaptive Smoothing 2478.2 General Linear Nonparametric Estimation 2488.2.1 Local Polynomial Regression 2488.2.2 Spline Smoothing 2508.2.3 Equivalent Kernels 2528.3 Robustness 2538.3.1 Resistant Estimators 2548.3.2 Modal Regression 2548.3.3 L1 Regression 2578.4 Regression in Several Dimensions 2598.4.1 Kernel Smoothing and WARPing 2598.4.2 Additive Modeling 2618.4.3 The Curse of Dimensionality 2628.5 Summary 265Problems 2669 Other Applications 2679.1 Classification, Discrimination, and Likelihood Ratios 2679.2 Modes and Bump Hunting 2739.2.1 Confidence Intervals 2739.2.2 Oversmoothing for Derivatives 2759.2.3 Critical Bandwidth Testing 2759.2.4 Clustering via Mixture Models and Modes 2779.2.4.1 Gaussian Mixture Modeling 2779.2.4.2 Modes for Clustering 2809.3 Specialized Topics 2869.3.1 Bootstrapping 2869.3.2 Confidence Intervals 2879.3.3 Survival Analysis 2899.3.4 High-Dimensional Holes 2909.3.5 Image Enhancement 2929.3.6 Nonparametric Inference 2929.3.7 Final Vignettes 2939.3.7.1 Principal Curves and Density Ridges 2939.3.7.2 Time Series Data 2949.3.7.3 Inverse Problems and Deconvolution 2949.3.7.4 Densities on the Sphere 294Problems 294APPENDIX A Computer Graphics in R3 296A.1 Bivariate and Trivariate Contouring Display 296A.1.1 Bivariate Contouring 296A.1.2 Trivariate Contouring 299A.2 Drawing 3-D Objects on the Computer 300APPENDIX B DataSets 302B.1 US Economic Variables Dataset 302B.2 University Dataset 304B.3 Blood Fat Concentration Dataset 305B.4 Penny Thickness Dataset 306B.5 Gas Meter Accuracy Dataset 307B.6 Old Faithful Dataset 309B.7 Silica Dataset 309B.8 LRL Dataset 310B.9 Buffalo Snowfall Dataset 310APPENDIX C Notation and Abbreviations 311C.1 General Mathematical and Probability Notation 311C.2 Density Abbreviations 312C.3 Error Measure Abbreviations 313C.4 Smoothing Parameter Abbreviations 313REFERENCES 315AUTHOR INDEX 334SUBJECT INDEX 339