Advanced Calculus with Applications in Statistics

ANDRE I. KHURI, PhD, is a Professor in the Department of Statistics at the University of Florida, Gainesville.

"This is an exceptional book, which I would recommend for anyone beginning a career in statistical research." (Journal of the American Statistical Association, September 2004)

• Preface xvPreface to the First Edition xvii1. An Introduction to Set Theory 11.1. The Concept of a Set 11.2. Set Operations 21.3. Relations and Functions 41.4. Finite Countable and Uncountable Sets 61.5. Bounded Sets 91.6. Some Basic Topological Concepts 101.7. Examples in Probability and Statistics 13Further Reading and Annotated Bibliography 15Exercises 172. Basic Concepts in Linear Algebra 212.1. Vector Spaces and Subspaces 212.2. Linear Transformations 252.3. Matrices and Determinants 272.3.1. Basic Operations on Matrices 282.3.2. The Rank of a Matrix 332.3.3. The Inverse of a Matrix 342.3.4. Generalized Inverse of a Matrix 362.3.5. Eigenvalues and Eigenvectors of a Matrix 362.3.6. Some Special Matrices 382.3.7. The Diagonalization of a Matrix 382.3.8. Quadratic Forms 392.3.9. The Simultaneous Diagonalization of Matrices 402.3.10. Bounds on Eigenvalues 412.4. Applications of Matrices in Statistics 432.4.1. The Analysis of the Balanced Mixed Model 432.4.2. The Singular-Value Decomposition 452.4.3. Extrema of Quadratic Forms 482.4.4. The Parameterization of Orthogonal Matrices 49Further Reading and Annotated Bibliography 50Exercises 533. Limits and Continuity of Functions 573.1. Limits of a Function 573.2. Some Properties Associated with Limits of Functions 633.3. The o O Notation 653.4. Continuous Functions 663.4.1. Some Properties of Continuous Functions 713.4.2. Lipschitz Continuous Functions 753.5. Inverse Functions 763.6. Convex Functions 793.7. Continuous and Convex Functions in Statistics 82Further Reading and Annotated Bibliography 87Exercises 884. Differentiation 934.1. The Derivative of a Function 934.2. The Mean Value Theorem 994.3. Taylor’s Theorem 1084.4. Maxima and Minima of a Function 1124.4.1. A Sufficient Condition for a Local Optimum 1144.5. Applications in Statistics 1154.5.1. Functions of Random Variables 1164.5.2. Approximating Response Functions 1214.5.3. The Poisson Process 1224.5.4. Minimizing the Sum of Absolute Deviations 124Further Reading and Annotated Bibliography 125Exercises 1275. Infinite Sequences and Series 1325.1. Infinite Sequences 1325.1.1. The Cauchy Criterion 1375.2. Infinite Series 1405.2.1. Tests of Convergence for Series of Positive Terms 1445.2.2. Series of Positive and Negative Terms 1585.2.3. Rearrangement of Series 1595.2.4. Multiplication of Series 1625.3. Sequences and Series of Functions 1655.3.1. Properties of Uniformly Convergent Sequences and Series 1695.4. Power Series 1745.5. Sequences and Series of Matrices 1785.6. Applications in Statistics 1825.6.1. Moments of a Discrete Distribution 1825.6.2. Moment and Probability Generating Functions 1865.6.3. Some Limit Theorems 1915.6.3.1. The Weak Law of Large Numbers ŽKhinchine’s Theorem. 1925.6.3.2. The Strong Law of Large Numbers ŽKolmogorov’s Theorem. 1925.6.3.3. The Continuity Theorem for Probability Generating Functions 1925.6.4. Power Series and Logarithmic Series Distributions 1935.6.5. Poisson Approximation to Power Series Distributions 1945.6.6. A Ridge Regression Application 195Further Reading and Annotated Bibliography 197Exercises 1996. Integration 2056.1. Some Basic Definitions 2056.2. The Existence of the Riemann Integral 2066.3. Some Classes of Functions That Are Riemann Integrable 2106.3.1. Functions of Bounded Variation 2126.4. Properties of the Riemann Integral 2156.4.1. Change of Variables in Riemann Integration 2196.5. Improper Riemann Integrals 2206.5.1. Improper Riemann Integrals of the Second Kind 2256.6. Convergence of a Sequence of Riemann Integrals 2276.7. Some Fundamental Inequalities 2296.7.1. The Cauchy_Schwarz Inequality 2296.7.2. H¨older’s Inequality 2306.7.3. Minkowski’s Inequality 2326.7.4. Jensen’s Inequality 2336.8. Riemann_Stieltjes Integral 2346.9. Applications in Statistics 2396.9.1. The Existence of the First Negative Moment of a Continuous Distribution 2426.9.2. Transformation of Continuous Random Variables 2466.9.3. The Riemann_Stieltjes Representation of the Expected Value 2496.9.4. Chebyshev’s Inequality 251Further Reading and Annotated Bibliography 252Exercises 2537. Multidimensional Calculus 2617.1. Some Basic Definitions 2617.2. Limits of a Multivariable Function 2627.3. Continuity of a Multivariable Function 2647.4. Derivatives of a Multivariable Function 2677.4.1. The Total Derivative 2707.4.2. Directional Derivatives 2737.4.3. Differentiation of Composite Functions 2767.5. Taylor’s Theorem for a Multivariable Function 2777.6. Inverse and Implicit Function Theorems 2807.7. Optima of a Multivariable Function 2837.8. The Method of Lagrange Multipliers 2887.9. The Riemann Integral of a Multivariable Function 2937.9.1. The Riemann Integral on Cells 2947.9.2. Iterated Riemann Integrals on Cells 2957.9.3. Integration over General Sets 2977.9.4. Change of Variables in n-Tuple Riemann Integrals 2997.10. Differentiation under the Integral Sign 3017.11. Applications in Statistics 3047.11.1. Transformations of Random Vectors 3057.11.2. Maximum Likelihood Estimation 3087.11.3. Comparison of Two Unbiased Estimators 3107.11.4. Best Linear Unbiased Estimation 3117.11.5. Optimal Choice of Sample Sizes in Stratified Sampling 313Further Reading and Annotated Bibliography 315Exercises 3168. Optimization in Statistics 3278.1. The Gradient Methods 3298.1.1. The Method of Steepest Descent 3298.1.2. The Newton_Raphson Method 3318.1.3. The Davidon_Fletcher_Powell Method 3318.2. The Direct Search Methods 3328.2.1. The Nelder_Mead Simplex Method 3328.2.2. Price’s Controlled Random Search Procedure 3368.2.3. The Generalized Simulated Annealing Method 3388.3. Optimization Techniques in Response Surface Methodology 3398.3.1. The Method of Steepest Ascent 3408.3.2. The Method of Ridge Analysis 3438.3.3. Modified Ridge Analysis 3508.4. Response Surface Designs 3558.4.1. First-Order Designs 3568.4.2. Second-Order Designs 3588.4.3. Variance and Bias Design Criteria 3598.5. Alphabetic Optimality of Designs 3628.6. Designs for Nonlinear Models 3678.7. Multiresponse Optimization 3708.8. Maximum Likelihood Estimation and the EM Algorithm 3728.8.1. The EM Algorithm 3758.9. Minimum Norm Quadratic Unbiased Estimation of Variance Components 3788.10. Scheff´e’s Confidence Intervals 3828.10.1. The Relation of Scheff´e’s Confidence Intervals to the F-Test 385Further Reading and Annotated Bibliography 391Exercises 3959. Approximation of Functions 4039.1. Weierstrass Approximation 4039.2. Approximation by Polynomial Interpolation 4109.2.1. The Accuracy of Lagrange Interpolation 4139.2.2. A Combination of Interpolation and Approximation 4179.3. Approximation by Spline Functions 4189.3.1. Properties of Spline Functions 4189.3.2. Error Bounds for Spline Approximation 4219.4. Applications in Statistics 4229.4.1. Approximate Linearization of Nonlinear Models by Lagrange Interpolation 4229.4.2. Splines in Statistics 4289.4.2.1. The Use of Cubic Splines in Regression 4289.4.2.2. Designs for Fitting Spline Models 4309.4.2.3. Other Applications of Splines in Statistics 431Further Reading and Annotated Bibliography 432Exercises 43410. Orthogonal Polynomials 43710.1. Introduction 43710.2. Legendre Polynomials 44010.2.1. Expansion of a Function Using Legendre Polynomials 44210.3. Jacobi Polynomials 44310.4. Chebyshev Polynomials 44410.4.1. Chebyshev Polynomials of the First Kind 44410.4.2. Chebyshev Polynomials of the Second Kind 44510.5. Hermite Polynomials 44710.6. Laguerre Polynomials 45110.7. Least-Squares Approximation with Orthogonal Polynomials 45310.8. Orthogonal Polynomials Defined on a Finite Set 45510.9. Applications in Statistics 45610.9.1. Applications of Hermite Polynomials 45610.9.1.1. Approximation of Density Functions and Quantiles of Distributions 45610.9.1.2. Approximation of a Normal Integral 46010.9.1.3. Estimation of Unknown Densities 46110.9.2. Applications of Jacobi and Laguerre Polynomials 46210.9.3. Calculation of Hypergeometric Probabilities Using Discrete Chebyshev Polynomials 462Further Reading and Annotated Bibliography 464Exercises 46611. Fourier Series 47111.1. Introduction 47111.2. Convergence of Fourier Series 47511.3. Differentiation and Integration of Fourier Series 48311.4. The Fourier Integral 48811.5. Approximation of Functions by Trigonometric Polynomials 49511.5.1. Parseval’s Theorem 49611.6. The Fourier Transform 49711.6.1. Fourier Transform of a Convolution 49911.7. Applications in Statistics 50011.7.1. Applications in Time Series 50011.7.2. Representation of Probability Distributions 50111.7.3. Regression Modeling 50411.7.4. The Characteristic Function 50511.7.4.1. Some Properties of Characteristic Functions 510Further Reading and Annotated Bibliography 510Exercises 51212. Approximation of Integrals 51712.1. The Trapezoidal Method 51712.1.1. Accuracy of the Approximation 51812.2. Simpson’s Method 52112.3. Newton_Cotes Methods 52312.4. Gaussian Quadrature 52412.5. Approximation over an Infinite Interval 52812.6. The Method of Laplace 53112.7. Multiple Integrals 53312.8. The Monte Carlo Method 53512.8.1. Variation Reduction 53712.8.2. Integrals in Higher Dimensions 54012.9. Applications in Statistics 54112.9.1. The Gauss_Hermite Quadrature 54212.9.2. Minimum Mean Squared Error Quadrature 54312.9.3. Moments of a Ratio of Quadratic Forms 54612.9.4. Laplace’s Approximation in Bayesian Statistics 54812.9.5. Other Methods of Approximating Integrals in Statistics 549Further Reading and Annotated Bibliography 550Exercises 552Appendix. Solutions to Selected Exercises 557General Bibliography 652Index 665