Matrix Algebra for Linear Models

MARVIN H. J. GRUBER, PHD, is Professor Emeritus in the School of Mathematical Sciences at Rochester Institute of Technology. He has authored several books and journal articles in his areas of research interest, which include improving the efficiency of regression estimators. Dr. Gruber is a member of the American Mathematical Society and the American Statistical Association.

“This book seems suitable for an advanced undergraduate and/or introductory master's level course . . . Four appealing features of this book are its inclusion of an overview, a summary, exercises (with answers provided), and numerical examples for all sections.”  (American Mathematical Society, 1 November 2015)“The book is suitable for graduate and postgraduate students and researchers. This book is highly recommended.”  (Zentralblatt, 1 April 2015)“This is an excellent and comprehensive presentation of the use of matrices for linear models. The writing is very clear, and the layout is excellent. It would serve well either as a class text or as the foundation for individual personal study.”  (International Statistical Review, 18 March 2014)

• Preface xiiiAcknowledgments xvPart I Basic Ideas about Matrices and Systems of Linear Equations 1Section 1 What Matrices are and Some Basic Operations with Them 31.1 Introduction 31.2 What are Matrices and why are they Interesting to a Statistician? 31.3 Matrix Notation Addition and Multiplication 61.4 Summary 10Exercises 10Section 2 Determinants and Solving a System of Equations 142.1 Introduction 142.2 Definition of and Formulae for Expanding Determinants 142.3 Some Computational Tricks for the Evaluation of Determinants 162.4 Solution to Linear Equations Using Determinants 182.5 Gauss Elimination 222.6 Summary 27Exercises 27Section 3 The Inverse of a Matrix 303.1 Introduction 303.2 The Adjoint Method of Finding the Inverse of a Matrix 303.3 Using Elementary Row Operations 313.4 Using the Matrix Inverse to Solve a System of Equations 333.5 Partitioned Matrices and Their Inverses 343.6 Finding the Least Square Estimator 383.7 Summary 44Exercises 44Section 4 Special Matrices and Facts about Matrices that will be used in the Sequel 474.1 Introduction 474.2 Matrices of the Form aIn + bJn 474.3 Orthogonal Matrices 494.4 Direct Product of Matrices 524.5 An Important Property of Determinants 534.6 The Trace of a Matrix 564.7 Matrix Differentiation 574.8 The Least Square Estimator Again 624.9 Summary 62Exercises 63Section 5 Vector Spaces 665.1 Introduction 665.2 What is a Vector Space? 665.3 The Dimension of a Vector Space 685.4 Inner Product Spaces 705.5 Linear Transformations 735.6 Summary 76Exercises 76Section 6 The Rank of a Matrix and Solutions to Systems of Equations 796.1 Introduction 796.2 The Rank of a Matrix 796.3 Solving Systems of Equations with Coefficient Matrix of Less than Full Rank 846.4 Summary 87Exercises 87Part II Eigenvalues the Singular Value Decomposition and Principal Components 91Section 7 Finding the Eigenvalues of a Matrix 937.1 Introduction 937.2 Eigenvalues and Eigenvectors of a Matrix 937.3 Nonnegative Definite Matrices 1017.4 Summary 104Exercises 105Section 8 The Eigenvalues and Eigenvectors of Special Matrices 1088.1 Introduction 1088.2 Orthogonal Nonsingular and Idempotent Matrices 1098.3 The Cayley–Hamilton Theorem 1128.4 The Relationship between the Trace the Determinant and the Eigenvalues of a Matrix 1148.5 The Eigenvalues and Eigenvectors of the Kronecker Product of Two Matrices 1168.6 The Eigenvalues and the Eigenvectors of a Matrix of the Form aI + bJ 1178.7 The Loewner Ordering 1198.8 Summary 121Exercises 122Section 9 The Singular Value Decomposition (SVD) 1249.1 Introduction 1249.2 The Existence of the SVD 1259.3 Uses and Examples of the SVD 1279.4 Summary 134Exercises 134Section 10 Applications of the Singular Value Decomposition 13710.1 Introduction 13710.2 Reparameterization of a Non-full-Rank Model to a Full-Rank Model 13710.3 Principal Components 14110.4 The Multicollinearity Problem 14310.5 Summary 144Exercises 145Section 11 Relative Eigenvalues and Generalizations of the Singular Value Decomposition 14611.1 Introduction 14611.2 Relative Eigenvalues and Eigenvectors 14611.3 Generalizations of the Singular Value Decomposition:Overview 15111.4 The First Generalization 15211.5 The Second Generalization 15711.6 Summary 160Exercises 160Part III Generalized Inverses 163Section 12 Basic Ideas about Generalized Inverses 16512.1 Introduction 16512.2 What is a Generalized Inverse and how is One Obtained? 16512.3 The Moore–Penrose Inverse 17012.4 Summary 173Exercises 173Section 13 Characterizations of Generalized Inverses Using the Singular Value Decomposition 17513.1 Introduction 17513.2 Characterization of the Moore–Penrose Inverse 17513.3 Generalized Inverses in Terms of the Moore–Penrose Inverse 17713.4 Summary 185Exercises 186Section 14 Least Square and Minimum Norm Generalized Inverses 18814.1 Introduction 18814.2 Minimum Norm Generalized Inverses 18914.3 Least Square Generalized Inverses 19314.4 An Extension of Theorem 7.3 to Positive-Semi-definite Matrices 19614.5 Summary 197Exercises 197Section 15 More Representations of Generalized Inverses 20015.1 Introduction 20015.2 Another Characterization of the Moore–Penrose Inverse 20015.3 Still another Representation of the Generalized Inverse 20415.4 The Generalized Inverse of a Partitioned Matrix 20715.5 Summary 211Exercises 211Section 16 Least Square Estimators for Less than Full-Rank Models 21316.1 Introduction 21316.2 Some Preliminaries 21316.3 Obtaining the LS Estimator 21416.4 Summary 221Exercises 221Part IV Quadratic Forms and the Analysis of Variance 223Section 17 Quadratic Forms and their Probability Distributions 22517.1 Introduction 22517.2 Examples of Quadratic Forms 22517.3 The Chi-Square Distribution 22817.4 When does the Quadratic Form of a Random Variable have a Chi-Square Distribution? 23017.5 When are Two Quadratic Forms with the Chi-Square Distribution Independent? 23117.6 Summary 234Exercises 235Section 18 Analysis of Variance: Regression Models and the One- and Two-Way Classification 23718.1 Introduction 23718.2 The Full-Rank General Linear Regression Model 23718.3 Analysis of Variance: One-Way Classification 24118.4 Analysis of Variance: Two-Way Classification 24418.5 Summary 249Exercises 249Section 19 More ANOVA 25319.1 Introduction 25319.2 The Two-Way Classification with Interaction 25419.3 The Two-Way Classification with One Factor Nested 25819.4 Summary 262Exercises 262Section 20 The General Linear Hypothesis 26420.1 Introduction 26420.2 The Full-Rank Case 26420.3 The Non-full-Rank Case 26720.4 Contrasts 27020.5 Summary 273Exercises 273Part V Matrix Optimization Problems 275Section 21 Unconstrained Optimization Problems 27721.1 Introduction 27721.2 Unconstrained Optimization Problems 27721.3 The Least Square Estimator Again 28121.4 Summary 283Exercises 283Section 22 Constrained Minimization Problems with Linear Constraints 28722.1 Introduction 28722.2 An Overview of Lagrange Multipliers 28722.3 Minimizing a Second-Degree Form with Respect to a Linear Constraint 29322.4 The Constrained Least Square Estimator 29522.5 Canonical Correlation 29922.6 Summary 302Exercises 302Section 23 The Gauss–Markov Theorem 30423.1 Introduction 30423.2 The Gauss–Markov Theorem and the Least Square Estimator 30423.3 The Modified Gauss–Markov Theorem and the Linear Bayes Estimator 30623.4 Summary 311Exercises 311Section 24 Ridge Regression-Type Estimators 31424.1 Introduction 31424.2 Minimizing a Second-Degree Form with Respect to a Quadratic Constraint 31424.3 The Generalized Ridge Regression Estimators 31524.4 The Mean Square Error of the Generalized Ridge Estimator without Averaging over the Prior Distribution 31724.5 The Mean Square Error Averaging over the Prior Distribution 32124.6 Summary 321Exercises 321Answers to Selected Exercises 324References 366Index 368