Matrix Differential Calculus with Applications in Statistics and Econometrics

JAN R. MAGNUS is Emeritus Professor at the Department of Econometrics & Operations Research, Tilburg University, and Extraordinary Professor at the Department of Econometrics & Operations Research, Vrije University, Amsterdam. He is research fellow of CentER and the Tinbergen Institute. He has co-authored nine books and is the author of over 100 scientific papers. HEINZ NEUDECKER (1933-2017) was Professor of Econometrics at the University of Amsterdam from 1972 until his retirement in 1998.

• Preface xiiiPart One — Matrices1 Basic properties of vectors and matrices 31 Introduction 32 Sets 33 Matrices: addition and multiplication 44 The transpose of a matrix 65 Square matrices 66 Linear forms and quadratic forms 77 The rank of a matrix 98 The inverse 109 The determinant 1010 The trace 1111 Partitioned matrices 1212 Complex matrices 1413 Eigenvalues and eigenvectors 1414 Schur’s decomposition theorem 1715 The Jordan decomposition 1816 The singular-value decomposition 2017 Further results concerning eigenvalues 2018 Positive (semi)definite matrices 2319 Three further results for positive definite matrices 2520 A useful result 2621 Symmetric matrix functions 27Miscellaneous exercises 28Bibliographical notes 302 Kronecker products, vec operator, and Moore-Penrose inverse 311 Introduction 312 The Kronecker product 313 Eigenvalues of a Kronecker product 334 The vec operator 345 The Moore-Penrose (MP) inverse 366 Existence and uniqueness of the MP inverse 377 Some properties of the MP inverse 388 Further properties 399 The solution of linear equation systems 41Miscellaneous exercises 43Bibliographical notes 453 Miscellaneous matrix results 471 Introduction 472 The adjoint matrix 473 Proof of Theorem 3.1 494 Bordered determinants 515 The matrix equation AX = 0 516 The Hadamard product 527 The commutation matrix Kmn 548 The duplication matrix Dn 569 Relationship between Dn+1 and Dn, I 5810 Relationship between Dn+1 and Dn, II 5911 Conditions for a quadratic form to be positive (negative) subject to linear constraints 6012 Necessary and sufficient conditions for r(A : B) = r(A) + r(B) 6313 The bordered Gramian matrix 6514 The equations X1A + X2B′ = G1,X1B = G2 67Miscellaneous exercises 69Bibliographical notes 70Part Two — Differentials: the theory4 Mathematical preliminaries 731 Introduction 732 Interior points and accumulation points 733 Open and closed sets 754 The Bolzano-Weierstrass theorem 775 Functions 786 The limit of a function 797 Continuous functions and compactness 808 Convex sets 819 Convex and concave functions 83Bibliographical notes 865 Differentials and differentiability 871 Introduction 872 Continuity 883 Differentiability and linear approximation 904 The differential of a vector function 915 Uniqueness of the differential 936 Continuity of differentiable functions 947 Partial derivatives 958 The first identification theorem 969 Existence of the differential, I 9710 Existence of the differential, II 9911 Continuous differentiability 10012 The chain rule 10013 Cauchy invariance 10214 The mean-value theorem for real-valued functions 10315 Differentiable matrix functions 10416 Some remarks on notation 10617 Complex differentiation 108Miscellaneous exercises 110Bibliographical notes 1106 The second differential 1111 Introduction 1112 Second-order partial derivatives 1113 The Hessian matrix 1124 Twice differentiability and second-order approximation, I 1135 Definition of twice differentiability 1146 The second differential 1157 Symmetry of the Hessian matrix 1178 The second identification theorem 1199 Twice differentiability and second-order approximation, II 11910 Chain rule for Hessian matrices 12111 The analog for second differentials 12312 Taylor’s theorem for real-valued functions 12413 Higher-order differentials 12514 Real analytic functions 12515 Twice differentiable matrix functions 126Bibliographical notes 1277 Static optimization 1291 Introduction 1292 Unconstrained optimization 1303 The existence of absolute extrema 1314 Necessary conditions for a local minimum 1325 Sufficient conditions for a local minimum: first-derivative test 1346 Sufficient conditions for a local minimum: second-derivative test 1367 Characterization of differentiable convex functions 1388 Characterization of twice differentiable convex functions 1419 Sufficient conditions for an absolute minimum 14210 Monotonic transformations 14311 Optimization subject to constraints 14412 Necessary conditions for a local minimum under constraints 14513 Sufficient conditions for a local minimum under constraints 14914 Sufficient conditions for an absolute minimum under constraints 15415 A note on constraints in matrix form 15516 Economic interpretation of Lagrange multipliers 155Appendix: the implicit function theorem 157Bibliographical notes 159Part Three — Differentials: the practice8 Some important differentials 1631 Introduction 1632 Fundamental rules of differential calculus 1633 The differential of a determinant 1654 The differential of an inverse 1685 Differential of the Moore-Penrose inverse 1696 The differential of the adjoint matrix 1727 On differentiating eigenvalues and eigenvectors 1748 The continuity of eigenprojections 1769 The differential of eigenvalues and eigenvectors: symmetric case 18010 Two alternative expressions for dλ 18311 Second differential of the eigenvalue function 185Miscellaneous exercises 186Bibliographical notes 1899 First-order differentials and Jacobian matrices 1911 Introduction 1912 Classification 1923 Derisatives 1924 Derivatives 1945 Identification of Jacobian matrices 1966 The first identification table 1977 Partitioning of the derivative 1978 Scalar functions of a scalar 1989 Scalar functions of a vector 19810 Scalar functions of a matrix, I: trace 19911 Scalar functions of a matrix, II: determinant 20112 Scalar functions of a matrix, III: eigenvalue 20213 Two examples of vector functions 20314 Matrix functions 20415 Kronecker products 20616 Some other problems 20817 Jacobians of transformations 209Bibliographical notes 21010 Second-order differentials and Hessian matrices 2111 Introduction 2112 The second identification table 2113 Linear and quadratic forms 2124 A useful theorem 2135 The determinant function 2146 The eigenvalue function 2157 Other examples 2158 Composite functions 2179 The eigenvector function 21810 Hessian of matrix functions, I 21911 Hessian of matrix functions, II 219Miscellaneous exercises 220Part Four — Inequalities11 Inequalities 2251 Introduction 2252 The Cauchy-Schwarz inequality 2263 Matrix analogs of the Cauchy-Schwarz inequality 2274 The theorem of the arithmetic and geometric means 2285 The Rayleigh quotient 2306 Concavity of λ1 and convexity of λn 2327 Variational description of eigenvalues 2328 Fischer’s min-max theorem 2349 Monotonicity of the eigenvalues 23610 The Poincar´e separation theorem 23611 Two corollaries of Poincar´e’s theorem 23712 Further consequences of the Poincar´e theorem 23813 Multiplicative version 23914 The maximum of a bilinear form 24115 Hadamard’s inequality 24216 An interlude: Karamata’s inequality 24217 Karamata’s inequality and eigenvalues 24418 An inequality concerning positive semidefinite matrices 24519 A representation theorem for ( ∑api )1/p 24620 A representation theorem for (trAp)1/p 24721 Hölder’s inequality 24822 Concavity of log|A| 25023 Minkowski’s inequality 25124 Quasilinear representation of |A|1/n 25325 Minkowski’s determinant theorem 25526 Weighted means of order p 25627 Schlömilch’s inequality 25828 Curvature properties of Mp(x, a) 25929 Least squares 26030 Generalized least squares 26131 Restricted least squares 26232 Restricted least squares: matrix version 264Miscellaneous exercises 265Bibliographical notes 269Part Five — The linear model12 Statistical preliminaries 2731 Introduction 2732 The cumulative distribution function 2733 The joint density function 2744 Expectations 2745 Variance and covariance 2756 Independence of two random variables 2777 Independence of n random variables 2798 Sampling 2799 The one-dimensional normal distribution 27910 The multivariate normal distribution 28011 Estimation 282Miscellaneous exercises 282Bibliographical notes 28313 The linear regression model 2851 Introduction 2852 Affine minimum-trace unbiased estimation 2863 The Gauss-Markov theorem 2874 The method of least squares 2905 Aitken’s theorem 2916 Multicollinearity 2937 Estimable functions 2958 Linear constraints: the case M(R′) ⊂M(X′) 2969 Linear constraints: the general case 30010 Linear constraints: the case M(R′) ∩M(X′) = {0} 30211 A singular variance matrix: the case M(X) ⊂M(V ) 30412 A singular variance matrix: the case r(X′V +X) = r(X) 30513 A singular variance matrix: the general case, I 30714 Explicit and implicit linear constraints 30715 The general linear model, I 31016 A singular variance matrix: the general case, II 31117 The general linear model, II 31418 Generalized least squares 31519 Restricted least squares 316Miscellaneous exercises 318Bibliographical notes 31914 Further topics in the linear model 3211 Introduction 3212 Best quadratic unbiased estimation of σ2 3223 The best quadratic and positive unbiased estimator of σ2 3224 The best quadratic unbiased estimator of σ2 3245 Best quadratic invariant estimation of σ2 3266 The best quadratic and positive invariant estimator of σ2 3277 The best quadratic invariant estimator of σ2 3298 Best quadratic unbiased estimation: multivariate normal case 3309 Bounds for the bias of the least-squares estimator of σ2, I 33210 Bounds for the bias of the least-squares estimator of σ2, II 33311 The prediction of disturbances 33512 Best linear unbiased predictors with scalar variance matrix 33613 Best linear unbiased predictors with fixed variance matrix, I 33814 Best linear unbiased predictors with fixed variance matrix, II 34015 Local sensitivity of the posterior mean 34116 Local sensitivity of the posterior precision 342Bibliographical notes 344Part Six — Applications to maximum likelihood estimation15 Maximum likelihood estimation 3471 Introduction 3472 The method of maximum likelihood (ML) 3473 ML estimation of the multivariate normal distribution 3484 Symmetry: implicit versus explicit treatment 3505 The treatment of positive definiteness 3516 The information matrix 3527 ML estimation of the multivariate normal distribution: distinct means 3548 The multivariate linear regression model 3549 The errors-in-variables model 35710 The nonlinear regression model with normal errors 35911 Special case: functional independence of mean and variance parameters 36112 Generalization of Theorem 15.6 362Miscellaneous exercises 364Bibliographical notes 36516 Simultaneous equations 3671 Introduction 3672 The simultaneous equations model 3673 The identification problem 3694 Identification with linear constraints on B and Γ only 3715 Identification with linear constraints on B, Γ, and ∑ 3716 Nonlinear constraints 3737 FIML: the information matrix (general case) 3748 FIML: asymptotic variance matrix (special case) 3769 LIML: first-order conditions 37810 LIML: information matrix 38111 LIML: asymptotic variance matrix 383Bibliographical notes 38817 Topics in psychometrics 3891 Introduction 3892 Population principal components 3903 Optimality of principal components 3914 A related result 3925 Sample principal components 3936 Optimality of sample principal components 3957 One-mode component analysis 3958 One-mode component analysis and sample principal components 3989 Two-mode component analysis 39910 Multimode component analysis 40011 Factor analysis 40412 A zigzag routine 40713 A Newton-Raphson routine 40814 Kaiser’s varimax method 41215 Canonical correlations and variates in the population 41416 Correspondence analysis 41717 Linear discriminant analysis 418Bibliographical notes 419Part Seven — Summary18 Matrix calculus: the essentials 4231 Introduction 4232 Differentials 4243 Vector calculus 4264 Optimization 4295 Least squares 4316 Matrix calculus 4327 Interlude on linear and quadratic forms 4348 The second differential 4349 Chain rule for second differentials 43610 Four examples 43811 The Kronecker product and vec operator 43912 Identification 44113 The commutation matrix 44214 From second differential to Hessian 44315 Symmetry and the duplication matrix 44416 Maximum likelihood 445Further reading 448Bibliography 449Index of symbols 467Subject index 471