Regression Analysis and Linear Models

Richard B. Darlington, PhD, is Emeritus Professor of Psychology at Cornell University. He is a Fellow of the American Association for the Advancement of Science and has published extensively on regression and related methods, the cultural bias of mental tests, the long-term effects of preschool programs, and, most recently, the neuroscience of brain development and evolution.Andrew F. Hayes, PhD, is Distinguished Research Professor in the Haskayne School of Business at the University of Calgary, Alberta, Canada. His research and writing on data analysis has been published widely. Dr. Hayes is the author of Introduction to Mediation, Moderation, and Conditional Process Analysis and Statistical Methods for Communication Science, as well as coauthor, with Richard B. Darlington, of Regression Analysis and Linear Models. He teaches data analysis, primarily at the graduate level, and frequently conducts workshops on statistical analysis throughout the world. His website is www.afhayes.com.

“This is a thorough and accessible introduction to regression analysis as conducted and reported in the psychology research literature. In addition to the basics, there is up-to-date coverage of more advanced topics--for example, interaction effects, path analysis, and mediation. Accompanying examples of statistical software code and output enable students to quickly utilize linear models in the analysis of their own data. This is the right textbook for first-year psychology graduate students, and I plan to continue using it."--Daniel Ozer, PhD, Department of Psychology, University of California, Riverside"This fantastic introduction to the general linear model takes the reader from first principles through to widely used techniques such as mediation and path analysis. The clear writing makes it a pleasure to read. Students will find the book an invaluable resource. There are plenty of insights, too, for even seasoned researchers and data analysts. Instructors and students will appreciate the logical structure and chapters that break the material up into manageable chunks."--Andy Field, PhD, Professor of Child Psychopathology, University of Sussex, United Kingdom"If you want to get the most bang for your buck out of your statistical training, investing in learning regression and linear models is the way to go. Nonetheless, many people find linear modeling to be confusing at first. This book breaks down all walls to mastering this fundamental analysis by providing a complete guide in an approachable, conversational style. The book begins with a comprehensive introduction to linear models and continues on to cover the most useful advanced topics, such as logistic regression and mediation and path analysis. A 'must-have' desk reference for entry-level learners and long-time practitioners alike."--Elizabeth Page-Gould, PhD, Canada Research Chair in Social Psychophysiology, University of Toronto"A terrific addition to the regression literature. I am often asked, 'How do I determine which regressor(s) is/are the most important?' The treatment of this topic is excellent, and the authors have done a fantastic job of bringing important issues to light. The applied nature of the text and the interweaving of software syntax and output are major improvements over similar books. I like the fact that the book has software package information for SPSS, SAS, and STATA. It has a nice balance; not too technical on the statistical side, but not simply a 'how to' on the software side. I could see this book being used as the main text in our department's graduate-level regression course."--Scott C. Roesch, PhD, Department of Psychology, San Diego State University"This is a great textbook for students who have only basic knowledge of statistics yet would like to gain a deep conceptual understanding of regression. The book is up to date in current methods in regression, with strong examples using SAS/SPSS/STATA.”--T. Chris Oshima, PhD, Department of Educational Policy Studies, Georgia State University -

• List of Symbols and Abbreviations1. Statistical Control and Linear Models1.1 Statistical Control1.1.1 The Need for Control1.1.2 Five Methods of Control1.1.3 Examples of Statistical Control1.2 An Overview of Linear Models1.2.1 What You Should Know Already1.2.2 Statistical Software for Linear Modeling and Statistical Control1.2.3 About Formulas1.2.4 On Symbolic Representations1.3 Chapter Summary2. The Simple Regression Model2.1 Scatterplots and Conditional Distributions2.1.1 Scatterplots2.1.2 A Line through Conditional Means2.1.3 Errors of Estimate2.2 The Simple Regression Model2.2.1 The Regression Line2.2.2 Variance, Covariance, and Correlation2.2.3 Finding the Regression Line2.2.4 Example Computations2.2.5 Linear Regression Analysis by Computer2.3 The Regression Coefficient versus the Correlation Coefficient2.3.1 Properties of the Regression and Correlation Coefficients2.3.2 Uses of the Regression and Correlation Coefficients2.4 Residuals2.4.1 The Three Components of Y2.4.2 Algebraic Properties of Residuals2.4.3 Residuals as Y Adjusted for Differences in X2.4.4 Residual Analysis2.5 Chapter Summary3. Partial Relationship and the Multiple Regression Model3.1 Regression Analysis with More Than One Predictor Variable3.1.1 An Example3.1.2 Regressors3.1.3 Models3.1.4 Representing a Model Geometrically3.1.5 Model Errors3.1.6 An Alternative View of the Model3.2 The Best-Fitting Model3.2.1 Model Estimation with Computer Software3.2.2 Partial Regression Coefficients3.2.3 The Regression Constant3.2.4 Problems with Three or More Regressors3.2.5 The Multiple Correlation R3.3 Scale-Free Measures of Partial Association3.3.1 Semipartial Correlation3.3.2 Partial Correlation3.3.3 The Standardized Regression Coefficient3.4 Some Relations among Statistics3.4.1 Relations among Simple, Multiple, Partial, and Semipartial Correlations3.4.2 Venn Diagrams3.4.3 Partial Relationships and Simple Relationships May Have Different Signs3.4.4 How Covariates Affect Regression Coefficients3.4.5 Formulas for bj, prj, srj, and R3.5 Chapter Summary4. Statistical Inference in Regression4.1 Concepts in Statistical Inference4.1.1 Statistics and Parameters4.1.2 Assumptions for Proper Inference4.1.3 Expected Values and Unbiased Estimation4.2 The ANOVA Summary Table4.2.1 Data = Model + Error4.2.2 Total and Regression Sums of Squares4.2.3 Degrees of Freedom4.2.4 Mean Squares4.3 Inference about the Multiple Correlation4.3.1 Biased and Less Biased Estimation of TR24.3.2 Testing a Hypothesis about TR4.4 The Distribution of and Inference about a Partial Regression Coefficient4.4.1 Testing a Null Hypothesis about Tbj4.4.2 Interval Estimates for Tbj4.4.3 Factors Affecting the Standard Error of bj4.4.4 Tolerance4.5 Inferences about Partial Correlations4.5.1 Testing a Null Hypothesis about Tprj and Tsrj4.5.2 Other Inferences about Partial Correlations4.6 Inferences about Conditional Means4.7 Miscellaneous Issues in Inference4.7.1 How Great a Drawback Is Collinearity?4.7.2 Contradicting Inferences4.7.3 Sample Size and Nonsignificant Covariates4.7.4 Inference in Simple Regression (When k = 1)4.8 Chapter Summary5. Extending Regression Analysis Principles5.1 Dichotomous Regressors5.1.1 Indicator or Dummy Variables5.1.2 Y Is a Group Mean5.1.3 The Regression Coefficient for an Indicator Is a Difference5.1.4 A Graphic Representation5.1.5 A Caution about Standardized Regression Coefficients for Dichotomous Regressors5.1.6 Artificial Categorization of Numerical Variables5.2 Regression to the Mean5.2.1 How Regression Got Its Name5.2.2 The Phenomenon5.2.3 Versions of the Phenomenon5.2.4 Misconceptions and Mistakes Fostered by Regression to the Mean5.2.5 Accounting for Regression to the Mean Using Linear Models5.3 Multidimensional Sets5.3.1 The Partial and Semipartial Multiple Correlation5.3.2 What It Means If PR = 0 or SR = 05.3.3 Inference Concerning Sets of Variables5.4 A Glance at the Big Picture5.4.1 Further Extensions of Regression5.4.2 Some Difficulties and Limitations5.5 Chapter Summary6. Statistical versus Experimental Control6.1 Why Random Assignment?6.1.1 Limitations of Statistical Control6.1.2 The Advantage of Random Assignment6.1.3 The Meaning of Random Assignment6.2 Limitations of Random Assignment6.2.1 Limitations Common to Statistical Control and Random Assignment6.2.2 Limitations Specific to Random Assignment6.2.3 Correlation and Causation6.3 Supplementing Random Assignment with Statistical Control6.3.1 Increased Precision and Power6.3.2 Invulnerability to Chance Differences between Groups6.3.3 Quantifying and Assessing Indirect Effects6.4 Chapter Summary7. Regression for Prediction7.1 Mechanical Prediction and Regression7.1.1 The Advantages of Mechanical Prediction7.1.2 Regression as a Mechanical Prediction Method7.1.3 A Focus on R Rather Than the Regression Weights7.2 Estimating True Validity7.2.1 Shrunken versus Adjusted R7.2.2 Estimating TRS7.2.3 Shrunken R Using Statistical Software7.3 Selecting Predictor Variables7.3.1 Stepwise Regression7.3.2 All Subsets Regression7.3.3 How Do Variable Selection Methods Perform?7.4 Predictor Variable Configurations7.4.1 Partial Redundancy (the Standard Configuration)7.4.2 Complete Redundancy7.4.3 Independence7.4.4 Complementarity7.4.5 Suppression7.4.6 How These Configurations Relate to the Correlation between Predictors7.4.7 Configurations of Three or More Predictors7.5 Revisiting the Value of Human Judgment7.6 Chapter Summary8. Assessing the Importance of Regressors8.1 What Does It Mean for a Variable to Be Important?8.1.1 Variable Importance in Substantive or Applied Terms8.1.2 Variable Importance in Statistical Terms8.2 Should Correlations Be Squared?8.2.1 Decision Theory8.2.2 Small Squared Correlations Can Reflect Noteworthy Effects8.2.3 Pearson’s r as the Ratio of a Regression Coefficient to Its Maximum Possible Value8.2.4 Proportional Reduction in Estimation Error8.2.5 When the Standard Is Perfection8.2.6 Summary8.3 Determining the Relative Importance of Regressors in a Single Regression Model8.3.1 The Limitations of the Standardized Regression Coefficient8.3.2 The Advantage of the Semipartial Correlation8.3.3 Some Equivalences among Measures8.3.4 Cohen’s f 28.3.5 Comparing Two Regression Coefficients in the Same Model8.4 Dominance Analysis8.4.1 Complete and Partial Dominance8.4.2 Example Computations8.4.3 Dominance Analysis Using a Regression Program8.5 Chapter Summary9. Multicategorical Regressors9.1 Multicategorical Variables as Sets9.1.1 Indicator (Dummy) Coding9.1.2 Constructing Indicator Variables9.1.3 The Reference Category9.1.4 Testing the Equality of Several Means9.1.5 Parallels with Analysis of Variance9.1.6 Interpreting Estimated Y and the Regression Coefficients9.2 Multicategorical Regressors as or with Covariates