Measuring Agreement

P. K. CHOUDHARY, PhD, is Professor in the Department of Mathematical Sciences at the University of Texas at Dallas. Currently, he is also the Associate Head of the department. His research interests include development of statistical methodology for biostatistical applications, and he has published extensively in the field of method comparison studies. H. N. NAGARAJA, PhD, is Professor Emeritus at The Ohio State University where he has served in the Departments of Statistics and Internal Medicine and the Division of Biostatistics. He is a fellow of the American Statistical Association and the American Association for the Advancement of Science, and an elected member of the International Statistical Institute. His published works include Order Statistics, Third Edition (with H. A. David) and Records (with B. C. Arnold and N. Balakrishnan), both published by Wiley.

• Preface xv1 Introduction 11.1 Preview 11.2 Notational Conventions 11.3 Basic Characteristics of a Measurement Method 21.3.1 A Statistical Model for Measurements 31.3.2 Quality Characteristics 31.4 Method Comparison Studies 51.5 Meaning of Agreement 61.6 A Measurement Error Model 81.6.1 Identifiability Issues 91.6.2 ModelBased Moments 101.6.3 Conditions for Perfect Agreement 101.6.4 Link to Test Theory 111.7 Similarity versus Agreement 111.7.1 Evaluation of Similarity 111.7.2 Evaluation of Agreement 121.8 A Toy Example 131.9 Controversies and Our View 141.10 Concepts Related to Agreement 151.11 Role of Confidence Intervals and Hypotheses Testing 161.11.1 Formulating the Agreement Hypotheses 161.11.2 Testing Hypotheses Using Confidence Bounds 171.11.3 Evaluation of Agreement Using Confidence Bounds 171.11.4 Evaluation of Similarity Using Confidence Intervals 181.12 Common Models for Paired Measurements Data 181.12.1 A Measurement Error Model 191.12.2 A MixedEffects Model 201.12.3 A Bivariate Normal Model 211.12.4 Limitations of the Paired Measurements Design 221.13 The BlandAltman Plot 231.13.1 The Ideal Plot 231.13.2 A Linear Trend in the BlandAltman Plot 251.13.3 Heteroscedasticity in the BlandAltman Plot 261.13.4 Variations of the BlandAltman Plot 271.14 Common Regression Approaches 291.14.1 Ordinary Linear Regression 291.14.2 Deming Regression 311.15 Inappropriate Use of Common Tests in Method Comparison Studies 341.15.1 Test of Zero Correlation 341.15.2 Paired ttest 361.15.3 PitmanMorganand BradleyBlackwood Tests 361.15.4 Test of Zero Intercept and Unit Slope 381.16 Key Steps in the Analysis of Method Comparison Data 391.17 Chapter Summary 401.18 Bibliographic Note 41Exercises 472 Common Approaches for Measuring Agreement 532.1 Preview 532.2 Introduction 532.3 Mean Squared Deviation 542.4 Concordance Correlation Coefficient 542.5 A Digression: Tolerance and Prediction intervals 572.5.1 Definitions 572.5.2 Normally Distributed Data 582.6 Lin’s Probability Criterion and BlandAltman Criterion 592.7 Limits of Agreement 602.7.1 The Approach 602.7.2 Why Ignore the Variability? 612.7.3 Limits of Agreement versus Prediction and Tolerance Intervals 622.8 Total Deviation Index and Coverage Probability 622.8.1 The Approaches 622.8.2 Normally Distributed Differences 632.9 Inference on Agreement Measures 642.10 Chapter Summary 642.11 Bibliographic Note 65Exercises 663 A General Approach for Modeling and Inference 713.1 Preview 713.2 MixedEffects Models 713.2.1 The Model 723.2.2 Prediction 733.2.3 Model Fitting 743.2.4 Model Diagnostics 753.3 A LargeSample Approach to Inference 763.3.1 Approximate Distributions 773.3.2 Confidence Intervals 783.3.3 Parameter Transformation 803.3.4 Bootstrap Confidence Intervals 813.3.5 Confidence Bands 833.3.6 Test of Homogeneity 833.3.7 Model Comparison 843.4 Modeling and Analysis of Method Comparison Data 853.5 Chapter Summary 883.6 Bibliographic Note 89Exercises 894 Paired Measurements Data 954.1 Preview 954.2 Modeling of Data 954.2.1 MixedEffects Model 954.2.2 Bivariate Normal Model 974.3 Evaluation of Similarity and Agreement 984.4 Case Studies 994.4.1 Oxygen Saturation Data 994.4.2 Plasma Volume Data 1014.4.3 Vitamin D Data 1034.5 Chapter Summary 1064.6 Technical Details 1064.6.1 MixedEffects Model 1064.6.2 Bivariate Normal Model 1074.7 Bibliographic Note 108Exercises 1085 Repeated Measurements Data 1115.1 Preview 1115.2 Introduction 1115.2.1 Types of Data 1125.2.2 Individual versus Average Measurement 1135.2.3 Example Datasets 1135.3 Displaying Data 1145.3.1 Basic Plots 1145.3.2 Interaction Plots 1165.4 Modeling of Data 1175.4.1 Unlinked Data 1185.4.2 Linked Data 1215.4.3 Model Fitting and Evaluation 1235.5 Evaluation of Similarity and Agreement 1235.6 Evaluation of Repeatability 1245.6.1 Unlinked Data 1255.6.2 Linked Data 1255.7 Case Studies 1265.7.1 Kiwi Data 1265.7.2 Oximetry Data 1295.8 Chapter Summary 1335.9 Technical Details 1345.9.1 Unlinked Data 1345.9.2 Linked Data 1345.10 Bibliographic Note 135Exercises 1376 Heteroscedastic Data 1416.1 Preview 1416.2 Introduction 1416.2.1 Diagnosing Heteroscedasticity 1426.2.2 Example Datasets 1436.3 Variance Function Models 1446.4 Repeated Measurements Data 1466.4.1 A Heteroscedastic MixedEffects Model 1476.4.2 Specifying the Variance Function 1496.4.3 Model Fitting and Evaluation 1506.4.4 Testing for Homoscedasticity 1516.4.5 Evaluation of Similarity, Agreement, and Repeatability 1516.4.6 Case Study: Cholesterol Data 1526.5 Paired Measurements Data 1626.5.1 A Heteroscedastic Bivariate Normal Model 1626.5.2 Specifying the Variance Function 1636.5.3 Model Fitting and Evaluation 1646.5.4 Testing for Homoscedasticity 1646.5.5 Evaluation of Similarity and Agreement 1646.5.6 Case Study: Cyclosporin Data 1656.6 Chapter Summary 1716.7 Technical Details 1716.7.1 Repeated Measurements Data 1716.7.2 Paired Measurements Data 1736.8 Bibliographic Note 174Exercises 1747 Data from Multiple Methods 1777.1 Preview 1777.2 Introduction 1777.3 Displaying Data 1797.4 Example Datasets 1797.4.1 Systolic Blood Pressure Data 1807.4.2 Tumor Size Data 1807.5 Modeling Unreplicated Data 1847.6 Modeling Repeated Measurements Data 1867.6.1 Unlinked Data 1867.6.2 Linked Data 1877.7 Model Fitting and Evaluation 1897.8 Evaluation of Similarity and Agreement 1907.9 Evaluation of Repeatability 1917.10 Case Studies 1927.10.1 Systolic Blood Pressure Data 1927.10.2 Tumor Size Data 1957.11 Chapter Summary 1987.12 Technical Details 1987.13 Bibliographic Note 200Exercises 2008 Data with Covariates 2058.1 Preview 2058.2 Introduction 2058.3 Modeling of Data 2068.3.1 Modeling Means of Methods 2068.3.2 Modeling Variances of Methods 2078.3.3 Data Models 2088.3.4 Model Fitting and Evaluation 2118.4 Evaluation of Similarity, Agreement, and Repeatability 2118.4.1 Measures of Agreement for Two methods 2128.4.2 Measures of Agreement for More Than Two Methods 2138.4.3 Measures of Repeatability 2138.4.4 Inference on Measures 2148.5 Case Study 2148.6 Chapter Summary 2248.7 Technical Details 2258.8 Bibliographic Note 226Exercises 2269 Longitudinal Data 2299.1 Preview 2299.2 Introduction 2299.2.1 Displaying Data 2319.2.2 Percentage Body Fat Data 2319.3 Modeling of Data 2349.3.1 The Longitudinal Data Model 2369.3.2 Specifying the Mean Functions 2379.3.3 Specifying the Correlation Function 2379.3.4 Model Fitting and Evaluation 2409.4 Evaluation of Similarity and Agreement 2419.5 Case Study 2429.6 Chapter Summary 2479.7 Technical Details 2479.8 Bibliographic Note 249Exercises 25010 A Nonparametric Approach 25310.1 Preview 25310.2 Introduction 25310.3 The Statistical Functional Approach 25510.3.1 A Weighted Empirical CDF 25610.3.2 Distributions Induced by Empirical CDF 25610.4 Evaluation of Similarity and Agreement 25810.5 Case Studies 25910.5.1 Unreplicated Blood Pressure Data 25910.5.2 Replicated Blood Pressure Data 26310.6 Chapter Summary 26710.7 Technical Details 26710.7.1 The  Matrix 26810.7.2 Estimation of  26910.7.3 Influence Functions for the Measures 27010.7.4 TDI Confidence Bounds 27010.7.5 Summary of Steps 27110.8 Bibliographic Note 271Exercises 27211 Sample Size Determination 27911.1 Preview 27911.2 Introduction 27911.3 The Sample Size Methodology 28111.3.1 Paired Measurements Design 28111.3.2 Repeated Measurements Design 28111.4 Case Study 28211.5 Chapter Summary 28611.6 Bibliographic Note 286Exercises 28712 Categorical Data 28912.1 Preview 28912.2 Introduction 28912.3 Experimental Setups and Examples 29012.3.1 Types of Data 29012.3.2 Illustrative Examples 29012.3.3 A Graphical Approach 29212.4 Cohen’s Kappa Coefficient for Dichotomous Data 29312.4.1 Definition and Basic Properties: Two Raters 29312.4.2 Sample Kappa Coefficient 29712.4.3 Agreement with a Gold Standard 29812.4.4 Unbiased Raters: Intraclass Kappa 29912.4.5 Multiple Raters 30012.4.6 Combining and Comparing Kappa Coefficients 30112.4.7 Sample Size Calculations 30212.5 Kappa Type Measures for More Than Two Categories 30312.5.1 Two Fixed Raters with Nominal Categories 30312.5.2 Two Raters with Ordinal Categories: Weighted Kappa 30312.5.3 Multiple Raters 30412.6 Case Studies 30512.6.1 Two Raters with Two Categories 30512.6.2 Weighted Kappa: Multiple Categories 30612.7 Models for Exploring Agreement 30612.7.1 Conditional Logistic Regression Models 30612.7.2 LogLinear Models 30712.7.3 A Generalized Linear MixedEffects Model 30812.8 Discussion 30912.9 Chapter Summary 31012.10 Bibliographic Note 311Exercises 312References 319Dataset List 331Index 333