Statistical Methods in Diagnostic Medicine

Xiao-Hua Zhou, fellow of the American Association for the Advancement of Science, fellow of the American Statistical Association, fellow of Institute of Mathematical Statistics, is PKU Distinguished Chair Professor and Chair of the Department of Biostatistics at Peking University, Beijing, China. His research focuses on statistical methods for diagnostic medicine, causal inference, and clinical trials, with extensive experience in regulatory statistics and biomedical research methodology. He has published more than 290 referred papers in those areas. Jiarui Sun is Senior Biostatistician in Shanghai Shengdi Pharmaceutical Co., Ltd. and received his Ph.D from the School of Mathematical Science at Peking University, Beijing, China. His research interests include statistical methods for diagnostic accuracy studies, biomarker evaluation, and computational approaches to medical statistics and diagnostic test validation. Gene A. Pennello, fellow of the American Statistical Association, is a statistical reviewer and research Mathematical Statistician at the U.S. Food and Drug Administration (FDA) in Silver Spring, Maryland. He specializes in statistical methods for medical device evaluation, diagnostic test assessment, and regulatory review processes for medical technologies. Nancy A. Obuchowski, fellow of the American Statistical Association, Professor of Medicine at the Cleveland Clinic Lerner College of Medicine at Case Western Reserve University, has extensive experience in the design, analysis, and development of new statistical methodology for the evaluation of diagnostic and screening tests and quantitative imaging biomarkers. Donna K. McClish, PhD, is Associate Professor and Graduate Program Director in Biostatistics at Virginia Commonwealth University. She has written more than 100 journal articles on statistical methods in epidemiology, diagnostic medicine, and health services research.

• Preface xivAcknowledgments xviPart I Basic Concepts and Methods 11 Introduction 31.1 Diagnostic Test Accuracy Studies 31.2 Case Studies 51.2.1 Case Study 1: Parathyroid Disease 51.2.2 Case Study 2: Colon Cancer Detection 61.2.3 Case Study 3: Carotid Artery Stenosis 71.3 Software 81.4 Topics Not Covered in This Book 82 Measures of Diagnostic Accuracy 92.1 Sensitivity and Specificity 92.1.1 Basic Measures of Test Accuracy: Case Study 2 112.1.2 Diagnostic Tests with Continuous Results: The Artificial Heart Valve Example 122.1.3 Diagnostic Tests with Ordinal Results: Case Study 1 132.1.4 Effect of Prevalence and Spectrum of Disease 142.1.5 Analogy to α and β Statistical Errors 142.2 Combined Measures of Sensitivity and Specificity 152.2.1 Problems Comparing Two or More Tests: Case Study 1 152.2.2 Probability of a Correct Test Result 152.2.3 Odds Ratio and Youden’s Index 162.3 ROC Curve 172.3.1 ROC Curves: Artificial Heart Valve and Case Study 1 172.3.2 ROC Curve Assumption 182.3.3 Smooth, Fitted ROC Curves 192.3.4 Advantages of ROC Curves 192.4 Area Under the ROC Curve 202.4.1 Interpretation of the Area Under the ROC Curve 202.4.2 Magnitudes of the Area Under the ROC Curve 212.4.3 Area Under the ROC Curve: Case Study 1 212.4.4 Misinterpretations of the Area Under the ROC Curve 232.5 Sensitivity at Fixed FPR 252.6 Partial Area Under the ROC Curve 252.7 Likelihood Ratios 262.7.1 Three Examples to Illustrate Likelihood Ratios 272.7.2 Limitations of Likelihood Ratios 282.7.3 Proper and Improper ROC Curves 292.8 ROC Analysis When the True Diagnosis Is Not Binary 302.9 C-statistics and Other Measures to Compare Prediction Models 322.10 Detection and Localization of Multiple Lesions 332.11 Positive and Negative Predictive Values, Bayes’ Theorem, and Case Study 2 352.11.1 Bayes’ Theorem 362.12 Optimal Decision Threshold on the ROC Curve 382.12.1 Optimal Thresholds for Maximizing Classification 382.12.2 Optimal Threshold for Minimizing Cost 392.12.3 Optimal Decision Threshold: Rapid Eye Movement as a Marker for Depression Example 392.13 Interpreting the Results of Multiple Tests 402.13.1 Parallel Testing 402.13.2 Serial, or Sequential, Testing 413 Design of Diagnostic Accuracy Studies 453.1 Establish the Objective of the Study 453.2 Identify the Target Patient Population 493.3 Select a Sampling Plan for Patients 503.3.1 Phase I: Exploratory Studies 503.3.2 Phase II: Challenge Studies 503.3.3 Phase III: Clinical Studies 523.4 Select the Gold Standard 563.5 Choose a Measure of Accuracy 613.6 Identify Target Reader Population 633.7 Select Sampling Plan for Readers 643.8 Plan Data Collection 643.8.1 Format for Test Results 643.8.2 Data Collection for Reader Studies 653.8.3 Reader Training 713.9 Plan Data Analyses 723.9.1 Statistical Hypotheses 723.9.2 Planning for Covariate Adjustment 733.9.3 Reporting Test Results 753.10 Determine Sample Size 774 Estimation and Hypothesis Testing in a Single Sample 794.1 Binary-scale Data 804.1.1 Sensitivity and Specificity 804.1.2 Predictive Value of a Positive or Negative 824.1.3 Sensitivity, Specificity, and Predictive Values with Clustered Binary-scale Data 844.1.4 Likelihood Ratio 864.1.5 Odds Ratio 884.2 Ordinal-scale Data 894.2.1 Empirical ROC Curve 904.2.2 Fitting a Smooth Curve 904.2.3 Estimation of Sensitivity at a Particular FPR 954.2.4 Area and Partial Area Under the ROC Curve (Parametric Methods) 974.2.5 ci Estimation 994.2.6 Area and Partial Area Under the ROC Curve (Nonparametric Methods) 1024.2.7 Nonparametric Analysis of Clustered Data 1054.2.8 Degenerate Data 1064.2.9 Choosing Between Parametric, Semi-parametric, and Nonparametric Methods 1084.3 Continuous-scale Data 1084.3.1 Empirical ROC Curve 1094.3.2 Fitting a Smooth ROC Curve – Parametric, Semi-parametric, and Nonparametric Methods 1104.3.3 Confidence Bands Around the Estimated ROC Curve 1154.3.4 Area and Partial Area Under the ROC Curve – Parametric, Nonparametric, and Semi-parametric Methods 1164.3.5 CIs for the Area Under the ROC Curve 1174.3.6 Fixed FPR – Sensitivity and the Decision Threshold 1194.3.7 Choosing the Optimal Operating Point and Decision Threshold 1224.3.8 Choosing Between Parametric, Semi-parametric, and Nonparametric Methods 1254.4 Testing the Hypothesis that the ROC Curve Area or Partial Area Is a Specific Value 1264.4.1 Testing Whether MRA Has Any Ability to Detect Significant Carotid Stenosis 1275 Comparing the Accuracy of Two Diagnostic Tests 1295.1 Binary-scale Data 1305.1.1 Sensitivity and Specificity 1305.1.2 Sensitivity and Specificity of Clustered Binary Data 1325.1.3 Predictive Probability of a Positive or Negative 1345.2 Ordinal- and Continuous-scale Data 1365.2.1 Testing the Equality of Two ROC Curves 1375.2.2 Comparing ROC Curves at a Particular Point 1405.2.3 Determining the Range of FPRs for Which TPRs Differ 1415.2.4 Comparison of the Area or Partial Area 1435.3 Tests of Equivalence 1485.3.1 Testing Whether ROC Curve Areas Are Equivalent: Case Study 3 1506 Sample Size Calculations 1536.1 Studies Estimating the Accuracy of a Single Test 1536.1.1 Sample Size Calculations for Estimating Sensitivity and/or Specificity – Case Study 1 1536.1.2 Sample Size for Estimating the Area Under the ROC Curve – Case Study 2 1556.1.3 Studies with Clustered Data 1576.1.4 Testing the Hypothesis That the ROC Area Is Equal to a Particular Value 1586.1.5 Sample Size for Estimating Sensitivity at Fixed FPR – Case Study 2 1586.1.6 Sample Size for Estimating the Partial Area Under the ROC Curve – Case Study 2 1606.2 Sample Size for Detecting a Difference in Accuracies of Two Tests 1616.2.1 Sample Size Software 1616.2.2 Sample Size for Comparing Tests’ Sensitivity and/or Specificity – Case Study 1 1616.2.3 Sample Size for Comparing Tests’ Positive and Negative Predictive Values – Case Study 1 1636.2.4 Sample Size for Comparing Tests’ Area Under the ROC Curve – Case Study 2 1646.2.5 Sample Size for Comparing Tests with Clustered Data 1656.2.6 Sample Size for Comparing Tests’ Sensitivity at Fixed FPR – Case Study 2 1666.2.7 Sample Size for Comparing Tests’ Partial Area Under the ROC Curve – Case Study 2 1676.3 Sample Size for Assessing Non-inferiority or Equivalency of Two Tests 1696.4 Sample Size for Determining a Suitable Cutoff Value 1726.5 Sample Size Determination for Multi-reader Studies 1736.5.1 MRMC Sample Size Software 1736.5.2 MRMC Sample Size Calculations with No Pilot Data 1746.5.3 MRMC Sample Size Calculations with Pilot Data 1796.6 Alternative to Sample Size Formulae 1807 Introduction to Meta-analysis for Diagnostic Accuracy Studies 1817.1 Objectives 1827.2 Retrieval of the Literature 1827.2.1 Literature Search: Meta-analysis of Ultrasound for PAD 1867.3 Inclusion/Exclusion Criteria 1867.3.1 Inclusion/Exclusion Criteria: Meta-analysis of Ultrasound for PAD 1887.4 Extracting Information from the Literature 1887.4.1 Data Abstraction: Meta-analysis of Ultrasound for PAD 1907.5 Statistical Analysis 1907.5.1 Binary-scale Data 1907.5.2 Ordinal- or Continuous-scale Data 1917.5.3 Area Under the ROC Curve 2007.5.4 Other Methods 2027.6 Public Presentation 2027.6.1 Presentation of Results: Meta-analysis of Ultrasound for PAD 204Part II Advanced Methods 2058 Regression Analysis for Independent ROC Data 2078.1 Four Clinical Studies 2088.1.1 Surgical Lesion in a Carotid Vessel Example 2088.1.2 Pancreatic Cancer Example 2088.1.3 Hearing Test Example 2088.1.4 Staging of Prostate Cancer Example 2098.2 Regression Models for Continuous-scale Tests 2108.2.1 Indirect Regression Models for ROC Curves 2118.2.2 Direct Regression Models for ROC Curves 2148.3 Regression Models for Ordinal-scale Tests 2288.3.1 Indirect Regression Models for Latent Smooth ROC Curves 2288.3.2 Direct Regression Model for Latent Smooth ROC Curves 2308.3.3 Detection of Periprostatic Invasion with Ultrasound 2328.4 Covariate AROC Curves of Continuous-scale Tests 2339 Analysis of Multiple Reader and/or Multiple Test Studies 2359.1 Studies Comparing Multiple Tests with Covariates 2359.1.1 Two Clinical Studies 2359.1.2 Indirect Regression Models for Ordinal-scale Tests 2369.1.3 Direct Regression Models for Continuous-scale Tests 2419.2 Studies with Multiple Readers and Multiple Tests 2459.2.1 Three MRMC Studies 2459.2.2 Statistical Methods for Analyzing MRMC Studies 2469.2.3 Analysis of the Interstitial Disease Example 2549.2.4 Comparisons Between MRMC Methods 25410 Methods for Correcting Verification Bias 25710.1 Examples 25810.1.1 Hepatic Scintigraph 25810.1.2 Screening Tests for Dementia Disorder Example 25810.1.3 Fever of Uncertain Origin 25910.1.4 CT and MRI for Staging Pancreatic Cancer Example 25910.1.5 NACC MDS on AD 25910.2 Impact of Verification Bias 26010.3 A Single Binary-scale Test 26110.3.1 Correction Methods Under the MAR Assumption 26110.3.2 Correction Methods Without the MAR Assumption 26310.3.3 Analysis of Hepatic Scintigraph Example, Continued 26510.4 Correlated Binary-scale Tests 26710.4.1 ml Approach Without Any Covariates 26710.4.2 Analysis of Two Screening Tests for Dementia Disorder Example 27210.4.3 ml Approach with Covariates 27310.4.4 Analysis of Two Screening Tests for Dementia Disorder Example, Continued 27510.5 A Single Ordinal-scale Test 27610.5.1 ML Approach Without Covariates 27610.5.2 Analysis of Fever of Uncertain Origin Example 28010.5.3 ML Approach with Covariates 28010.5.4 Analysis of New Screening Test for Dementia Disorder 28410.6 Correlated Ordinal-scale Tests 28610.6.1 Weighted Estimating Equation Approaches for Latent Smooth ROC Curves 28710.6.2 Likelihood-based Approach for ROC Areas 29110.6.3 Analysis of CT and MRI for Staging Pancreatic Cancer 29510.7 Continuous-scale Tests 29610.7.1 Estimation of ROC Curves and Their Areas Under the MAR Assumption 29710.7.2 Estimation of ROC Curves and Areas Under a Non-MAR Process 30311 Methods for Correcting Imperfect Gold Standard Bias 31311.1 Examples 31411.1.1 Binary Stool Test for Strongyloides Infection 31411.1.2 Binary Tine Test for Tuberculosis 31411.1.3 Binary-scale X-rays for Pleural Thickening 31411.1.4 Bioassays for HIV 31511.1.5 Ordinal-scale Evaluation by Pathologists for Detecting Carcinoma In Situ of the Uterine Cervix 31511.1.6 Ordinal-scale and Continuous-scale MRA for Carotid Artery Stenosis 31511.2 Impact of Imperfect Gold Standard Bias 31511.3 One Single Binary Test in a Single Population 31711.3.1 Conditions for Model Identifiability 31811.3.2 The Frequentist-based ML Method Under an Identifiable Model 31911.3.3 Bayesian Methods Under a Non-identifiable Model 32011.3.4 Analysis of Strongyloides Infection Example 32211.4 One Single Binary Test in G Populations 32411.4.1 Estimation Methods 32411.4.2 Tuberculosis Example 32711.5 Multiple Binary Tests in One Single Population 32911.5.1 Checking for Model Identifiability 32911.5.2 ml Estimates Under the CIA 33011.5.3 Assessment of Pleural Thickening Example 33111.5.4 ml Approaches Under Identifiable Conditional Dependence Models 33111.5.5 Bioassays for HIV Example 33611.5.6 Bayesian Methods Under Conditional Dependence Models 33911.5.7 Analysis of the MRA for Carotid Stenosis Example 34011.6 Multiple Binary Tests in G Populations 34111.6.1 ml Approaches Under the CIA 34211.6.2 ml Approach Without the CIA Assumption 34311.7 Multiple Ordinal-scale Tests in One Single Population 34311.7.1 Nonparametric Estimation of ROC Curves Under the CIA 34311.7.2 Estimation of ROC Curves Under Some Conditional Dependence Models 34511.7.3 Analysis of Ordinal-scale Tests for Detecting Carcinoma In Situ of the Uterine Cervix 34611.8 Multiple-scale Tests in One Single Population 34711.8.1 Reanalysis of the Accuracy of Continuous-scale MRA for Detection of Significant Carotid Stenosis 35112 Location-specific ROC Methods for Diagnostic Imaging 35312.1 Examples 35312.1.1 Example One: A Clinical Study of Computer-aided Detection of Mammographic Masses 35312.1.2 Example Two: A Clinical Study of Pulmonary Computer-aided Diagnosis Medical Software 35412.1.3 Example Three: A Clinical Study of 3D Magnetic Resonance Angiography 35412.2 LROC Approach 35512.2.1 Swensson’s Parametric Model 35612.2.2 Nonparametric LROC Approach 35812.2.3 Analysis of Example One 35912.3 FROC Approach 36012.3.1 FROC-type Curves 36112.3.2 Radiological Search Model 36712.3.3 Resampling Methods 36812.3.4 FROC Analysis Method in MRMC Study 36912.3.5 Analysis of Example Two 37312.4 ROI Approach 37712.4.1 Nonparametric ROI Analysis Method 37712.4.2 iROI Paradigm 37912.4.3 Analysis of Example Two 38212.4.4 Analysis of Example Three 38212.5 Comparison Between Location-specific ROC Methods 38313 Technical Performance (“Accuracy”) of Quantitative Imaging Biomarkers 38513.1 Quantitative Imaging Biomarkers 38513.1.1 Definitions 38513.1.2 Technical Performance Characteristics 38513.1.3 Illustrative Example of QIBs for Nonalcoholic Fatty Liver Disease 38713.2 Technical Performance Characteristics of a QIB 38713.2.1 Basic Model 38713.2.2 Limits of Detection and Quantitation 38813.3 Precision 38913.3.1 Study Design for Measuring Precision of QIBs 38913.3.2 Precision Metrics and Their Estimation 39213.3.3 Sample Size Considerations for Precision Studies 39613.3.4 Precision Profile 39813.4 Bias and Linearity 39813.4.1 Study Design for Assessing Bias 39813.4.2 Bias Metrics and Their Estimation 40113.4.3 Sample Size Considerations for Bias Studies 40413.4.4 Bias Profile 40413.5 Other Metrics of QIB Performance 40513.5.1 Limits of Agreement 40513.5.2 Coverage Probability 40613.5.3 Total Deviation Index 40613.5.4 Mean Squared Deviation 40713.6 Clinical Performance 40713.6.1 Integrated Biomarkers 40813.6.2 Integral Biomarkers 40913.6.3 Multi-parametric Applications 41114 Medical Test Efficacy and Effectiveness 41314.1 General Notation 41414.2 Prognostic Effects 41614.3 Predictive Effects 41614.4 Test Strategies for Assigning Treatments 41714.5 Explanatory Versus Pragmatic Trials of Tests 41714.6 Explanatory Trial Designs 41814.6.1 Stratified Design 41814.6.2 Enrichment (Targeted) Design 41814.6.3 Discordant Pairs Design 41914.7 Pragmatic Trial Designs 42014.7.1 Test Strategy Design 42014.7.2 Comparative Test Strategy Design 42214.8 Adaptive Treatment Strategy Trial Designs 42314.9 Treatment Selection Tests 42414.10 Follow-on Treatment Selection Tests 42514.10.1 Bridging Studies 42514.10.2 Concordance Study and Indirect Estimation of Drug Efficacy 42714.11 Bibliographic Notes 42814.11.1 Biomarkers and Markers 42814.11.2 Prognostic and Predictive Factors 42814.11.3 Medical Test Trial Design Literature 42914.11.4 Stratified Trial Design 42914.11.5 Enrichment Design 43014.11.6 Causal Effects of Testing 43015 Statistical Analysis for Meta-analysis 43315.1 Binary-scale Data 43315.1.1 Random Effects Model: Meta-analysis of Ultrasound for PAD 43415.2 Ordinal- or Continuous-scale Data 43515.2.1 Random Effects Model 43515.2.2 Bivariate Approach 43615.2.3 Binary Regression Model 43815.2.4 Hierarchical SROC Curve 43915.2.5 Other Methods 44115.3 ROC Curve Area 44115.3.1 EB Method: Meta-analysis of DST 44315.4 Publication Bias 44316 Risk Prediction 44916.1 Risk Calculators 45116.1.1 Breast Cancer Calculators 45116.1.2 The Gail Model 45216.1.3 Framingham Heart Study Risk Calculator 45316.2 Calibration 45416.2.1 Definitions of Calibration 45416.2.2 Log-logistic Regression 45516.2.3 Cox Proportional Hazards 45616.2.4 Predictiveness Curve 45716.2.5 Calibration Plot 45816.2.6 Goodness-of-fit Tests 45916.2.7 Testing for Equivalence 46116.2.8 Brier Score 46316.2.9 Model-based ROC Curve 46316.3 Discrimination 46416.3.1 Risk Distribution 46516.3.2 Time-dependent Predictive Values 46616.3.3 Time-dependent ROC Curves 46716.4 Appendix 16-A: Survival Analysis 46916.4.1 Structure of Survival Data 47016.4.2 Basic Concepts in Survival Analysis 47116.4.3 Estimation 47216.5 Appendix 16-B: Survival Analysis for Competing Risks 47516.5.1 Competing Risk Survival Analysis 47516.5.2 Survival Analysis for Competing Risks 47616.5.3 Estimation 47816.5.4 Kaplan–Meier Estimator of Pure Risk 47816.5.5 Aalen–Johansen Estimator of Absolute Risk 47916.6 Appendix 16-C: Stochastic Processes for Survival Analysis 48016.7 Appendix 16-D: Bibliographic Notes 48116.7.1 Bibliographic Notes for Section 16.3 48116.7.2 Bibliographic Notes for Section 16.5 481Appendix-A: Case Studies and Chapter 8 Data 485Appendix-B: Jackknife and Bootstrap Methods of Estimating Variances and Confidence Intervals 513Bibliography 517Index 555