Total Survey Error in Practice

Paul P. Biemer, PhD, is distinguished fellow at RTI International and associate director of Survey Research and Development at the Odum Institute, University of North Carolina, USA.Edith de Leeuw, PhD, is professor of survey methodology in the Department of Methodology and Statistics at Utrecht University, the Netherlands.Stephanie Eckman, PhD, is fellow at RTI International, USA.Brad Edwards is vice president, director of Field Services, and deputy area director at Westat, USA.Frauke Kreuter, PhD, is professor and director of the Joint Program in Survey Methodology, University of Maryland, USA; professor of statistics and methodology at the University of Mannheim, Germany; and head of the Statistical Methods Research Department at the Institute for Employment Research, Germany.Lars E. Lyberg, PhD, is senior advisor at Inizio, Sweden.N. Clyde Tucker, PhD, is principal survey methodologist at the American Institutes for Research, USA.Brady T. West, PhD, is research associate professor in the Survey Research Center, located within the Institute for Social Research at the University of Michigan (U-M), and also serves as statistical consultant on the Consulting for Statistics, Computing and Analytics Research (CSCAR) team at U-M, USA.

• Notes on Contributors xixPreface xxvSection 1 The Concept of TSE and the TSE Paradigm 11 The Roots and Evolution of the Total Survey Error Concept 3Lars E. Lyberg and Diana Maria Stukel1.1 Introduction and Historical Backdrop 31.2 Specific Error Sources and Their Control or Evaluation 51.3 Survey Models and Total Survey Design 101.4 The Advent of More Systematic Approaches Toward Survey Quality 121.5 What the Future Will Bring 16References 182 Total Twitter Error: Decomposing Public Opinion Measurement on Twitter from a Total Survey Error Perspective 23Yuli Patrick Hsieh and Joe Murphy2.1 Introduction 232.2 Social Media: An Evolving Online Public Sphere 252.3 Components of Twitter Error 272.4 Studying Public Opinion on the Twittersphere and the Potential Error Sources of Twitter Data: Two Case Studies 312.5 Discussion 402.6 Conclusion 42References 433 Big Data: A Survey Research Perspective 47Reg Baker3.1 Introduction 473.2 Definitions 483.3 The Analytic Challenge: From Database Marketing to Big Data and Data Science 563.4 Assessing Data Quality 583.5 Applications in Market, Opinion, and Social Research 593.6 The Ethics of Research Using Big Data 623.7 The Future of Surveys in a Data-Rich Environment 62References 654 The Role of Statistical Disclosure Limitation in Total Survey Error 71Alan F. Karr4.1 Introduction 714.2 Primer on SDL 724.3 TSE-Aware SDL 754.4 Edit-Respecting SDL 794.5 SDL-Aware TSE 834.6 Full Unification of Edit, Imputation, and SDL 844.7 “Big Data” Issues 874.8 Conclusion 89Acknowledgments 91References 92Section 2 Implications for Survey Design 955 The Undercoverage–Nonresponse Tradeoff 97Stephanie Eckman and Frauke Kreuter5.1 Introduction 975.2 Examples of the Tradeoff 985.3 Simple Demonstration of the Tradeoff 995.4 Coverage and Response Propensities and Bias 1005.5 Simulation Study of Rates and Bias 1025.6 Costs 1105.7 Lessons for Survey Practice 111References 1126 Mixing Modes: Tradeoffs Among Coverage, Nonresponse, and Measurement Error 115Roger Tourangeau6.1 Introduction 1156.2 The Effect of Offering a Choice of Modes 1186.3 Getting People to Respond Online 1196.4 Sequencing Different Modes of Data Collection 1206.5 Separating the Effects of Mode on Selection and Reporting 1226.6 Maximizing Comparability Versus Minimizing Error 1276.7 Conclusions 129References 1307 Mobile Web Surveys: A Total Survey Error Perspective 133Mick P. Couper, Christopher Antoun, and Aigul Mavletova7.1 Introduction 1337.2 Coverage 1357.3 Nonresponse 1377.4 Measurement Error 1427.5 Links Between Different Error Sources 1487.6 The Future of Mobile Web Surveys 149References 1508 The Effects of a Mid-Data Collection Change in Financial Incentives on Total Survey Error in the National Survey of Family Growth: Results from a Randomized Experiment 155James Wagner, Brady T. West, Heidi Guyer, Paul Burton, Jennifer Kelley, Mick P. Couper, and William D. Mosher8.1 Introduction 1558.2 Literature Review: Incentives in Face-to-Face Surveys 1568.3 Data and Methods 1598.4 Results 1638.5 Conclusion 173References 1759 A Total Survey Error Perspective on Surveys in Multinational, Multiregional, and Multicultural Contexts 179Beth-Ellen Pennell, Kristen Cibelli Hibben, Lars E. Lyberg, Peter Ph. Mohler, and Gelaye Worku9.1 Introduction 1799.2 TSE in Multinational, Multiregional, and Multicultural Surveys 1809.3 Challenges Related to Representation and Measurement Error Components in Comparative Surveys 1849.4 QA and QC in 3MC Surveys 192References 19610 Smartphone Participation in Web Surveys: Choosing Between the Potential for Coverage, Nonresponse, and Measurement Error 203Gregg Peterson, Jamie Griffin, John LaFrance, and JiaoJiao Li10.1 Introduction 20310.2 Prevalence of Smartphone Participation in Web Surveys 20610.3 Smartphone Participation Choices 20910.4 Instrument Design Choices 21210.5 Device and Design Treatment Choices 21610.6 Conclusion 21810.7 Future Challenges and Research Needs 219Appendix 10.A: Data Sources 220Appendix 10.B: Smartphone Prevalence in Web Surveys 221Appendix 10.C: Screen Captures from Peterson et al. (2013) Experiment 225Appendix 10.D: Survey Questions Used in the Analysis of the Peterson et al. (2013) Experiment 229References 23111 Survey Research and the Quality of Survey Data Among Ethnic Minorities 235Joost Kappelhof11.1 Introduction 23511.2 On the Use of the Terms Ethnicity and Ethnic Minorities 23611.3 On the Representation of Ethnic Minorities in Surveys 237 Ethnic Minorities 24111.4 Measurement Issues 24211.5 Comparability, Timeliness, and Cost Concerns 24411.6 Conclusion 247References 248Section 3 Data Collection and Data Processing Applications 25312 Measurement Error in Survey Operations Management: Detection, Quantification, Visualization, and Reduction 255Brad Edwards, Aaron Maitland, and Sue Connor12.1 TSE Background on Survey Operations 25612.2 Better and Better: Using Behavior Coding (CARIcode) and Paradata to Evaluate and Improve Question (Specification) Error and Interviewer Error 25712.3 Field-Centered Design: Mobile App for Rapid Reporting and Management 26112.4 Faster and Cheaper: Detecting Falsification With GIS Tools 26512.5 Putting It All Together: Field Supervisor Dashboards 26812.6 Discussion 273References 27513 Total Survey Error for Longitudinal Surveys 279Peter Lynn and Peter J. Lugtig13.1 Introduction 27913.2 Distinctive Aspects of Longitudinal Surveys 28013.3 TSE Components in Longitudinal Surveys 28113.4 Design of Longitudinal Surveys from a TSE Perspective 28513.5 Examples of Tradeoffs in Three Longitudinal Surveys 29013.6 Discussion 294References 29514 Text Interviews on Mobile Devices 299Frederick G. Conrad, Michael F. Schober, Christopher Antoun, Andrew L. Hupp, and H. Yanna Yan14.1 Texting as a Way of Interacting 30014.2 Contacting and Inviting Potential Respondents through Text 30314.3 Texting as an Interview Mode 30314.4 Costs and Efficiency of Text Interviewing 31214.5 Discussion 314References 31515 Quantifying Measurement Errors in Partially Edited Business Survey Data 319Thomas Laitila, Karin Lindgren, Anders Norberg, and Can Tongur15.1 Introduction 31915.2 Selective Editing 32015.3 Effects of Errors Remaining After SE 32515.4 Case Study: Foreign Trade in Goods Within the European Union 32815.5 Editing Big Data 33415.6 Conclusions 335References 335Section 4 Evaluation and Improvement 33916 Estimating Error Rates in an Administrative Register and Survey Questions Using a Latent Class Model 341Daniel L. Oberski16.1 Introduction 34116.2 Administrative and Survey Measures of Neighborhood 34216.3 A Latent Class Model for Neighborhood of Residence 34516.4 Results 348Appendix 16.A: Program Input and Data 355Acknowledgments 357References 35717 ASPIRE: An Approach for Evaluating and Reducing the Total Error in Statistical Products with Application to Registers and the National Accounts 359Paul P. Biemer, Dennis Trewin, Heather Bergdahl, and Yingfu Xie17.1 Introduction and Background 35917.2 Overview of ASPIRE 36017.3 The ASPIRE Model 36217.4 Evaluation of Registers 36717.5 National Accounts 37117.6 A Sensitivity Analysis of GDP Error Sources 37617.7 Concluding Remarks 379Appendix 17.A: Accuracy Dimension Checklist 381References 38418 Classification Error in Crime Victimization Surveys: A Markov Latent Class Analysis 387Marcus E. Berzofsky and Paul P. Biemer18.1 Introduction 38718.2 Background 38918.3 Analytic Approach 39218.4 Model Selection 39618.5 Results 39918.6 Discussion and Summary of Findings 40418.7 Conclusions 407Appendix 18.A: Derivation of the Composite False-Negative Rate 407Appendix 18.B: Derivation of the Lower Bound for False-Negative Rates from a Composite Measure 408Appendix 18.C: Examples of Latent GOLD Syntax 408References 41019 Using Doorstep Concerns Data to Evaluate and Correct for Nonresponse Error in a Longitudinal Survey 413Ting Yan19.1 Introduction 41319.2 Data and Methods 41619.3 Results 41819.4 Discussion 428Acknowledgment 430References 43020 Total Survey Error Assessment for Sociodemographic Subgroups in the 2012 U.S. National Immunization Survey 433Kirk M. Wolter, Vicki J. Pineau, Benjamin Skalland, Wei Zeng, James A. Singleton, Meena Khare, Zhen Zhao, David Yankey, and Philip J. Smith20.1 Introduction 43320.2 TSE Model Framework 43420.3 Overview of the National Immunization Survey 43720.4 National Immunization Survey: Inputs for TSE Model 44020.5 National Immunization Survey TSE Analysis 44520.6 Summary 452References 45321 Establishing Infrastructure for the Use of Big Data to Understand Total Survey Error: Examples from Four Survey Research Organizations Overview 457Brady T. WestPart 1 Big Data Infrastructure at the Institute for Employment Research (IAB) 458Antje Kirchner, Daniela Hochfellner, Stefan BenderAcknowledgments 464References 464Part 2 Using Administrative Records Data at the U.S. Census Bureau: Lessons Learned from Two Research Projects Evaluating Survey Data 467Elizabeth M. Nichols, Mary H. Mulry, and Jennifer Hunter ChildsAcknowledgments and Disclaimers 472References 472Part 3 Statistics New Zealand’s Approach to Making Use of Alternative Data Sources in a New Era of Integrated Data 474Anders Holmberg and Christine BycroftReferences 478Part 4 Big Data Serving Survey Research: Experiences at the University of Michigan Survey Research Center 478Grant Benson and Frost HubbardAcknowledgments and Disclaimers 484References 484Section 5 Estimation and Analysis 48722 Analytic Error as an Important Component of Total Survey Error: Results from a Meta-Analysis 489Brady T. West, Joseph W. Sakshaug, and Yumi Kim22.1 Overview 48922.2 Analytic Error as a Component of TSE 49022.3 Appropriate Analytic Methods for Survey Data 49222.4 Methods 49522.5 Results 49722.6 Discussion 505Acknowledgments 508References 50823 Mixed-Mode Research: Issues in Design and Analysis 511Joop Hox, Edith de Leeuw, and Thomas Klausch23.1 Introduction 51123.2 Designing Mixed-Mode Surveys 51223.3 Literature Overview 51423.4 Diagnosing Sources of Error in Mixed-Mode Surveys 51623.5 Adjusting for Mode Measurement Effects 52323.6 Conclusion 527References 52824 The Effect of Nonresponse and Measurement Error on Wage Regression across Survey Modes: A Validation Study 531Antje Kirchner and Barbara Felderer24.1 Introduction 53124.2 Nonresponse and Response Bias in Survey Statistics 53224.3 Data and Methods 53424.4 Results 54124.5 Summary and Conclusion 546Acknowledgments 547Appendix 24.A 548Appendix 24.B 549References 55425 Errors in Linking Survey and Administrative Data 557Joseph W. Sakshaug and Manfred Antoni25.1 Introduction 55725.2 Conceptual Framework of Linkage and Error Sources 55925.3 Errors Due to Linkage Consent 56125.4 Erroneous Linkage with Unique Identifiers 56525.5 Erroneous Linkage with Nonunique Identifiers 56725.6 Applications and Practical Guidance 56825.7 Conclusions and Take-Home Points 571References 571Index 575