Cognitive Interviewing Methodology

Kristen Miller, PhD, is Director of the Questionnaire Design Research Laboratory, National Center for Health Statistics (NCHS). She designs and implements research projects on data quality and comparability, question response, questionnaire design, and cognitive methods theory and methodology. Dr. Miller is a coeditor of Question Evaluation Methods: Contributing to the Science of Data Quality, also published by Wiley. Stephanie Willson, PhD, is Senior Research Methodologist at the Questionnaire Design Research Laboratory, NCHS. Her research interests include the study of construct validity in survey questions and the methodological evaluation of cognitive interviewing, with particular emphasis on data collection and data analysis. Valerie Chepp, PhD, is Assistant Professor in the Department of Sociology at Hamline University. Dr. Chepp has 5 years of experience working as a research methodologist at the Questionnaire Design Research Laboratory, NCHS, and her research interests include cultural sociology, theories of inequality and social change, and qualitative research methods. José-Luis Padilla, PhD, is Professor in the Department of Social Psychology at University of Granada, Spain. His research interests include cross-cultural psychology, psychometrics, and social research methods.

• Foreword xiiiGordon WillisAcknowledgments xvContributors xvii1 Introduction 1Kristen Miller1.1 Cognitive Interviewing Methodology, 22 Foundations and New Directions 7Valerie Chepp and Caroline Gray2.1 Introduction, 72.2 Sociology and the Interpretivist Tradition, 82.3 New Directions: Interpretation and Cognition, 92.4 Methodological Implications for Cognitive Interviewing, 112.5 Conclusion, 143 Data Collection 15Stephanie Willson and Kristen Miller3.1 Introduction, 153.2 Cognitive Interviewing Study Sample, 153.2.1 Considerations of Sample Design, 163.3 The Cognitive Interview, 203.3.1 Differing Approaches to Cognitive Interviewing, 203.3.2 Different Kinds of Data: Respondent as Evaluator versus Respondent as Story Teller, 223.4 The Role of Interviewer, 283.4.1 Interviewer as Data Collector, 293.4.2 Interviewer as Researcher, 303.5 Conclusion, 334 Analysis 35Kristen Miller, Stephanie Willson, Valerie Chepp, and J. Michael Ryan4.1 Introduction, 354.2 Analysis of Cognitive Interviews: Overview, 364.3 Analytic Steps for Cognitive Interviews, 384.3.1 Step 1: Conducting the Interview, 394.3.2 Step Two: Producing Interview Summaries, 404.3.3 Step Three: Developing a Thematic Schema, 424.3.4 Step Four: Developing an Advanced Schema, 444.3.5 Step Five: Making Conclusions, 474.4 The Benefits of a Complete Analysis, 494.5 Conclusion, 505 Assessing Translated Questions via Cognitive Interviewing 51Alis´u Schoua-Glusberg and Ana Villar5.1 Introduction, 515.2 Why Use Cognitive Testing in Multilingual Survey Research, 515.2.1 Multilingual Research Settings, 525.2.2 Instrument Production in Multilingual Settings, 535.3 Translation and Translation Assessment Procedures, 545.3.1 Team Translation Approaches, 545.3.2 Translation Assessment Procedures, 545.3.3 Pretesting as Part of Translation Assessment, 555.4 Cognitively Testing Translations of Survey Questions, 575.4.1 Cognitive Interviewers, 585.4.2 Respondent Selection, 585.4.3 Introduction, Protocol, and Implementation, 595.4.4 Analysis, 595.5 Problems Uncovered by Cognitive Testing of Translations, 605.5.1 Uncovering Translation Problems, 615.5.2 Uncovering Problems with the Source Question, 645.5.3 Uncovering Problems Related to Cultural Differences – Need for Adaptation, 655.6 Conclusion, 666 Conveying Results 69Valerie Chepp and Paul Scanlon6.1 Introduction, 696.2 Contents of a Cognitive Interviewing Report, 706.2.1 Introduction, 706.2.2 Summary of Findings, 716.2.3 Methods, 736.2.4 Question-by-Question Review, 776.2.5 Appendix, 806.3 Characteristics of a Cognitive Interviewing Report, 806.3.1 Transparency, 816.3.2 Reflexivity, 826.4 Conclusion, 847 Case Study: Evaluation of a Sexual Identity Question 85Kristen Miller and J. Michael Ryan7.1 Introduction, 857.2 Background, 867.2.1 Intended Construct for the National Health Interview Survey Sexual Identity Question, 867.2.2 Review of Data Quality Problems, 877.2.3 Development of an Improved Sexual Identity Question, 897.3 Case Study: Cognitive Interviewing Evaluation of the National Health Interview Survey Revised Sexual Identity Question, 937.3.1 Recruitment and Respondent Demographics, 937.3.2 Interviewing Procedures, 937.3.3 Data Analysis, 957.4 Case Study Findings, 967.4.1 Summary of Question Performance, 967.4.2 Basis of Respondents’ Answers, 977.4.3 Cases of Response Problems, 997.4.4 Interpretation of Categories, 1007.4.5 Study Conclusions, 1057.5 Conclusion, 1068 Analysis Software for Cognitive Interviewing Studies: Q-Notes and Q-Bank 107Justin Mezetin and Meredith Massey8.1 Introduction, 1078.2 Q-Notes Analysis Features, 1088.2.1 Level 1: Conducting Interviews, 1088.2.2 Level 2: Summarizing Interview Notes, 1108.2.3 Level 3: Comparing Across Respondents, 1128.2.4 Level 4: Comparing Across Groups, 1158.2.5 Level 5: Drawing Conclusions about Question Performance, 1188.3 Project Management Features, 1188.3.1 Streamlined Communication, 1198.3.2 Interview Data Collection, 1198.3.3 Respondent Descriptors, 1218.3.4 Controlled Access by Project, 1218.3.5 Adding Questions, 1228.3.6 Question Translations, 1248.3.7 Coding Schemes, 1248.4 Q-Bank: Making Cognitive Interview Findings Publicly Accessible, 1258.5 Q-Bank Features, 1268.5.1 Searching for Questions, 1268.5.2 Advanced Search, 1278.5.3 Question Details, 1288.5.4 Value of Q-Bank, 1298.6 Q-Bank: Challenges for the Past and Future, 1308.7 Conclusion, 1309 Cognitive Interviewing in Mixed Research 133Isabel Benitez Baena and Jos´e-Luis Padilla9.1 Introduction, 1339.2 The Mixed Research Paradigm: Characteristics and Design, 1349.2.1 Cognitive Interviewing Studies and Research Design, 1359.3 Mixed Method Research and Survey Question Evaluation, 1369.3.1 Case 1: Cognitive Interviewing and Survey Field Testing, 1379.3.2 Case 2: Cognitive Interviewing and Differential Item Functioning (DIF), 1459.3.3 Case 3: Cognitive Interviewing and Psychometric Scales, 1489.4 Conclusion, 15210 Conclusion 153Kristen Miller, Stephanie Willson, Valerie Chepp, and Jos´e-Luis Padilla10.1 Introduction, 15310.2 Summary of Practices, 15410.2.1 Data Collection, 15410.2.2 Analysis, 15610.2.3 Documenting Study Findings, 15710.3 New Directions, 15910.3.1 Topics for Examination, 15910.3.2 Mixed Method Research, 16010.3.3 Accepted Standards of Cognitive Interviewing Studies, 161Key Concepts 163Question Evaluation Resources 167References 171Index 181