Crowdsourcing for Speech Processing

Maxine Eskenazi, Carnegie Mellon University, USADr. Eskenazi is Principal Systems Scientist at the Language Technologies Institute, Carnegie Mellon University, USA. She has authored over 100 scientific papers in the areas of computer assisted language learning and speech and spoken dialog systems. Her work has produced such systems as the Let's Go spoken dialog system and the REAP vocabulary tutor. She is also the founder and CTO of the Carnegie Speech Company.Gina-Anne Levow, University of Washington, USADr. Levow is currently an Assistant Professor in the Department of Linguistics, University of Washington, USA. Prior to joining the faculty at the University of Washington, she served on the faculty at the University of Chicago in the Department of Computer Science and as a Research Fellow at the University of Manchester, UK. She served on the Editorial Board of Computational Linguistics and as Associate Editor of ACM Transactions on Asian Language Processing.Helen Meng, The Chinese University of Hong Kong, Hong KongDr. Meng is Founder and Director of the Human-Computer Communications Laboratory at The Chinese University of Hong Kong, and is also the Founder and Co-Director of the Microsoft-CUHK Joint Laboratory for Human-Centric Computing and Interface Technologies, which was conferred the national status of the Ministry of Education of China (MoE) Key Laboratory in 2008. Prof. Meng also served as an Associate Dean (Research) of the Faculty of Engineering from 2006 to 2010. She serves as Editor-in-Chief of the IEEE Transactions on Audio, Speech and Language Processing.Gabriel Parent, Amazon.com, USAGabriel Parent is a Software Development Engineer at Amazon.com working on solving natural language related problems. His main research focuses were human-computer interaction through spoken dialog systems and crowdsourcing.David Suendermann, Baden-Wuerttemberg Cooperative State University, GermanyDr. Sundermann is currently full Professor of Computer Science at the Baden-Wuerttemberg Cooperative State University, Stuttgart, Germany. He is also the Principal Speech Scientist of SpeechCycle, New York, USA which has been recognized by Deloitte as a "Technology Fast 500" company based on revenue growth. He has authored more than 70 publications and patents, including a book and six book chapters.

• ContentsList of Contributors xiiiPreface xv1 An Overview 1Maxine Eskénazi1.1 Origins of Crowdsourcing 21.2 Operational Definition of Crowdsourcing 31.3 Functional Definition of Crowdsourcing 31.4 Some Issues 41.5 Some Terminology 61.6 Acknowledgments 6References 62 The Basics 8Maxine Eskénazi2.1 An Overview of the Literature on Crowdsourcing for Speech Processing 82.2 Alternative Solutions 142.3 Some Ready-Made Platforms for Crowdsourcing 152.4 Making Task Creation Easier 172.5 Getting Down to Brass Tacks 172.6 Quality Control 292.7 Judging the Quality of the Literature 322.8 Some Quick Tips 332.9 Acknowledgments 33References 33Further reading 353 Collecting Speech from Crowds 37Ian McGraw3.1 A Short History of Speech Collection 383.2 Technology for Web-Based Audio Collection 433.3 Example: WAMI Recorder 493.4 Example: The WAMI Server 523.5 Example: Speech Collection on Amazon Mechanical Turk 593.6 Using the Platform Purely for Payment 653.7 Advanced Methods of Crowdsourced Audio Collection 673.8 Summary 693.9 Acknowledgments 69References 704 Crowdsourcing for Speech Transcription 72Gabriel Parent4.1 Introduction 724.2 Transcribing Speech 734.3 Preparing the Data 804.4 Setting Up the Task 834.5 Submitting the Open Call 914.6 Quality Control 954.7 Conclusion 1024.8 Acknowledgments 103References 1035 How to Control and Utilize Crowd-Collected Speech 106Ian McGraw and Joseph Polifroni5.1 Read Speech 1075.2 Multimodal Dialog Interactions 1115.3 Games for Speech Collection 1205.4 Quizlet 1215.5 Voice Race 1235.6 Voice Scatter 1295.7 Summary 1355.8 Acknowledgments 135References 1366 Crowdsourcing in Speech Perception 137Martin Cooke, Jon Barker, and Maria Luisa Garcia Lecumberri6.1 Introduction 1376.2 Previous Use of Crowdsourcing in Speech and Hearing 1386.3 Challenges 1406.4 Tasks 1456.5 BigListen: A Case Study in the Use of Crowdsourcing to Identify Words in Noise 1496.6 Issues for Further Exploration 1676.7 Conclusions 169References 1697 Crowdsourced Assessment of Speech Synthesis 173Sabine Buchholz, Javier Latorre, and Kayoko Yanagisawa7.1 Introduction 1737.2 Human Assessment of TTS 1747.3 Crowdsourcing for TTS: What Worked and What Did Not 1777.4 Related Work: Detecting and Preventing Spamming 1937.5 Our Experiences: Detecting and Preventing Spamming 1957.6 Conclusions and Discussion 212References 2148 Crowdsourcing for Spoken Dialog System Evaluation 217Zhaojun Yang, Gina-Anne Levow, and Helen Meng8.1 Introduction 2178.2 Prior Work on Crowdsourcing: Dialog and Speech Assessment 2208.3 Prior Work in SDS Evaluation 2218.4 Experimental Corpus and Automatic Dialog Classification 2258.5 Collecting User Judgments on Spoken Dialogs with Crowdsourcing 2268.6 Collected Data and Analysis 2308.7 Conclusions and Future Work 2388.8 Acknowledgments 238References 2399 Interfaces for Crowdsourcing Platforms 241Christoph Draxler9.1 Introduction 2419.2 Technology 2429.3 Crowdsourcing Platforms 2539.4 Interfaces to Crowdsourcing Platforms 2619.5 Summary 278References 27810 Crowdsourcing for Industrial Spoken Dialog Systems 280David Suendermann and Roberto Pieraccini10.1 Introduction 28010.2 Architecture 28310.3 Transcription 28710.4 Semantic Annotation 29010.5 Subjective Evaluation of Spoken Dialog Systems 29610.6 Conclusion 300References 30011 Economic and Ethical Background of Crowdsourcing for Speech 303Gilles Adda, Joseph J. Mariani, Laurent Besacier, and Hadrien Gelas11.1 Introduction 30311.2 The Crowdsourcing Fauna 30411.3 Economic and Ethical Issues 30711.4 Under-Resourced Languages: A Case Study 31611.5 Toward Ethically Produced Language Resources 32211.6 Conclusion 330Disclaimer 331References 331Index 335