Laila Dybkjær - Böcker

This book is based on publications from the ISCA Tutorial and Research Workshop on Multi-Modal Dialogue in Mobile Environments held at Kloster Irsee, Germany, in 2002. The workshop covered various aspects of devel- ment and evaluation of spoken multimodal dialogue systems and components with particular emphasis on mobile environments, and discussed the state-- the-art within this area. On the development side the major aspects addressed include speech recognition, dialogue management, multimodal output gene- tion, system architectures, full applications, and user interface issues. On the evaluation side primarily usability evaluation was addressed. A number of high quality papers from the workshop were selected to form the basis of this book. The volume is divided into three major parts which group together the ov- all aspects covered by the workshop. The selected papers have all been - tended, reviewed and improved after the workshop to form the backbone of the book. In addition, we have supplemented each of the three parts by an invited contribution intended to serve as an overview chapter.

This book is based on publications from the ISCA Tutorial and Research Workshop on Multi-Modal Dialogue in Mobile Environments held at Kloster Irsee, Germany, in 2002. The workshop covered various aspects of devel- ment and evaluation of spoken multimodal dialogue systems and components with particular emphasis on mobile environments, and discussed the state-- the-art within this area. On the development side the major aspects addressed include speech recognition, dialogue management, multimodal output gene- tion, system architectures, full applications, and user interface issues. On the evaluation side primarily usability evaluation was addressed. A number of high quality papers from the workshop were selected to form the basis of this book. The volume is divided into three major parts which group together the ov- all aspects covered by the workshop. The selected papers have all been - tended, reviewed and improved after the workshop to form the backbone of the book. In addition, we have supplemented each of the three parts by an invited contribution intended to serve as an overview chapter.

References 74 Part II Annotation and Analysis of Multimodal Data: Speech and Gesture 4 FORM 79 Craig H. Martell 1. Introduction 79 2. Structure of FORM 80 3. Annotation Graphs 85 4. Annotation Example 86 5. Preliminary Inter-Annotator Agreement Results 88 6. Conclusion: Applications to HLT and HCI? 90 Appendix: Other Tools, Schemes and Methods of Gesture Analysis 91 References 95 5 97 On the Relationships among Speech, Gestures, and Object Manipulation in Virtual Environments: Initial Evidence Andrea Corradini and Philip R. Cohen 1. Introduction 97 2. Study 99 3. Data Analysis 101 4. Results 103 5. Discussion 106 6. Related Work 106 7. Future Work 108 8. Conclusions 108 Appendix: Questionnaire MYST III - EXILE 110 References 111 6 113 Analysing Multimodal Communication Patrick G. T. Healey, Marcus Colman and Mike Thirlwell 1. Introduction 113 2. Breakdown and Repair 117 3. Analysing Communicative Co-ordination 125 4. Discussion 126 References 127 7 131 Do Oral Messages Help Visual Search? Noëlle Carbonell and Suzanne Kieffer 1. Context and Motivation 131 2. Methodology and Experimental Set-Up 134 3. Results: Presentation and Discussion 141 4. Conclusion 153 References 154 Contents vii 8 159 Geometric and Statistical Approaches to Audiovisual Segmentation Trevor Darrell, John W. Fisher III, Kevin W. Wilson, and Michael R. Siracusa 1. Introduction 159 2. Related Work 160 3. Multimodal Multisensor Domain 162 4. Results 166 5. Single Multimodal Sensor Domain 167 6.

References 74 Part II Annotation and Analysis of Multimodal Data: Speech and Gesture 4 FORM 79 Craig H. Martell 1. Introduction 79 2. Structure of FORM 80 3. Annotation Graphs 85 4. Annotation Example 86 5. Preliminary Inter-Annotator Agreement Results 88 6. Conclusion: Applications to HLT and HCI? 90 Appendix: Other Tools, Schemes and Methods of Gesture Analysis 91 References 95 5 97 On the Relationships among Speech, Gestures, and Object Manipulation in Virtual Environments: Initial Evidence Andrea Corradini and Philip R. Cohen 1. Introduction 97 2. Study 99 3. Data Analysis 101 4. Results 103 5. Discussion 106 6. Related Work 106 7. Future Work 108 8. Conclusions 108 Appendix: Questionnaire MYST III - EXILE 110 References 111 6 113 Analysing Multimodal Communication Patrick G. T. Healey, Marcus Colman and Mike Thirlwell 1. Introduction 113 2. Breakdown and Repair 117 3. Analysing Communicative Co-ordination 125 4. Discussion 126 References 127 7 131 Do Oral Messages Help Visual Search? Noëlle Carbonell and Suzanne Kieffer 1. Context and Motivation 131 2. Methodology and Experimental Set-Up 134 3. Results: Presentation and Discussion 141 4. Conclusion 153 References 154 Contents vii 8 159 Geometric and Statistical Approaches to Audiovisual Segmentation Trevor Darrell, John W. Fisher III, Kevin W. Wilson, and Michael R. Siracusa 1. Introduction 159 2. Related Work 160 3. Multimodal Multisensor Domain 162 4. Results 166 5. Single Multimodal Sensor Domain 167 6.

In its nine chapters, this book provides an overview of the state-of-the-art and best practice in several sub-fields of evaluation of text and speech systems and components. The evaluation aspects covered include speech and speaker recognition, speech synthesis, animated talking agents, part-of-speech tagging, parsing, and natural language software like machine translation, information retrieval, question answering, spoken dialogue systems, data resources, and annotation schemes. With its broad coverage and original contributions this book is unique in the field of evaluation of speech and language technology.This book is of particular relevance to advanced undergraduate students, PhD students, academic and industrial researchers, and practitioners.

In its nine chapters, this book provides an overview of the state-of-the-art and best practice in several sub-fields of evaluation of text and speech systems and components. The evaluation aspects covered include speech and speaker recognition, speech synthesis, animated talking agents, part-of-speech tagging, parsing, and natural language software like machine translation, information retrieval, question answering, spoken dialogue systems, data resources, and annotation schemes. With its broad coverage and original contributions this book is unique in the field of evaluation of speech and language technology.This book is of particular relevance to advanced undergraduate students, PhD students, academic and industrial researchers, and practitioners.

This book edition highlights recent trends and important issues that still remain only partially solved or even unsolved within the broad ?eld of discourse and dialogue. The ?eld is discussed and illustrated both from an overall spoken (multimodal) dialogue system perspective as well as from a more compone- related perspective. Issues discussed include, for example, discourse and dialogue modelling in research versus industrial spoken dialogue systems, evaluation, miscommunication and error handling, grounding, statistical and corpus-based approaches to discourse and dialogue modelling, data analysis, and corpus annotation and annotation tools. We believe that jointly this collection of chapters provides a good picture of how far we are today within discourse and dialogue and of important ch- lenges ahead. On this background we hope that computer scientists, engineers, and others who work in the broad area of discourse and dialogue, no matter if from an academic or industrial perspective, may bene?t from the book and ?nd it useful to their own work. Graduate students and Ph.D. students focusing on topics in discourse and dialogue may also ?nd the book interesting and pro?t from reading it. This book edition is based on a selected subset of papers from the successful 6th SIGdial Workshop on Discourse and Dialogue held in Lisbon, Portugal, in September 2005 in conjunction with the 9th Eurospeech (Interspeech) conf- ence. SIGdial is a special interest group on discourse and dialogue sponsored jointly by the two parent organisations ACL (Association for Computational Linguistics) and ISCA (International Speech Communication Association).

This preface tells the story of how Multimodal Usability responds to a special challenge. Chapter 1 describes the goals and structure of this book. The idea of describing how to make multimodal computer systems usable arose in the European Network of Excellence SIMILAR – “Taskforce for cre- ing human-machine interfaces SIMILAR to human-human communication”, 2003– 2007, www. similar. cc. SIMILAR brought together people from multimodal signal processing and usability with the aim of creating enabling technologies for new kinds of multimodal systems and demonstrating results in research prototypes. Most of our colleagues in the network were, in fact, busy extracting features and guring out how to demonstrate progress in working interactive systems, while claiming not to have too much of a notion of usability in system development and evaluation. It was proposed that the authors support the usability of the many multimodal pro- types underway by researching and presenting a methodology for building usable multimodal systems. We accepted the challenge, rst and foremost, no doubt, because the formidable team spirit in SIMILAR could make people accept outrageous things. Second, h- ing worked for nearly two decades on making multimodal systems usable, we were curious – curious at the opportunity to try to understand what happens to traditional usability work, that is, work in human–computer interaction centred around tra- tional graphical user interfaces (GUIs), when systems become as multimodal and as advanced in other ways as those we build in research today.

This preface tells the story of how Multimodal Usability responds to a special challenge. Chapter 1 describes the goals and structure of this book. The idea of describing how to make multimodal computer systems usable arose in the European Network of Excellence SIMILAR – “Taskforce for cre- ing human-machine interfaces SIMILAR to human-human communication”, 2003– 2007, www. similar. cc. SIMILAR brought together people from multimodal signal processing and usability with the aim of creating enabling technologies for new kinds of multimodal systems and demonstrating results in research prototypes. Most of our colleagues in the network were, in fact, busy extracting features and guring out how to demonstrate progress in working interactive systems, while claiming not to have too much of a notion of usability in system development and evaluation. It was proposed that the authors support the usability of the many multimodal pro- types underway by researching and presenting a methodology for building usable multimodal systems. We accepted the challenge, rst and foremost, no doubt, because the formidable team spirit in SIMILAR could make people accept outrageous things. Second, h- ing worked for nearly two decades on making multimodal systems usable, we were curious – curious at the opportunity to try to understand what happens to traditional usability work, that is, work in human–computer interaction centred around tra- tional graphical user interfaces (GUIs), when systems become as multimodal and as advanced in other ways as those we build in research today.

This book constitutes the refereed proceedings of the international Tutorial and Research Workshop on Affective Dialogue Systems, ADS 2004, held in Kloster Irsee, Germany in June 2004.The 21 revised full papers and 14 revised short papers presented were carefully reviewed and selected for presentation. The papers are organized in topical sections on emotion recognition; affective user modeling; emotional databases, annotation schemes, and tools; affective conversational agents and dialogue systems; synthesis of emotional speech and facial animations; affective tutoring systems; evaluation of affective dialogue systems; and demonstrations.

This book constitutes the refereed proceedings of the International Tutorial and Research Workshop on Perception and Interactive Technologies, PIT 2006, held at Kloster Irsee, Germany, June 2006. The book presents 16 revised full papers together with 4 revised poster papers and 6 system demonstration papers, organized in topical sections on head pose and eye gaze tracking, modeling and simulation of perception, integrating information from multiple channels, and more.

This book constitutes the refereed proceedings of the 4th IEEE Tutorial and Research Workshop on Perception and Interactive Technologies for Speech-Based Systems, PIT 2008, held in Kloster Irsee, Germany, in June 2008. The 37 revised full papers presented together with 1 invited keynote lecture were carefully selected from numerous submissions for inclusion in the book. The papers are organized in topical sections on multimodal and spoken dialogue systems, classification of dialogue acts and sound, recognition of eye gaze, head poses, mimics and speech as well as combinations of modalities, vocal emotion recognition, human-like and social dialogue systems, and evaluation methods for multimodal dialogue systems.

A description of the design and implementation of spoken language dialogue within the context of spoken language dialogue systems development. Using an applications-oriented SLDS developed through the Danish Dialogue project, the authors describe the complete process involved; and in so doing present several innovative practical tools, such as dialogue design guidelines, in-depth evaluation methodologies and speech functionality analysis. Their approach is firmly applications-oriented, describing the results applicable to industry and showing how the development of advanced applications drives research rather than vice versa. For everyone working on the R&D of spoken language services, especially in the area of telecommunications.