Laila Dybkjær – författare

This book is based on publications from the ISCA Tutorial and Research Workshop on Multi-Modal Dialogue in Mobile Environments held at Kloster Irsee, Germany, in 2002. The workshop covered various aspects of devel- ment and evaluation of spoken multimodal dialogue systems and components with particular emphasis on mobile environments, and discussed the state-- the-art within this area. On the development side the major aspects addressed include speech recognition, dialogue management, multimodal output gene- tion, system architectures, full applications, and user interface issues. On the evaluation side primarily usability evaluation was addressed. A number of high quality papers from the workshop were selected to form the basis of this book. The volume is divided into three major parts which group together the ov- all aspects covered by the workshop. The selected papers have all been - tended, reviewed and improved after the workshop to form the backbone of the book. In addition, we have supplemented each of the three parts by an invited contribution intended to serve as an overview chapter.

This book is based on publications from the ISCA Tutorial and Research Workshop on Multi-Modal Dialogue in Mobile Environments held at Kloster Irsee, Germany, in 2002. The workshop covered various aspects of devel- ment and evaluation of spoken multimodal dialogue systems and components with particular emphasis on mobile environments, and discussed the state-- the-art within this area. On the development side the major aspects addressed include speech recognition, dialogue management, multimodal output gene- tion, system architectures, full applications, and user interface issues. On the evaluation side primarily usability evaluation was addressed. A number of high quality papers from the workshop were selected to form the basis of this book. The volume is divided into three major parts which group together the ov- all aspects covered by the workshop. The selected papers have all been - tended, reviewed and improved after the workshop to form the backbone of the book. In addition, we have supplemented each of the three parts by an invited contribution intended to serve as an overview chapter.

This book is based on publications from the ISCA Tutorial and Research Workshop on Multi-Modal Dialogue in Mobile Environments held at Kloster Irsee, Germany, in 2002. The workshop covered various aspects of devel- ment and evaluation of spoken multimodal dialogue systems and components with particular emphasis on mobile environments, and discussed the state-- the-art within this area. On the development side the major aspects addressed include speech recognition, dialogue management, multimodal output gene- tion, system architectures, full applications, and user interface issues. On the evaluation side primarily usability evaluation was addressed. A number of high quality papers from the workshop were selected to form the basis of this book. The volume is divided into three major parts which group together the ov- all aspects covered by the workshop. The selected papers have all been - tended, reviewed and improved after the workshop to form the backbone of the book. In addition, we have supplemented each of the three parts by an invited contribution intended to serve as an overview chapter.

References 74 Part II Annotation and Analysis of Multimodal Data: Speech and Gesture 4 FORM 79 Craig H. Martell 1. Introduction 79 2. Structure of FORM 80 3. Annotation Graphs 85 4. Annotation Example 86 5. Preliminary Inter-Annotator Agreement Results 88 6. Conclusion: Applications to HLT and HCI? 90 Appendix: Other Tools, Schemes and Methods of Gesture Analysis 91 References 95 5 97 On the Relationships among Speech, Gestures, and Object Manipulation in Virtual Environments: Initial Evidence Andrea Corradini and Philip R. Cohen 1. Introduction 97 2. Study 99 3. Data Analysis 101 4. Results 103 5. Discussion 106 6. Related Work 106 7. Future Work 108 8. Conclusions 108 Appendix: Questionnaire MYST III - EXILE 110 References 111 6 113 Analysing Multimodal Communication Patrick G. T. Healey, Marcus Colman and Mike Thirlwell 1. Introduction 113 2. Breakdown and Repair 117 3. Analysing Communicative Co-ordination 125 4. Discussion 126 References 127 7 131 Do Oral Messages Help Visual Search? Noëlle Carbonell and Suzanne Kieffer 1. Context and Motivation 131 2. Methodology and Experimental Set-Up 134 3. Results: Presentation and Discussion 141 4. Conclusion 153 References 154 Contents vii 8 159 Geometric and Statistical Approaches to Audiovisual Segmentation Trevor Darrell, John W. Fisher III, Kevin W. Wilson, and Michael R. Siracusa 1. Introduction 159 2. Related Work 160 3. Multimodal Multisensor Domain 162 4. Results 166 5. Single Multimodal Sensor Domain 167 6.

References 74 Part II Annotation and Analysis of Multimodal Data: Speech and Gesture 4 FORM 79 Craig H. Martell 1. Introduction 79 2. Structure of FORM 80 3. Annotation Graphs 85 4. Annotation Example 86 5. Preliminary Inter-Annotator Agreement Results 88 6. Conclusion: Applications to HLT and HCI? 90 Appendix: Other Tools, Schemes and Methods of Gesture Analysis 91 References 95 5 97 On the Relationships among Speech, Gestures, and Object Manipulation in Virtual Environments: Initial Evidence Andrea Corradini and Philip R. Cohen 1. Introduction 97 2. Study 99 3. Data Analysis 101 4. Results 103 5. Discussion 106 6. Related Work 106 7. Future Work 108 8. Conclusions 108 Appendix: Questionnaire MYST III - EXILE 110 References 111 6 113 Analysing Multimodal Communication Patrick G. T. Healey, Marcus Colman and Mike Thirlwell 1. Introduction 113 2. Breakdown and Repair 117 3. Analysing Communicative Co-ordination 125 4. Discussion 126 References 127 7 131 Do Oral Messages Help Visual Search? Noëlle Carbonell and Suzanne Kieffer 1. Context and Motivation 131 2. Methodology and Experimental Set-Up 134 3. Results: Presentation and Discussion 141 4. Conclusion 153 References 154 Contents vii 8 159 Geometric and Statistical Approaches to Audiovisual Segmentation Trevor Darrell, John W. Fisher III, Kevin W. Wilson, and Michael R. Siracusa 1. Introduction 159 2. Related Work 160 3. Multimodal Multisensor Domain 162 4. Results 166 5. Single Multimodal Sensor Domain 167 6.

References 74 Part II Annotation and Analysis of Multimodal Data: Speech and Gesture 4 FORM 79 Craig H. Martell 1. Introduction 79 2. Structure of FORM 80 3. Annotation Graphs 85 4. Annotation Example 86 5. Preliminary Inter-Annotator Agreement Results 88 6. Conclusion: Applications to HLT and HCI? 90 Appendix: Other Tools, Schemes and Methods of Gesture Analysis 91 References 95 5 97 On the Relationships among Speech, Gestures, and Object Manipulation in Virtual Environments: Initial Evidence Andrea Corradini and Philip R. Cohen 1. Introduction 97 2. Study 99 3. Data Analysis 101 4. Results 103 5. Discussion 106 6. Related Work 106 7. Future Work 108 8. Conclusions 108 Appendix: Questionnaire MYST III - EXILE 110 References 111 6 113 Analysing Multimodal Communication Patrick G. T. Healey, Marcus Colman and Mike Thirlwell 1. Introduction 113 2. Breakdown and Repair 117 3. Analysing Communicative Co-ordination 125 4. Discussion 126 References 127 7 131 Do Oral Messages Help Visual Search? Noëlle Carbonell and Suzanne Kieffer 1. Context and Motivation 131 2. Methodology and Experimental Set-Up 134 3. Results: Presentation and Discussion 141 4. Conclusion 153 References 154 Contents vii 8 159 Geometric and Statistical Approaches to Audiovisual Segmentation Trevor Darrell, John W. Fisher III, Kevin W. Wilson, and Michael R. Siracusa 1. Introduction 159 2. Related Work 160 3. Multimodal Multisensor Domain 162 4. Results 166 5. Single Multimodal Sensor Domain 167 6.

In its nine chapters, this book provides an overview of the state-of-the-art and best practice in several sub-fields of evaluation of text and speech systems and components. The evaluation aspects covered include speech and speaker recognition, speech synthesis, animated talking agents, part-of-speech tagging, parsing, and natural language software like machine translation, information retrieval, question answering, spoken dialogue systems, data resources, and annotation schemes. With its broad coverage and original contributions this book is unique in the field of evaluation of speech and language technology.This book is of particular relevance to advanced undergraduate students, PhD students, academic and industrial researchers, and practitioners.

In its nine chapters, this book provides an overview of the state-of-the-art and best practice in several sub-fields of evaluation of text and speech systems and components. The evaluation aspects covered include speech and speaker recognition, speech synthesis, animated talking agents, part-of-speech tagging, parsing, and natural language software like machine translation, information retrieval, question answering, spoken dialogue systems, data resources, and annotation schemes. With its broad coverage and original contributions this book is unique in the field of evaluation of speech and language technology.This book is of particular relevance to advanced undergraduate students, PhD students, academic and industrial researchers, and practitioners.

This book edition highlights recent trends and important issues that still remain only partially solved or even unsolved within the broad ?eld of discourse and dialogue. The ?eld is discussed and illustrated both from an overall spoken (multimodal) dialogue system perspective as well as from a more compone- related perspective. Issues discussed include, for example, discourse and dialogue modelling in research versus industrial spoken dialogue systems, evaluation, miscommunication and error handling, grounding, statistical and corpus-based approaches to discourse and dialogue modelling, data analysis, and corpus annotation and annotation tools. We believe that jointly this collection of chapters provides a good picture of how far we are today within discourse and dialogue and of important ch- lenges ahead. On this background we hope that computer scientists, engineers, and others who work in the broad area of discourse and dialogue, no matter if from an academic or industrial perspective, may bene?t from the book and ?nd it useful to their own work. Graduate students and Ph.D. students focusing on topics in discourse and dialogue may also ?nd the book interesting and pro?t from reading it. This book edition is based on a selected subset of papers from the successful 6th SIGdial Workshop on Discourse and Dialogue held in Lisbon, Portugal, in September 2005 in conjunction with the 9th Eurospeech (Interspeech) conf- ence. SIGdial is a special interest group on discourse and dialogue sponsored jointly by the two parent organisations ACL (Association for Computational Linguistics) and ISCA (International Speech Communication Association).

This book edition highlights recent trends and important issues that still remain only partially solved or even unsolved within the broad ?eld of discourse and dialogue. The ?eld is discussed and illustrated both from an overall spoken (multimodal) dialogue system perspective as well as from a more compone- related perspective. Issues discussed include, for example, discourse and dialogue modelling in research versus industrial spoken dialogue systems, evaluation, miscommunication and error handling, grounding, statistical and corpus-based approaches to discourse and dialogue modelling, data analysis, and corpus annotation and annotation tools. We believe that jointly this collection of chapters provides a good picture of how far we are today within discourse and dialogue and of important ch- lenges ahead. On this background we hope that computer scientists, engineers, and others who work in the broad area of discourse and dialogue, no matter if from an academic or industrial perspective, may bene?t from the book and ?nd it useful to their own work. Graduate students and Ph.D. students focusing on topics in discourse and dialogue may also ?nd the book interesting and pro?t from reading it. This book edition is based on a selected subset of papers from the successful 6th SIGdial Workshop on Discourse and Dialogue held in Lisbon, Portugal, in September 2005 in conjunction with the 9th Eurospeech (Interspeech) conf- ence. SIGdial is a special interest group on discourse and dialogue sponsored jointly by the two parent organisations ACL (Association for Computational Linguistics) and ISCA (International Speech Communication Association).

Designing Interactive Speech Systems describes the design and implementation of spoken language dialogue within the context of SLDS (spoken language dialogue systems) development. Using an applications-oriented SLDS developed through the Danish Dialogue project, the authors describe the complete process involved in designing such a system; and in doing so present several innovative practical tools, such as dialogue design guideline s, in-depth evaluation methodologies, and speech functionality analysis. The approach taken is firmly applications-oriented, describing the results of research applicable to industry and showing how the development of advanced applications drives research rather than the other way around. All those working on the research and development of spoken language services, especially in the area of telecommunications, will benefit from reading this book.

This preface tells the story of how Multimodal Usability responds to a special challenge. Chapter 1 describes the goals and structure of this book. The idea of describing how to make multimodal computer systems usable arose in the European Network of Excellence SIMILAR – “Taskforce for cre- ing human-machine interfaces SIMILAR to human-human communication”, 2003– 2007, www. similar. cc. SIMILAR brought together people from multimodal signal processing and usability with the aim of creating enabling technologies for new kinds of multimodal systems and demonstrating results in research prototypes. Most of our colleagues in the network were, in fact, busy extracting features and guring out how to demonstrate progress in working interactive systems, while claiming not to have too much of a notion of usability in system development and evaluation. It was proposed that the authors support the usability of the many multimodal pro- types underway by researching and presenting a methodology for building usable multimodal systems. We accepted the challenge, rst and foremost, no doubt, because the formidable team spirit in SIMILAR could make people accept outrageous things. Second, h- ing worked for nearly two decades on making multimodal systems usable, we were curious – curious at the opportunity to try to understand what happens to traditional usability work, that is, work in human–computer interaction centred around tra- tional graphical user interfaces (GUIs), when systems become as multimodal and as advanced in other ways as those we build in research today.

This preface tells the story of how Multimodal Usability responds to a special challenge. Chapter 1 describes the goals and structure of this book. The idea of describing how to make multimodal computer systems usable arose in the European Network of Excellence SIMILAR – “Taskforce for cre- ing human-machine interfaces SIMILAR to human-human communication”, 2003– 2007, www. similar. cc. SIMILAR brought together people from multimodal signal processing and usability with the aim of creating enabling technologies for new kinds of multimodal systems and demonstrating results in research prototypes. Most of our colleagues in the network were, in fact, busy extracting features and guring out how to demonstrate progress in working interactive systems, while claiming not to have too much of a notion of usability in system development and evaluation. It was proposed that the authors support the usability of the many multimodal pro- types underway by researching and presenting a methodology for building usable multimodal systems. We accepted the challenge, rst and foremost, no doubt, because the formidable team spirit in SIMILAR could make people accept outrageous things. Second, h- ing worked for nearly two decades on making multimodal systems usable, we were curious – curious at the opportunity to try to understand what happens to traditional usability work, that is, work in human–computer interaction centred around tra- tional graphical user interfaces (GUIs), when systems become as multimodal and as advanced in other ways as those we build in research today.

This preface tells the story of how Multimodal Usability responds to a special challenge. Chapter 1 describes the goals and structure of this book. The idea of describing how to make multimodal computer systems usable arose in the European Network of Excellence SIMILAR – “Taskforce for cre- ing human-machine interfaces SIMILAR to human-human communication”, 2003– 2007, www. similar. cc. SIMILAR brought together people from multimodal signal processing and usability with the aim of creating enabling technologies for new kinds of multimodal systems and demonstrating results in research prototypes. Most of our colleagues in the network were, in fact, busy extracting features and guring out how to demonstrate progress in working interactive systems, while claiming not to have too much of a notion of usability in system development and evaluation. It was proposed that the authors support the usability of the many multimodal pro- types underway by researching and presenting a methodology for building usable multimodal systems. We accepted the challenge, rst and foremost, no doubt, because the formidable team spirit in SIMILAR could make people accept outrageous things. Second, h- ing worked for nearly two decades on making multimodal systems usable, we were curious – curious at the opportunity to try to understand what happens to traditional usability work, that is, work in human–computer interaction centred around tra- tional graphical user interfaces (GUIs), when systems become as multimodal and as advanced in other ways as those we build in research today.

Human conversational partners are able, at least to a certain extent, to detect the speaker’s or listener’s emotional state and may attempt to respond to it accordingly. When instead one of the interlocutors is a computer a number of questions arise, such as the following: To what extent are dialogue systems able to simulate such behaviors? Can we learn the mechanisms of emotional be- viors from observing and analyzing the behavior of human speakers? How can emotionsbeautomaticallyrecognizedfromauser’smimics,gesturesandspeech? What possibilities does a dialogue system have to express emotions itself? And, very importantly, would emotional system behavior be desirable at all? Given the state of ongoing research into incorporating emotions in dialogue systems we found it timely to organize a Tutorial and Research Workshop on A?ectiveDialogueSystems(ADS2004)atKlosterIrseein GermanyduringJune 14–16, 2004. After two successful ISCA Tutorial and Research Workshops on Multimodal Dialogue Systems at the same location in 1999 and 2002, we felt that a workshop focusing on the role of a?ect in dialogue would be a valuable continuation of the workshop series. Due to its interdisciplinary nature, the workshop attracted submissions from researchers with very di?erent backgrounds and from many di?erent research areas, working on, for example, dialogue processing, speech recognition, speech synthesis, embodied conversational agents, computer graphics, animation, user modelling, tutoring systems, cognitive systems, and human-computer inter- tion.

Human conversational partners are able, at least to a certain extent, to detect the speaker’s or listener’s emotional state and may attempt to respond to it accordingly. When instead one of the interlocutors is a computer a number of questions arise, such as the following: To what extent are dialogue systems able to simulate such behaviors? Can we learn the mechanisms of emotional be- viors from observing and analyzing the behavior of human speakers? How can emotionsbeautomaticallyrecognizedfromauser’smimics,gesturesandspeech? What possibilities does a dialogue system have to express emotions itself? And, very importantly, would emotional system behavior be desirable at all? Given the state of ongoing research into incorporating emotions in dialogue systems we found it timely to organize a Tutorial and Research Workshop on A?ectiveDialogueSystems(ADS2004)atKlosterIrseein GermanyduringJune 14–16, 2004. After two successful ISCA Tutorial and Research Workshops on Multimodal Dialogue Systems at the same location in 1999 and 2002, we felt that a workshop focusing on the role of a?ect in dialogue would be a valuable continuation of the workshop series. Due to its interdisciplinary nature, the workshop attracted submissions from researchers with very di?erent backgrounds and from many di?erent research areas, working on, for example, dialogue processing, speech recognition, speech synthesis, embodied conversational agents, computer graphics, animation, user modelling, tutoring systems, cognitive systems, and human-computer inter- tion.

This book constitutes the refereed proceedings of the International Tutorial and Research Workshop on Perception and Interactive Technologies, PIT 2006, held at Kloster Irsee, Germany, June 2006. The book presents 16 revised full papers together with 4 revised poster papers and 6 system demonstration papers, organized in topical sections on head pose and eye gaze tracking, modeling and simulation of perception, integrating information from multiple channels, and more.

This book constitutes the refereed proceedings of the 4th IEEE Tutorial and Research Workshop on Perception and Interactive Technologies for Speech-Based Systems, PIT 2008, held in Kloster Irsee, Germany, in June 2008. The 37 revised full papers presented together with 1 invited keynote lecture were carefully selected from numerous submissions for inclusion in the book. The papers are organized in topical sections on multimodal and spoken dialogue systems, classification of dialogue acts and sound, recognition of eye gaze, head poses, mimics and speech as well as combinations of modalities, vocal emotion recognition, human-like and social dialogue systems, and evaluation methods for multimodal dialogue systems.

This book constitutes the refereed proceedings of the 4th IEEE Tutorial and Research Workshop on Perception and Interactive Technologies for Speech-Based Systems, PIT 2008, held in Kloster Irsee, Germany, in June 2008. The 37 revised full papers presented together with 1 invited keynote lecture were carefully selected from numerous submissions for inclusion in the book. The papers are organized in topical sections on multimodal and spoken dialogue systems, classification of dialogue acts and sound, recognition of eye gaze, head poses, mimics and speech as well as combinations of modalities, vocal emotion recognition, human-like and social dialogue systems, and evaluation methods for multimodal dialogue systems.

Designing Interactive Speech Systems describes the design and implementation of spoken language dialogue within the context of SLDS (spoken language dialogue systems) development. Using an applications-oriented SLDS developed through the Danish Dialogue project, the authors describe the complete process involved in designing such a system; and in doing so present several innovative practical tools, such as dialogue design guideline s, in-depth evaluation methodologies, and speech functionality analysis. The approach taken is firmly applications-oriented, describing the results of research applicable to industry and showing how the development of advanced applications drives research rather than the other way around. All those working on the research and development of spoken language services, especially in the area of telecommunications, will benefit from reading this book.