Audio Source Separation and Speech Enhancement

EMMANUEL VINCENT is a Senior Research Scientist with Inria, Nancy, France. His research focuses on machine learning for speech and audio signal processing. He has been working on audio source separation for 15 years and co-authored over 180 publications in this field. His contributions include harmonic nonnegative matrix factorization, full-rank spatial covariance modeling, joint spatial/spectral estimation, deep learning based multichannel source separation, and objective performance metrics. He has given several keynotes, tutorials and summer school lectures, including at Interspeech 2012 and 2016, WASPAA 2015 and LVA/ICA 2015. He is a founding chair of the series of Signal Separation Evaluation Campaigns (SiSEC) and CHiME Speech Separation and Recognition Challenges and the chair of ISCA's special interest group on Robust Speech Processing. TUOMAS VIRTANEN is a Professor with the Laboratory of Signal Processing, Tampere University of Technology, Finland, where he is leading the Audio Research Group. He is known for his pioneering work on single-channel sound source separation using nonnegative matrix factorization, and its application to noise-robust speech recognition, music content analysis, and sound event detection. His research interests also include content analysis and processing of audio signals in general. He has authored more than 170 publications and received four best paper awards. He is an IEEE Senior Member, a member of the Audio and Acoustic Signal Processing Technical Committee of IEEE Signal Processing Society, Associate Editor of IEEE/ACM Transaction on Audio, Speech, and Language Processing, and recipient of the ERC 2014 Starting Grant. SHARON GANNOT is a Full Professor at the Faculty of Engineering, Bar-Ilan University, Israel, where he is heading the Speech and Signal Processing laboratory and the Signal Processing Track. His research interests include multi-microphone speech processing; distributed algorithms for noise reduction and speaker separation; array processing on manifold; dereverberation; single-microphone speech enhancement; and speaker localization and tracking. He received the Bar-Ilan University's Outstanding Lecturer Award for 2010 and 2014 and the Bar-Ilan Rector Innovation in Research Award in 2018. He has co-authored over 200 publications and lectured tutorials at ICASSP 2012, EUSIPCO 2012, ICASSP 2013, and EUSIPCO 2013 and a keynote address at IWAENC 2012. He was a co-editor of the book Speech Processing in Modern Communication: Challenges and Perspectives (Springer, 2012). He also served as an Associate Editor and a Senior Area Chair of the IEEE Transactions on Speech, Audio and Language Processing. He currently serves as the Chair of the IEEE Audio and Acoustic Signal Processing (AASP) Technical Committee.

• List of Authors xviiPreface xxiAcknowledgment xxiiiNotations xxvAcronyms xxixAbout the Companion Website xxxiPart I Prerequisites 11 Introduction 3Emmanuel Vincent, Sharon Gannot, and Tuomas Virtanen1.1 Why are Source Separation and Speech Enhancement Needed? 31.2 What are the Goals of Source Separation and Speech Enhancement? 41.3 How can Source Separation and Speech Enhancement be Addressed? 91.4 Outline 11Bibliography 122 Time-Frequency Processing: Spectral Properties 15Tuomas Virtanen, Emmanuel Vincent, and Sharon Gannot2.1 Time-Frequency Analysis and Synthesis 152.2 Source Properties in the Time-Frequency Domain 232.3 Filtering in the Time-Frequency Domain 252.4 Summary 28Bibliography 283 Acoustics: Spatial Properties 31Emmanuel Vincent, Sharon Gannot, and Tuomas Virtanen3.1 Formalization of the Mixing Process 313.2 Microphone Recordings 323.3 Artificial Mixtures 363.4 Impulse Response Models 373.5 Summary 43Bibliography 434 Multichannel Source Activity Detection, Localization, and Tracking 47Pasi Pertilä, Alessio Brutti, Piergiorgio Svaizer, and Maurizio Omologo4.1 Basic Notions in Multichannel Spatial Audio 474.2 Multi-Microphone Source Activity Detection 524.3 Source Localization 544.4 Summary 60Bibliography 60Part II Single-Channel Separation and Enhancement 655 Spectral Masking and Filtering 67Timo Gerkmann and Emmanuel Vincent5.1 Time-Frequency Masking 675.2 Mask Estimation Given the Signal Statistics 705.3 Perceptual Improvements 815.4 Summary 82Bibliography 836 Single-Channel Speech Presence Probability Estimation and Noise Tracking 87Rainer Martin and Israel Cohen6.1 Speech Presence Probability and its Estimation 876.2 Noise Power Spectrum Tracking 936.3 Evaluation Measures 1026.4 Summary 104Bibliography 1047 Single-Channel Classification and Clustering Approaches 107FelixWeninger, Jun Du, Erik Marchi, and Tian Gao7.1 Source Separation by Computational Auditory Scene Analysis 1087.2 Source Separation by Factorial HMMs 1117.3 Separation Based Training 1137.4 Summary 125Bibliography 1258 Nonnegative Matrix Factorization 131Roland Badeau and Tuomas Virtanen8.1 NMF and Source Separation 1318.2 NMF Theory and Algorithms 1378.3 NMF Dictionary LearningMethods 1458.4 Advanced NMF Models 1488.5 Summary 156Bibliography 1569 Temporal Extensions of Nonnegative Matrix Factorization 161Cédric Févotte, Paris Smaragdis, NasserMohammadiha, and Gautham J.Mysore9.1 Convolutive NMF 1619.2 Overview of DynamicalModels 1699.3 Smooth NMF 1709.4 Nonnegative State-Space Models 1749.5 Discrete DynamicalModels 1789.6 The Use of DynamicModels in Source Separation 1829.7 Which Model to Use? 1839.8 Summary 1849.9 Standard Distributions 184Bibliography 185Part III Multichannel Separation and Enhancement 18910 Spatial Filtering 191Shmulik Markovich-Golan,Walter Kellermann, and Sharon Gannot10.1 Fundamentals of Array Processing 19210.2 Array Topologies 19710.3 Data-Independent Beamforming 19910.4 Data-Dependent Spatial Filters: Design Criteria 20210.5 Generalized Sidelobe Canceler Implementation 20910.6 Postfilters 21010.7 Summary 211Bibliography 21211 Multichannel Parameter Estimation 219Shmulik Markovich-Golan,Walter Kellermann, and Sharon Gannot11.1 Multichannel Speech Presence Probability Estimators 21911.2 Covariance Matrix Estimators Exploiting SPP 22711.3 Methods forWeakly Guided and Strongly Guided RTF Estimation 22811.4 Summary 231Bibliography 23112 Multichannel Clustering and Classification Approaches 235Michael I.Mandel, Shoko Araki, and Tomohiro Nakatani12.1 Two-Channel Clustering 23612.2 Multichannel Clustering 24412.3 Multichannel Classification 25112.4 Spatial Filtering Based on Masks 25512.5 Summary 257Bibliography 25813 Independent Component and Vector Analysis 263Hiroshi Sawada and Zbynˇek Koldovský13.1 Convolutive Mixtures and their Time-Frequency Representations 26413.2 Frequency-Domain Independent Component Analysis 26513.3 Independent Vector Analysis 27913.4 Example 28013.5 Summary 284Bibliography 28414 Gaussian Model Based Multichannel Separation 289Alexey Ozerov and Hirokazu Kameoka14.1 Gaussian Modeling 28914.2 Library of Spectral and SpatialModels 29514.3 Parameter Estimation Criteria and Algorithms 30014.4 Detailed Presentation of Some Methods 30514.5 Summary 312Acknowledgment 312Bibliography 31215 Dereverberation 317Emanuël A.P. Habets and Patrick A. Naylor15.1 Introduction to Dereverberation 31715.2 Reverberation Cancellation Approaches 31915.3 Reverberation Suppression Approaches 32915.4 Direct Estimation 33515.5 Evaluation of Dereverberation 33615.6 Summary 337Bibliography 337Part IV Application Scenarios and Perspectives 34516 Applying Source Separation to Music 347Bryan Pardo, Antoine Liutkus, Zhiyao Duan, and Gaël Richard16.1 Challenges and Opportunities 34816.2 Nonnegative Matrix Factorization in the Case of Music 34916.3 Taking Advantage of the Harmonic Structure of Music 35416.4 Nonparametric Local Models: Taking Advantage of Redundancies in Music 35816.5 Taking Advantage of Multiple Instances 36316.6 Interactive Source Separation 36716.7 Crowd-Based Evaluation 36716.8 Some Examples of Applications 36816.9 Summary 370Bibliography 37017 Application of Source Separation to Robust Speech Analysis and Recognition 377ShinjiWatanabe, Tuomas Virtanen, and Dorothea Kolossa17.1 Challenges and Opportunities 37717.2 Applications 38017.3 Robust Speech Analysis and Recognition 39017.4 Integration of Front-End and Back-End 39717.5 Use of Multimodal Information with Source Separation 40317.6 Summary 404Bibliography 40518 Binaural Speech Processing with Application to Hearing Devices 413Simon Doclo, Sharon Gannot, Daniel Marquardt, and Elior Hadad18.1 Introduction to Binaural Processing 41318.2 Binaural Hearing 41518.3 Binaural Noise Reduction Paradigms 41618.4 The Binaural Noise Reduction Problem 42018.5 Extensions for Diffuse Noise 42518.6 Extensions for Interfering Sources 43118.7 Summary 437Bibliography 43719 Perspectives 443Emmanuel Vincent, Tuomas Virtanen, and Sharon Gannot19.1 Advancing Deep Learning 44319.2 Exploiting Phase Relationships 44719.3 AdvancingMultichannel Processing 45019.4 Addressing Multiple-Device Scenarios 45319.5 TowardsWidespread Commercial Use 455Acknowledgment 457Bibliography 457Index 465