Digital Signal Processing with Kernel Methods

JOSÉ LUIS ROJO-ÁLVAREZ, PhD, is a Professor in the Department of Signal Theory and Communications at the University Rey Juan Carlos, Fuenlabrada (Madrid) and Center for Computational Simulation, Universidad Politécnica de Madrid, Spain. MANEL MARTÍNEZ-RAMÓN, PhD, is a Professor in the Department of Electrical and Computer Engineering at the University of New Mexico, Albuquerque, USA. JORDI MUÑOZ-MARÍ, PhD, is an Associate Professor in the Department of Electronics Engineering at the Universitat de València, Spain. GUSTAU CAMPS-VALLS, PhD, is an Associate Professor in the Department of Electronics Engineering at the Universitat de València, Spain.

• About the Authors xiiiPreface xviiAcknowledgements xxiList of Abbreviations xxiiiPart I Fundamentals and Basic Elements 11 From Signal Processing to Machine Learning 31.1 A New Science is Born: Signal Processing 31.1.1 Signal Processing Before Being Coined 31.1.2 1948: Birth of the Information Age 41.1.3 1950s: Audio Engineering Catalyzes Signal Processing 41.2 From Analog to Digital Signal Processing 51.2.1 1960s: Digital Signal Processing Begins 51.2.2 1970s: Digital Signal Processing Becomes Popular 61.2.3 1980s: Silicon Meets Digital Signal Processing 61.3 Digital Signal Processing Meets Machine Learning 71.3.1 1990s: New Application Areas 71.3.2 1990s: Neural Networks, Fuzzy Logic, and Genetic Optimization 71.4 Recent Machine Learning in Digital Signal Processing 81.4.1 Traditional Signal Assumptions Are No Longer Valid 81.4.2 Encoding Prior Knowledge 81.4.3 Learning and Knowledge from Data 91.4.4 From Machine Learning to Digital Signal Processing 91.4.5 From Digital Signal Processing to Machine Learning 102 Introduction to Digital Signal Processing 13 2.1 Outline of the Signal Processing Field 132.1.1 Fundamentals on Signals and Systems 142.1.2 Digital Filtering 212.1.3 Spectral Analysis 242.1.4 Deconvolution 282.1.5 Interpolation 302.1.6 System Identification 312.1.7 Blind Source Separation 362.2.3 Sparsity, Compressed Sensing, and Dictionary Learning 442.3 Multidimensional Signals and Systems 482.3.1 Multidimensional Signals 492.3.2 Multidimensional Systems 512.4 Spectral Analysis on Manifolds 522.4.1 Theoretical Fundamentals 522.4.2 Laplacian Matrices 542.5 Tutorials and Application Examples 572.5.1 Real and Complex Signal Processing and Representations 572.5.2 Convolution, Fourier Transform, and Spectrum 632.5.3 Continuous-Time Signals and Systems 672.5.4 Filtering Cardiac Signals 702.5.5 Nonparametric Spectrum Estimation 742.5.6 Parametric Spectrum Estimation 772.5.7 Source Separation 812.5.8 Time–Frequency Representations and Wavelets 842.5.9 Examples for Spectral Analysis on Manifolds 872.6 Questions and Problems 943 Signal Processing Models 973.1 Introduction 973.2 Vector Spaces, Basis, and Signal Models 983.2.1 Basic Operations for Vectors 983.2.2 Vector Spaces 1003.2.3 Hilbert Spaces 1013.2.4 Signal Models 1023.2.5 Complex Signal Models 1043.2.6 Standard Noise Models in Digital Signal Processing 1053.2.7 The Role of the Cost Function 1073.2.8 The Role of the Regularizer 1093.3 Digital Signal Processing Models 1113.3.1 Sinusoidal Signal Models 1123.3.2 System Identification Signal Models 1133.3.3 Sinc Interpolation Models 1163.3.4 Sparse Deconvolution 1203.3.5 Array Processing 1213.4 Tutorials and Application Examples 1223.4.1 Examples of Noise Models 1233.4.2 Autoregressive Exogenous System Identification Models 1323.4.3 Nonlinear System Identification Using Volterra Models 1383.4.4 Sinusoidal Signal Models 1403.4.5 Sinc-based Interpolation 1443.4.6 Sparse Deconvolution 1523.4.7 Array Processing 1573.5 Questions and Problems 1603.A MATLABsimpleInterp Toolbox Structure 1614 Kernel Functions and Reproducing Kernel Hilbert Spaces 1654.1 Introduction 1654.2 Kernel Functions and Mappings 1694.2.1 Measuring Similarity with Kernels 1694.2.2 Positive-Definite Kernels 1694.2.3 Reproducing Kernel in Hilbert Space and Reproducing Property 1704.2.4 Mercer’s Theorem 1734.3 Kernel Properties 1744.3.1 Tikhonov’s Regularization 1754.3.2 Representer Theorem and Regularization Properties 1764.3.3 Basic Operations with Kernels 1784.4 Constructing Kernel Functions 1794.4.1 Standard Kernels 1794.4.2 Properties of Kernels 1804.4.3 Engineering Signal Processing Kernels 1814.5 Complex Reproducing Kernel in Hilbert Spaces 1844.6 Support Vector Machine Elements for Regression and Estimation 1864.6.1 Support Vector Regression Signal Model and Cost Function 1864.6.2 Minimizing Functional 1874.7 Tutorials and Application Examples 1914.7.1 Kernel Calculations and Kernel Matrices 1914.7.2 Basic Operations with Kernels 1944.7.3 Constructing Kernels 1974.7.4 Complex Kernels 1994.7.5 Application Example for Support Vector Regression Elements 2024.8 Concluding Remarks 2054.9 Questions and Problems 205Part II Function Approximation and Adaptive Filtering 209 5 A Support Vector Machine Signal Estimation Framework 2115.1 Introduction 2115.2 A Framework for Support Vector Machine Signal Estimation 2135.3 Primal Signal Models for Support Vector Machine Signal Processing 2165.3.1 Nonparametric Spectrum and System Identification 2185.3.2 Orthogonal Frequency Division Multiplexing Digital Communications 2205.3.3 Convolutional Signal Models 2225.3.4 Array Processing 2255.4 Tutorials and Application Examples 2275.4.1 Nonparametric Spectral Analysis with Primal Signal Models 2275.4.2 System Identification with Primal Signal Model ;;-filter 2285.4.3 Parametric Spectral Density Estimation with Primal Signal Models 2305.4.4 Temporal Reference Array Processing with Primal Signal Models 2315.4.5 Sinc Interpolation with Primal Signal Models 2336 Reproducing Kernel Hilbert Space Models for Signal Processing 2416.1 Introduction 2416.2 Reproducing Kernel Hilbert Space Signal Models 2426.2.1 Kernel Autoregressive Exogenous Identification 2446.2.2  Kernel Finite Impulse Response and the ;;-Filter 2476.2.3 Kernel Array Processing with Spatial Reference 2486.2.4 Kernel Semiparametric Regression 2496.3 Tutorials and Application Examples 2586.3.1 Nonlinear System Identification with Support Vector Machine–Autoregressive and Moving Average 2586.3.2 Nonlinear System Identification with the ;;-filter 2606.3.3 Electric Network Modeling with Semiparametric Regression 2646.3.4 Promotional Data 2726.3.5 Spatial and Temporal Antenna Array Kernel Processing 2756.4 Questions and Problems 2797 Dual Signal Models for Signal Processing 281 7.1 Introduction 2817.2 Dual Signal Model Elements 2817.3 Dual Signal Model Instantiations 2837.3.1 Dual Signal Model for Nonuniform Signal Interpolation 2837.3.2 Dual Signal Model for Sparse Signal Deconvolution 2847.3.3 Spectrally Adapted Mercer Kernels 2857.4 Tutorials and Application Examples 2897.4.1 Nonuniform Interpolation with the Dual Signal Model 2907.4.2 Sparse Deconvolution with the Dual Signal Model 2927.4.3 Doppler Ultrasound Processing for Fault Detection 2947.4.4 Spectrally Adapted Mercer Kernels 2967.4.5 Interpolation of Heart Rate Variability Signals 3047.4.6 Denoising in Cardiac Motion-Mode Doppler Ultrasound Images 309?m7.4.7 Indoor Location from Mobile Devices Measurements 3167.4.8 Electroanatomical Maps in Cardiac Navigation Systems 3227.5 Questions and Problems 3318 Advances in Kernel Regression and Function Approximation 3338.1 Introduction 3338.2 Kernel-Based Regression Methods 3338.2.1 Advances in Support Vector Regression 3348.2.2 Multi-output Support Vector Regression 3388.2.3 Kernel Ridge Regression 3398.2.4 Kernel Signal-To-Noise Regression 3418.2.5 Semisupervised Support Vector Regression 3438.2.6 Model Selection in Kernel Regression Methods 3458.4.1 Comparing Support Vector Regression, Relevance Vector Machines, and Gaussian Process Regression 3608.4.2 Profile-Dependent Support Vector Regression 3628.4.3 Multi-output Support Vector Regression 3648.4.4 Kernel Signal-to-Noise Ratio Regression 3668.4.5 Semisupervised Support Vector Regression 3688.4.6 Bayesian Nonparametric Model 3698.4.7 Gaussian Process Regression 3708.4.8 Relevance Vector Machines 3798.5 Concluding Remarks 3828.6 Questions and Problems 3839 Adaptive Kernel Learning for Signal Processing 3879.1 Introduction 3879.2 Linear Adaptive Filtering 3879.2.1 Least Mean Squares Algorithm 3889.2.2 Recursive Least-Squares Algorithm 3899.3 Kernel Adaptive Filtering 3929.4 Kernel Least Mean Squares 3929.4.1 Derivation of Kernel Least Mean Squares 3939.4.2 Implementation Challenges and Dual Formulation 3949.5.3 Prediction of the Mackey–Glass Time Series with Kernel Recursive Least Squares 4019.5.4 Beyond the Stationary Model 4029.5.5 Example on Nonlinear Channel Identification and Reconvergence 4059.6 Explicit Recursivity for Adaptive Kernel Models 4069.6.1 Recursivity in Hilbert Spaces 4069.6.2 Recursive Filters in Reproducing Kernel Hilbert Spaces 4089.7 Online Sparsification with Kernels 4119.7.1 Sparsity by Construction 4119.7.2 Sparsity by Pruning 4139.8 Probabilistic Approaches to Kernel Adaptive Filtering 4149.8.1 Gaussian Processes and Kernel Ridge Regression 4159.8.2 Online Recursive Solution for Gaussian Processes Regression 4169.8.3 Kernel Recursive Least Squares Tracker 4179.8.4 Probabilistic Kernel Least Mean Squares 4189.9 Further Reading 4189.9.1 Selection of Kernel Parameters 4189.9.2 Multi-Kernel Adaptive Filtering 4199.9.3 Recursive Filtering in Kernel Hilbert Spaces 4199.10 Tutorials and Application Examples 4199.10.1 Kernel Adaptive Filtering Toolbox 4209.10.2 Prediction of a Respiratory Motion Time Series 4219.10.3 Online Regression on the KIN?h?eK Dataset 4239.10.4 The Mackey–Glass Time Series 4259.10.5 Explicit Recursivity on Reproducing Kernel in Hilbert Space and Electroencephalogram Prediction 4279.10.6 Adaptive Antenna Array Processing 4289.11 Questions and Problems 430Part III Classification, Detection, and Feature Extraction 43310 Support Vector Machine and Kernel Classification Algorithms 43510.1 Introduction 43510.2 Support Vector Machine and Kernel Classifiers 43510.2.1 Support Vector Machines 43510.2.2 Multiclass and Multilabel Support Vector Machines 44110.2.3 Least-Squares Support Vector Machine 44710.2.4 Kernel Fisher’s Discriminant Analysis 44810.3 Advances in Kernel-Based Classification 45210.3.1 Large Margin Filtering 45210.3.2 Semisupervised Learning 45410.3.3 Multiple Kernel Learning 46010.3.4 Structured-Output Learning 46210.3.5 Active Learning 46810.4 Large-Scale Support Vector Machines 47710.4.1 Large-Scale Support Vector Machine Implementations 47710.4.2 Random Fourier Features 47810.4.3 Parallel Support Vector Machine 48010.4.4 Outlook 48310.5 Tutorials and Application Examples 48510.5.1 Examples of Support Vector Machine Classification 48510.5.2 Example of Least-Squares Support Vector Machine 49210.5.3 Kernel-Filtering Support Vector Machine for Brain–Computer Interface Signal Classification 49310.5.4 Example of Laplacian Support Vector Machine 49410.5.5 Example of Graph-Based Label Propagation 49810.5.6 Examples of Multiple Kernel Learning 49810.6 Concluding Remarks 50110.7 Questions and Problems 50211 Clustering and Anomaly Detection with Kernels 50311.1 Introduction 50311.2 Kernel Clustering 50611.2.1 Kernelization of the Metric 50611.2.2 Clustering in Feature Spaces 50811.3 Domain Description Via Support Vectors 51411.3.1 Support Vector Domain Description 51411.3.2 One-Class Support Vector Machine 51511.3.3 Relationship Between Support Vector Domain Description and Density Estimation 51611.3.4 Semisupervised One-Class Classification 51711.4 Kernel Matched Subspace Detectors 51811.4.1 Kernel Orthogonal Subspace Projection 51811.4.2 Kernel Spectral Angle Mapper 52011.5 Kernel Anomaly Change Detection 52211.5.1 Linear Anomaly Change Detection Algorithms 52211.5.2 Kernel Anomaly Change Detection Algorithms 52311.6 Hypothesis Testing with Kernels 52511.6.1 Distribution Embeddings 52611.6.3 Maximum Mean Discrepancy 52711.6.3 One-Class Support Measure Machine 52811.7 Tutorials and Application Examples 52911.7.1 Example on Kernelization of the Metric 52911.7.2 Example on Kernel k-Means 53011.7.3 Domain Description Examples 53111.7.4 Kernel Spectral Angle Mapper and Kernel Orthogonal Subspace Projection Examples 53411.7.5 Example of Kernel Anomaly Change Detection Algorithms 53611.7.6 Example on Distribution Embeddings and Maximum Mean Discrepancy 54011.8 Concluding Remarks 54111.9 Questions and Problems 54212 Kernel Feature Extraction in Signal Processing 543 12.1 Introduction 54312.2 Multivariate Analysis in Reproducing Kernel Hilbert Spaces 54512.2.1 Problem Statement and Notation 54512.2.2 Linear Multivariate Analysis 54612.2.3 Kernel Multivariate Analysis 54912.2.4 Multivariate Analysis Experiments 55112.3 Feature Extraction with Kernel Dependence Estimates 55512.3.1 Feature Extraction Using Hilbert–Schmidt Independence Criterion 55612.3.2 Blind Source Separation Using Kernels 56312.4 Extensions for Large-Scale and Semisupervised Problems 57012.4.2 Efficiency with the Incomplete Cholesky Decomposition 57012.4.3 Efficiency with Random Fourier Features 57012.4.3 Sparse Kernel Feature Extraction 57112.4.4 Semisupervised Kernel Feature Extraction 57312.5 Domain Adaptation with Kernels 57512.5.1 Kernel Mean Matching 57812.5.2 Transfer Component Analysis 57912.5.3 Kernel Manifold Alignment 58112.5.4 Relations between Domain Adaptation Methods 58512.5.5 Experimental Comparison between Domain Adaptation Methods12.6 Concluding Remarks 58712.7 Questions and Problems 588References 589Index 631