Complex Valued Nonlinear Adaptive Filters

Danilo Mandic, Department of Electrical and Electronic Engineering, Imperial College London, LondonDr Mandic is currently a Reader in Signal Processing at Imperial College, London. He is an experienced author, having written the book Recurrent Neural Networks for Prediction: Learning Algorithms, Architectures and Stability (Wiley, 2001), and more than 150 published journal and conference papers on signal and image processing. His research interests include nonlinear adaptive signal processing, multimodal signal processing and nonlinear dynamics, and he is an Associate Editor for the journals IEEE Transactions on Circuits and Systems and the International Journal of Mathematical Modelling and Algorithms. Dr Mandic is also on the IEEE Technical Committee on Machine Learning for Signal Processing, and he has produced award winning papers and products resulting from his collaboration with industry. Su-Lee Goh, Royal Dutch Shell plc, HollandDr Goh is currently working as a Reservoir Imaging Geophysicist at Shell in Holland. Her research interests include nonlinear signal processing, adaptive filters, complex-valued analysis, and imaging and forecasting. She received her PhD in nonlinear adaptive signal processing from Imperial College, London and is a member of the IEEE and the Society of Exploration Geophysicists.

• Preface xiiiAcknowledgements xvii1 The Magic of Complex Numbers 11.1 History of Complex Numbers 21.1.1 Hypercomplex Numbers 71.2 History of Mathematical Notation 81.3 Development of Complex Valued Adaptive Signal Processing 92 Why Signal Processing in the Complex Domain? 132.1 Some Examples of Complex Valued Signal Processing 132.1.1 Duality Between Signal Representations in R and c 182.2 Modelling in C is Not Only Convenient But Also Natural 192.3 Why Complex Modelling of Real Valued Processes? 202.3.1 Phase Information in Imaging 202.3.2 Modelling of Directional Processes 222.4 Exploiting the Phase Information 232.4.1 Synchronisation of Real Valued Processes 242.4.2 Adaptive Filtering by Incorporating Phase Information 252.5 Other Applications of Complex Domain Processing of Real Valued Signals 262.6 Additional Benefits of Complex Domain Processing 293 Adaptive Filtering Architectures 333.1 Linear and Nonlinear Stochastic Models 343.2 Linear and Nonlinear Adaptive Filtering Architectures 353.2.1 Feedforward Neural Networks 363.2.2 Recurrent Neural Networks 373.2.3 Neural Networks and Polynomial Filters 383.3 State Space Representation and Canonical Forms 394 Complex Nonlinear Activation Functions 434.1 Properties of Complex Functions 434.1.1 Singularities of Complex Functions 454.2 Universal Function Approximation 464.2.1 Universal Approximation in R 474.3 Nonlinear Activation Functions for Complex Neural Networks 484.3.1 Split-complex Approach 494.3.2 Fully Complex Nonlinear Activation Functions 514.4 Generalised Splitting Activation Functions (GSAF) 534.4.1 The Clifford Neuron 534.5 Summary: Choice of the Complex Activation Function 545 Elements of CR Calculus 555.1 Continuous Complex Functions 565.2 The Cauchy–Riemann Equations 565.3 Generalised Derivatives of Functions of Complex Variable 575.3.1 CR Calculus 595.3.2 Link between R- and C-derivatives 605.4 CR-derivatives of Cost Functions 625.4.1 The Complex Gradient 625.4.2 The Complex Hessian 645.4.3 The Complex Jacobian and Complex Differential 645.4.4 Gradient of a Cost Function 656 Complex Valued Adaptive Filters 696.1 Adaptive Filtering Configurations 706.2 The Complex Least Mean Square Algorithm 736.2.1 Convergence of the CLMS Algorithm 756.3 Nonlinear Feedforward Complex Adaptive Filters 806.3.1 Fully Complex Nonlinear Adaptive Filters 806.3.2 Derivation of CNGD using CR calculus 826.3.3 Split-complex Approach 836.3.4 Dual Univariate Adaptive Filtering Approach (DUAF) 846.4 Normalisation of Learning Algorithms 856.5 Performance of Feedforward Nonlinear Adaptive Filters 876.6 Summary: Choice of a Nonlinear Adaptive Filter 897 Adaptive Filters with Feedback 917.1 Training of IIR Adaptive Filters 927.1.1 Coefficient Update for Linear Adaptive IIR Filters 937.1.2 Training of IIR filters with Reduced Computational Complexity 967.2 Nonlinear Adaptive IIR Filters: Recurrent Perceptron 977.3 Training of Recurrent Neural Networks 997.3.1 Other Learning Algorithms and Computational Complexity 1027.4 Simulation Examples 1028 Filters with an Adaptive Stepsize 1078.1 Benveniste Type Variable Stepsize Algorithms 1088.2 Complex Valued GNGD Algorithms 1108.2.1 Complex GNGD for Nonlinear Filters (CFANNGD) 1128.3 Simulation Examples 1139 Filters with an Adaptive Amplitude of Nonlinearity 1199.1 Dynamical Range Reduction 1199.2 FIR Adaptive Filters with an Adaptive Nonlinearity 1219.3 Recurrent Neural Networks with Trainable Amplitude of Activation Functions 1229.4 Simulation Results 12410 Data-reusing Algorithms for Complex Valued Adaptive Filters 12910.1 The Data-reusing Complex Valued Least Mean Square (DRCLMS) Algorithm 12910.2 Data-reusing Complex Nonlinear Adaptive Filters 13110.2.1 Convergence Analysis 13210.3 Data-reusing Algorithms for Complex RNNs 13411 Complex Mappings and Möbius Transformations 13711.1 Matrix Representation of a Complex Number 13711.2 The Möbius Transformation 14011.3 Activation Functions and Möbius Transformations 14211.4 All-pass Systems as Möbius Transformations 14611.5 Fractional Delay Filters 14712 Augmented Complex Statistics 15112.1 Complex Random Variables (CRV) 15212.1.1 Complex Circularity 15312.1.2 The Multivariate Complex Normal Distribution 15412.1.3 Moments of Complex Random Variables (CRV) 15712.2 Complex Circular Random Variables 15812.3 Complex Signals 15912.3.1 Wide Sense Stationarity, Multicorrelations, and Multispectra 16012.3.2 Strict Circularity and Higher-order Statistics 16112.4 Second-order Characterisation of Complex Signals 16112.4.1 Augmented Statistics of Complex Signals 16112.4.2 Second-order Complex Circularity 16413 Widely Linear Estimation and Augmented CLMS (ACLMS) 16913.1 Minimum Mean Square Error (MMSE) Estimation in c 16913.1.1 Widely Linear Modelling in c 17113.2 Complex White Noise 17213.3 Autoregressive Modelling in c 17313.3.1 Widely Linear Autoregressive Modelling in c 17413.3.2 Quantifying Benefits of Widely Linear Estimation 17413.4 The Augmented Complex LMS (ACLMS) Algorithm 17513.5 Adaptive Prediction Based on ACLMS 17813.5.1 Wind Forecasting Using Augmented Statistics 18014 Duality Between Complex Valued and Real Valued Filters 18314.1 A Dual Channel Real Valued Adaptive Filter 18414.2 Duality Between Real and Complex Valued Filters 18614.2.1 Operation of Standard Complex Adaptive Filters 18614.2.2 Operation of Widely Linear Complex Filters 18714.3 Simulations 18815 Widely Linear Filters with Feedback 19115.1 The Widely Linear ARMA (WL-ARMA) Model 19215.2 Widely Linear Adaptive Filters with Feedback 19215.2.1 Widely Linear Adaptive IIR Filters 19515.2.2 Augmented Recurrent Perceptron Learning Rule 19615.3 The Augmented Complex Valued RTRL (ACRTRL) Algorithm 19715.4 The Augmented Kalman Filter Algorithm for RNNs 19815.4.1 EKF Based Training of Complex RNNs 20015.5 Augmented Complex Unscented Kalman Filter (ACUKF) 20015.5.1 State Space Equations for the Complex Unscented Kalman Filter 20115.5.2 ACUKF Based Training of Complex RNNs 20215.6 Simulation Examples 20316 Collaborative Adaptive Filtering 20716.1 Parametric Signal Modality Characterisation 20716.2 Standard Hybrid Filtering in R 20916.3 Tracking the Linear/Nonlinear Nature of Complex Valued Signals 21016.3.1 Signal Modality characterisation in c 21116.4 Split vs Fully Complex Signal Natures 21416.5 Online Assessment of the Nature of Wind Signal 21616.5.1 Effects of Averaging on Signal Nonlinearity 21616.6 Collaborative Filters for General Complex Signals 21716.6.1 Hybrid Filters for Noncircular Signals 21816.6.2 Online Test for Complex Circularity 22017 Adaptive Filtering Based on EMD 22117.1 The Empirical Mode Decomposition Algorithm 22217.1.1 Empirical Mode Decomposition as a Fixed Point Iteration 22317.1.2 Applications of Real Valued EMD 22417.1.3 Uniqueness of the Decomposition 22517.2 Complex Extensions of Empirical Mode Decomposition 22617.2.1 Complex Empirical Mode Decomposition 22717.2.2 Rotation Invariant Empirical Mode Decomposition (RIEMD) 22817.2.3 Bivariate Empirical Mode Decomposition (BEMD) 22817.3 Addressing the Problem of Uniqueness 23017.4 Applications of Complex Extensions of EMD 23018 Validation of Complex Representations – Is This Worthwhile? 23318.1 Signal Modality Characterisation in R 23418.1.1 Surrogate Data Methods 23518.1.2 Test Statistics: The DVV Method 23718.2 Testing for the Validity of Complex Representation 23918.2.1 Complex Delay Vector Variance Method (CDVV) 24018.3 Quantifying Benefits of Complex Valued Representation 24318.3.1 Pros and Cons of the Complex DVV Method 244Appendix A: Some Distinctive Properties of Calculus in C 245Appendix B: Liouville’s Theorem 251Appendix C: Hypercomplex and Clifford Algebras 253C. 1 Definitions of Algebraic Notions of Group, Ring and Field 253C. 2 Definition of a Vector Space 254C. 3 Higher Dimension Algebras 254C. 4 The Algebra of Quaternions 255C. 5 Clifford Algebras 256Appendix D: Real Valued Activation Functions 257D. 1 Logistic Sigmoid Activation Function 257D. 2 Hyperbolic Tangent Activation Function 258Appendix E: Elementary Transcendental Functions (ETF) 259Appendix F: The O Notation and Standard Vector and Matrix Differentiation 263F. 1 The O Notation 263F. 2 Standard Vector and Matrix Differentiation 263Appendix G: Notions From Learning Theory 265G. 1 Types of Learning 266G. 2 The Bias–Variance Dilemma 266G. 3 Recursive and Iterative Gradient Estimation Techniques 267G. 4 Transformation of Input Data 267Appendix H: Notions from Approximation Theory 269Appendix I: Terminology Used in the Field of Neural Networks 273Appendix J: Complex Valued Pipelined Recurrent Neural Network (CPRNN) 275J.1 The Complex RTRL Algorithm (CRTRL) for CPRNN 275J.1.1 Linear Subsection Within the PRNN 277Appendix K: Gradient Adaptive Step Size (GASS) Algorithms in R 279K. 1 Gradient Adaptive Stepsize Algorithms Based on ∂J/∂μ 280K. 2 Variable Stepsize Algorithms Based on ∂J/∂ε 281Appendix L: Derivation of Partial Derivatives from Chapter 8 283L. 1 Derivation of ∂e(k)/∂w n (k) 283L. 2 Derivation of ∂e ∗ (k)/∂ε(k − 1) 284L. 3 Derivation of ∂w(k)/∂ε(k − 1) 286Appendix M: A Posteriori Learning 287M.1 A Posteriori Strategies in Adaptive Learning 288Appendix N: Notions from Stability Theory 291Appendix O: Linear Relaxation 293O. 1 Vector and Matrix Norms 293O. 2 Relaxation in Linear Systems 294O.2.1 Convergence in the Norm or State Space? 297Appendix P: Contraction Mappings, Fixed Point Iteration and Fractals 299P. 1 Historical Perspective 303P. 2 More on Convergence: Modified Contraction Mapping 305P. 3 Fractals and Mandelbrot Set 308References 309Index 321