Classification, Parameter Estimation and State Estimation

Professor  Bangjun Lei, Dr. Guangzhu Xu,  and Dr. Ming Feng are with The Institute of Intelligent Vision and Image Information, China Three Gorges University, China.Professor Yaobin Zou is an associate professor at China Three Gorges University.Dr. Ferdinand van der Heijden, Ph.D., is on the faculty of theDepartment of Signals and Systems, University of Twente, Netherlands. Professor Dick de Ridder is Professor at the Bioinformatics lab at Wageningen University, Netherlands. Professor David M. J. Tax, is a researcher with the Pattern Recognition laboratory, Delft University of Technology.

• Preface xiAbout the Companion Website xv1 Introduction 11.1 The Scope of the Book 21.1.1 Classification 31.1.2 Parameter Estimation 41.1.3 State Estimation 51.1.4 Relations between the Subjects 71.2 Engineering 101.3 The Organization of the Book 121.4 Changes from First Edition 141.5 References 152 PRTools Introduction 172.1 Motivation 172.2 Essential Concepts 182.3 PRTools Organization Structure and Implementation 222.4 Some Details about PRTools 262.4.1 Datasets 262.4.2 Datafiles 302.4.3 Datafiles Help Information 312.4.4 Classifiers and Mappings 342.4.5 Mappings Help Information 362.4.6 How to Write Your Own Mapping 382.5 Selected Bibliography 423 Detection and Classification 433.1 Bayesian Classification 463.1.1 Uniform Cost Function and Minimum Error Rate 533.1.2 Normal Distributed Measurements; Linear and Quadratic Classifiers 563.2 Rejection 623.2.1 Minimum Error Rate Classification with Reject Option 633.3 Detection: The Two-Class Case 663.4 Selected Bibliography 74Exercises 744 Parameter Estimation 774.1 Bayesian Estimation 794.1.1 MMSE Estimation 864.1.2 MAP Estimation 874.1.3 The Gaussian Case with Linear Sensors 884.1.4 Maximum Likelihood Estimation 894.1.5 Unbiased Linear MMSE Estimation 914.2 Performance Estimators 944.2.1 Bias and Covariance 954.2.2 The Error Covariance of the Unbiased Linear MMSE Estimator 994.3 Data Fitting 1004.3.1 Least Squares Fitting 1014.3.2 Fitting Using a Robust Error Norm 1044.3.3 Regression 1074.4 Overview of the Family of Estimators 1104.5 Selected Bibliography 111Exercises 1125 State Estimation 1155.1 A General Framework for Online Estimation 1175.1.1 Models 1175.1.2 Optimal Online Estimation 1235.2 Infinite Discrete-Time State Variables 1255.2.1 Optimal Online Estimation in Linear-Gaussian Systems 1255.2.2 Suboptimal Solutions for Non-linear Systems 1335.3 Finite Discrete-Time State Variables 1475.3.1 Hidden Markov Models 1485.3.2 Online State Estimation 1525.3.3 Offline State Estimation 1565.4 Mixed States and the Particle Filter 1635.4.1 Importance Sampling 1645.4.2 Resampling by Selection 1665.4.3 The Condensation Algorithm 1675.5 Genetic State Estimation 1705.5.1 The Genetic Algorithm 1705.5.2 Genetic State Estimation 1765.5.3 Computational Issues 1775.6 State Estimation in Practice 1835.6.1 System Identification 1855.6.2 Observability, Controllability and Stability 1885.6.3 Computational Issues 1935.6.4 Consistency Checks 1965.7 Selected Bibliography 201Exercises 2046 Supervised Learning 2076.1 Training Sets 2086.2 Parametric Learning 2106.2.1 Gaussian Distribution, Mean Unknown 2116.2.2 Gaussian Distribution, Covariance Matrix Unknown 2126.2.3 Gaussian Distribution, Mean and Covariance Matrix Both Unknown 2136.2.4 Estimation of the Prior Probabilities 2156.2.5 Binary Measurements 2166.3 Non-parametric Learning 2176.3.1 Parzen Estimation and Histogramming 2186.3.2 Nearest Neighbour Classification 2236.3.3 Linear Discriminant Functions 2306.3.4 The Support Vector Classifier 2376.3.5 The Feedforward Neural Network 2426.4 Adaptive Boosting – Adaboost 2456.5 Convolutional Neural Networks (CNNs) 2496.5.1 Convolutional Neural Network Structure 2496.5.2 Computation and Training of CNNs 2516.6 Empirical Evaluation 2526.7 Selected Bibliography 257Exercises 2577 Feature Extraction and Selection 2597.1 Criteria for Selection and Extraction 2617.1.1 Interclass/Intraclass Distance 2627.1.2 Chernoff–Bhattacharyya Distance 2677.1.3 Other Criteria 2707.2 Feature Selection 2727.2.1 Branch-and-Bound 2737.2.2 Suboptimal Search 2757.2.3 Several New Methods of Feature Selection 2787.2.4 Implementation Issues 2877.3 Linear Feature Extraction 2887.3.1 Feature Extraction Based on the Bhattacharyya Distance with Gaussian Distributions 2917.3.2 Feature Extraction Based on Inter/Intra Class Distance 2967.4 References 300Exercises 3008 Unsupervised Learning 3038.1 Feature Reduction 3048.1.1 Principal Component Analysis 3048.1.2 Multidimensional Scaling 3098.1.3 Kernel Principal Component Analysis 3158.2 Clustering 3208.2.1 Hierarchical Clustering 3238.2.2 K-Means Clustering 3278.2.3 Mixture of Gaussians 3298.2.4 Mixture of probabilistic PCA 3358.2.5 Self-Organizing Maps 3368.2.6 Generative Topographic Mapping 3428.3 References 345Exercises 3469 Worked Out Examples 3499.1 Example on Image Classification with PRTools 3499.1.1 Example on Image Classification 3499.1.2 Example on Face Classification 3549.1.3 Example on Silhouette Classification 3579.2 Boston Housing Classification Problem 3619.2.1 Dataset Description 3619.2.2 Simple Classification Methods 3639.2.3 Feature Extraction 3659.2.4 Feature Selection 3679.2.5 Complex Classifiers 3689.2.6 Conclusions 3719.3 Time-of-Flight Estimation of an Acoustic Tone Burst 3729.3.1 Models of the Observed Waveform 3749.3.2 Heuristic Methods for Determining the ToF 3769.3.3 Curve Fitting 3779.3.4 Matched Filtering 3799.3.5 ml Estimation Using Covariance Models for the Reflections 3809.3.6 Optimization and Evaluation 3859.4 Online Level Estimation in a Hydraulic System 3929.4.1 Linearized Kalman Filtering 3949.4.2 Extended Kalman Filtering 3979.4.3 Particle Filtering 3989.4.4 Discussion 4039.5 References 406Appendix A: Topics Selected from Functional Analysis 407Appendix B: Topics Selected from Linear Algebra and Matrix Theory 421Appendix C: Probability Theory 437Appendix D: Discrete-Time Dynamic Systems 453Index 459