Signal Processing and Machine Learning with Applications

Professor Michael M. Richter taught at the University of Texas at Austin and at RWTH Aachen, in addition to numerous visiting professorships.  He is one of the founding scientific director of the DFKI (German Research Center for Artificial Intelligence). He taught, researched, and published extensively in the areas of mathematical logic and artificial intelligence. Professor Richter was one of the pioneers of case-based reasoning: he founded the leading European event on the subject, he led many of the key academic research projects, and demonstrated the real-world viability of the approach with successful commercial products. Michael Richter passed away during the final publishing phase of this book. Dr. Sheuli Paul is a scientist in Defence Research and Development Canada, engaged in  applied research in the areas of signal processing, machine learning, artificial intelligence and  human-robot interaction.   Trying to solve complex problems in interdisciplinary areas is her passion.Dr. Veton  Këpuska  is an  inventor of  Wake-Up-Word Speech Recognition, a method of communication with machines for which he was granted two patents. He joined Florida Institute of Technology (FIT) in 2003 and engaged in numerous research activities in speech and image processing, digital processes, and machine learning. Dr. Këpuska won the First Annual Digital Signal Processing Design competition by applying his Wake-up-Word on embedded Analog Devices Platform. Dr. Këpuska won numerous awards including “the Kerry Bruce Clark” award for teaching excellence and received numerous best paper awards.Prof. Marius Silaghi has taught, researched, and published in the areas of artificial intelligence and networking. Professor Silaghi is involved in human-machine interaction research and proposed techniques for motion capture, speech recognition, and robotics. He founded the conference on Distributed Constraint Optimization and gave multiple tutorials on the topic at the main artificial intelligence conferences. He received numerous best paper awards.

• Part I Realms of Signal Processing.- 1 Digital Signal Representation.- 1.1 Introduction.- 1.2 Numbers.- 1.2.1 Numbers and Numerals.- 1.2.2 Types of Numbers.- 1.2.3 Positional Number Systems.- 1.3 Sampling and Reconstruction of Signals.- 1.3.1 Scalar Quantization.- 1.3.2 Quantization Noise.- 1.3.3 Signal-To-Noise Ratio.- 1.3.4 Transmission Rate.- 1.3.5 Nonuniform Quantizer.- 1.3.6 Companding.- 1.4 Data Representations.- 1.4.1 Fixed-Point Number Representations.- 1.4.2 Sign-Magnitude Format.- 1.4.3 One’s-Complement Format.- 1.4.4 Two’s-Complement Format.- 1.5 Fix-Point DSP’s.- 1.6 Fixed-Point Representations Based on Radix-Point.- 1.7 Dynamic Range.- 1.8 Precision.- 1.9 Background Information.- 1.10 Exercises.- 2 Signal Processing Background.- 2.1 Basic Concepts.- 2.2 Signals and Information.- 2.3 Signal Processing.- ix.- x Contents.- 2.4 Discrete Signal Representations.- 2.5 Delta and Impulse Function.- 2.6 Parseval’s Theorem.- 2.7 Gibbs Phenomenon.- 2.8 Wold Decomposition.- 2.9 State Space Signal Processing.- 2.10 Common Measurements.- 2.10.1 Convolution.- 2.10.2 Correlation.- 2.10.3 Auto Covariance.- 2.10.4 Coherence.- 2.10.5 Power Spectral Density (PSD).- 2.10.6 Estimation and Detection.- 2.10.7 Central Limit Theorem.- 2.10.8 Signal Information Processing Types.- 2.10.9 Machine Learning.- 2.10.10Exercises.- 3 Fundamentals of Signal Transformations.- 3.1 Transformation Methods.- 3.1.1 Laplace Transform.- 3.1.2 Z-Transform .- 3.1.3 Fourier Series.- 3.1.4 Fourier Transform.- 3.1.5 Discrete Fourier Transform and Fast Fourier Transform .- 3.1.6 Zero Padding.- 3.1.7 Overlap-Add and Overlap-Save Convolution.- Algorithms.- 3.1.8 Short Time Fourier Transform (STFT).- 3.1.9 Wavelet Transform.- 3.1.10 Windowing Signal and the DCT Transforms.- 3.2 Analysis and Comparison of Transformations.- 3.3 Background Information.- 3.4 Exercises.- 3.5 References.- 4 Digital Filters.- 4.1 Introduction.- 4.1.1 FIR and IIR Filters.- 4.1.2 Bilinear Transform.-4.2 Windowing for Filtering.- 4.3 Allpass Filters.- 4.4 Lattice Filters.- 4.5 All-Zero Lattice Filter.- 4.6 Lattice Ladder Filters.- Contents xi.- 4.7 Comb Filter.- 4.8 Notch Filter.- 4.9 Background Information.- 4.10 Exercises.- 5 Estimation and Detection.- 5.1 Introduction.- 5.2 Hypothesis Testing.- 5.2.1 Bayesian Hypothesis Testing.- 5.2.2 MAP Hypothesis Testing.- 5.3 Maximum Likelihood (ML) Hypothesis Testing.- 5.4 Standard Analysis Techniques.- 5.4.1 Best Linear Unbiased Estimator (BLUE).- 5.4.2 Maximum Likelihood Estimator (MLE).- 5.4.3 Least Squares Estimator (LSE).- 5.4.4 Linear Minimum Mean Square Error Estimator.- (LMMSE).- 5.5 Exercises.- 6 Adaptive Signal Processing.- 6.1 Introduction.- 6.2 Parametric Signal Modeling.- 6.2.1 Parametric Estimation.- 6.3 Wiener Filtering.- 6.4 Kalman Filter.- 6.4.1 Smoothing.- 6.5 Particle Filter.- 6.6 Fundamentals of Monte Carl.- 6.6.1 Importance Sampling (IS).- 6.7 Non-Parametric Signal Modeling.- 6.8 Non-Parametric Estimation.- 6.8.1 Correlogram.- 6.8.2 Periodogram.- 6.9 Filter Bank Method.- 6.10 Quadrature Mirror Filter Bank (QMF).- 6.11 Background Information.- 6.12 Exercises.- 7 Spectral Analysis.- 7.1 Introduction.- 7.2 Adaptive Spectral Analysis.- 7.3 Multivariate Signal Processing.- 7.3.1 Sub-band Coding and Subspace Analysis.- 7.4 Wavelet Analysis.- 7.5 Adaptive Beam Forming.- xii Contents.- 7.6 Independent Component Analysis (ICA).- 7.7 Principal Component Analysis (PCA).- 7.8 Best Basis Algorithms.- 7.9 Background Information.- 7.10 Exercises.- Part II Machine Learning and Recognition.- 8 General Learning.- 8.1 Introduction to Learning.- 8.2 The Learning Phases.- 8.2.1 Search and Utility.- 8.3 Search.- 8.3.1 General Search Model.- 8.3.2 Preference relations.- 8.3.3 Different learning methods.- 8.3.4 Similarities .- 8.3.5 Learning to Recognize.- 8.3.6 Learning again.- 8.4 Background Information.- 8.5 Exercises.- 9 Signal Processes, Learning, and Recognition.- 9.1 Learning.- 9.2Bayesian Formalism.- 9.2.1 Dynamic Bayesian Theory.- 9.2.2 Recognition and Search.- 9.2.3 Influences.- 9.3 Subjectivity.- 9.4 Background Information.- 9.5 Exercises.- 10 Stochastic Processes.- 10.1 Preliminaries on Probabilities.- 10.2 Basic Concepts of Stochastic Processes.- 10.2.1 Markov Processes.- 10.2.2 Hidden Stochastic Models (HSM).- 10.2.3 HSM Topology.- 10.2.4 Learning Probabilities.- 10.2.5 Re-estimation.- 10.2.6 Redundancy.- 10.2.7 Data Preparation.- 10.2.8 Proper Redundancy Removal.- 10.3 Envelope Detection.- 10.3.1 Silence Threshold Selection.- 10.3.2 Pre-emphasis.- Contents xiii.- 10.4 Several Processes.- 10.4.1 Similarity.- 10.4.2 The Local-Global Principle.- 10.4.3 HSM Similarities.- 10.5 Conflict and Support.- 10.6 Examples and Applications.- 10.7 Predictions.- 10.8 Background Information.- 10.9 Exercises.- 11 Feature Extraction.- 11.1 Feature Extractions.- 11.2 Basic Techniques.- 11.2.1 Spectral Shaping.- 11.3 Spectral Analysis and Feature Transformation.- 11.3.1 Parametric Feature Transformations and Cepstrum.- 11.3.2 Standard Feature Extraction Techniques.- 11.3.3 Frame Energy.- 11.4 Linear Prediction Coe_cients (LPC).- 11.5 Linear Prediction Cepstral Coe_cients (LPCC).- 11.6 Adaptive Perceptual Local Trigonometric Transformation.- (APLTT).- 11.7 Search.- 11.7.1 General Search Model.- 11.8 Predictions.- 11.8.1 Purpose.- 11.8.2 Linear Prediction.- 11.8.3 Mean Squared Error Minimization.- 11.8.4 Computation of Probability of an Observation Sequence.- 11.8.5 Forward and Backward Prediction.- 11.8.6 Forward-Backward Prediction.- 11.9 Background Information.- 11.10Exercises.- 12 Unsupervised Learning.- 12.1 Generalities.- 12.2 Clustering Principles.- 12.3 Cluster Analysis Methods.- 12.4 Special Methods.- 12.4.1 K-means.- 12.4.2 Vector Quantization (VQ).- 12.4.3 Expectation Maximization (EM).- 12.4.4 GMM Clustering.- 12.5 Background Information.- 12.6 Exercises.- xiv Contents.- 13 Markov Model and Hidden Stochastic Model.-13.1 Markov Process.- 13.2 Gaussian Mixture Model (GMM).- 13.3 Advantages of using GMM.- 13.4 Linear Prediction Analysis.- 13.4.1 Autocorrelation Method.- 13.4.2 Yule-Walker Approach.- 13.4.3 Covariance Method.- 13.4.4 Comparison of Correlation and Covariance methods.- 13.5 The ULS Approach.- 13.6 Comparison of ULS and Covariance Methods.- 13.7 Forward Prediction.- 13.8 Backward Prediction.- 13.9 Forward-Backward Prediction.- 13.10Baum-Welch Algorithm.- 13.11Viterbi Algorithm.- 13.12Background Information.- 13.13Exercises.- 14 Fuzzy Logic and Rough Sets.- 14.1 Rough Sets.- 14.2 Fuzzy Sets.- 14.2.1 Basis Elements.- 14.2.2 Possibility and Necessity.- 14.3 Fuzzy Clustering.- 14.4 Fuzzy Probabilities.- 14.5 Background Information.- 14.6 Exercises.- 15 Neural Networks.- 15.1 Neural Network Types.- 15.1.1 Neural Network Training.- 15.1.2 Neural Network Topology.- 15.2 Parallel Distributed Processing.- 15.2.1 Forward and Backward Uses.- 15.2.2 Learning.- 15.3 Applications to Signal Processing.- 15.4 Background Information.- 15.5 Exercises.- Part III Real Aspects and Applications.- Contents xv.- 16 Noisy Signals.- 16.1 Introduction.- 16.2 Noise Questions.- 16.3 Sources of Noise.- 16.4 Noise Measurement.- 16.5 Weights and A-Weights.- 16.6 Signal to Noise Ratio (SNR).- 16.7 Noise Measuring Filters and Evaluation.- 16.8 Types of noise.- 16.9 Origin of noises.- 16.10Box Plot Evaluation.- 16.11Individual noise types.- 16.11.1Residual.- 16.11.2Mild.- 16.11.3Steady-unsteady Time varying Noise.- 16.11.4Strong Noise.- 16.12Solution to Strong Noise: Matched Filter.- 16.13Background Information.- 16.14Exercises.- 17 Reasoning Methods and Noise Removal.- 17.1 Generalities.- 17.2 Special Noise Removal Methods.- 17.2.1 Residual Noise.- 17.2.2 Mild Noise.- 17.2.3 Steady-Unsteady Noise.- 17.2.4 Strong Noise.- 17.3 Poisson Distribution.- 17.3.1 Outliers and Shots.- 17.3.2 Underlying probability of Shots.- 17.4 Kalman Filter.- 17.4.1 Prediction Estimates.-17.4.2 White noise Kalman filtering.- 17.4.3 Application of Kalman filter.- 17.5 Classification, Recognition and Learning.- 17.5.1 Summary of the used concepts.- 17.6 Principle Component Analysis (PCA).- 17.7 Reasoning Methods.- 17.7.1 Case-Based Reasoning (CBR).- 17.8 Background Information.- 17.9 Exercises.- xvi Contents.- 18 Audio Signals and Speech Recognition.- 18.1 Generalities of Speech.- 18.2 Categories of Speech Recognition.- 18.3 Automatic Speech Recognition.- 18.3.1 System Structure.- 18.4 Speech Production Model.- 18.5 Acoustics.- 18.6 Human Speech Production.- 18.6.1 The Human Speech Generation.- 18.6.2 Excitation.- 18.6.3 Voiced Speech.- 18.6.4 Unvoiced Speech.- 18.7 Silence Regions.- 18.8 Glottis.- 18.9 Lips.- 18.10Plosive Speech Source.- 18.11Vocal-Tract.- 18.12Parametric and Non-Parametric Models.- 18.13Formants.- 18.14Strong Noise.- 18.15Background Information.- 18.16Exercises.- 19 Noisy Speech.- 19.1 Introduction.- 19.2 Colored Noise.- 19.2.1 Additionaltypes of Colored Noise.- 19.3 Poisson Processes and Shots.- 19.4 Matched Filters.- 19.5 Shot Noise.- 19.6 Background Information.- 19.7 Exercises.- 20 Aspects Of Human Hearing.- 20.1 Human Ear.- 20.2 Human Auditory System.- 20.3 Critical Bands and Scales.- 20.3.1 Mel Scale.- 20.3.2 Bark Scale.- 20.3.3 Erb Scale.- 20.3.4 Greenwood Scale.- 20.4 Filter Banks.- 20.4.1 ICA Network.- 20.4.2 Auditory Filter Banks.- 20.4.3 Filter Banks.- Contents xvii.- 20.4.4 Mel Critical Filter Bank.- 20.5 Psycho-acoustic Phenomena.- 20.5.1 Perceptual Measurement.- 20.5.2 Human Hearing and Perception.- 20.5.3 Sound Pressure Level (SPL).- 20.5.4 Absolute Threshold of Hearing (ATH).- 20.6 Perceptual Adaptation.- 20.7 Auditory System and Hearing Model.- 20.8 Auditory Masking and Masking Frequency.- 20.9 Perceptual Spectral Features.- 20.10Critical Band Analysis.- 20.11Equal Loudness Pre-emphasis.- 20.12Perceptual Transformation.- 20.13Feature Transformation.- 20.14Filters and Human Ear.- 20.15Temporal Aspects.- 20.16Background Information.- 20.17Exercises.- 21 Speech Features.- 21.1 Generalities.- 21.2 Cost Functions.- 21.3 Special Feature Extractions.- 21.3.1 MFCC Features.- 21.3.2 Feature Transformation applying DCT.- 21.4 Background Information.- 21.5 Exercises.- 22 Hidden Stochastic Model for Speech.- 22.1 General.- 22.2 Hidden Stochastic Model.- 22.3 Forward and Backward Predictions.- 22.3.1 Forward Algorithm.- 22.3.2 Backward Algorithm.- 22.4 Forward-Backward Prediction.- 22.5 Burg Approach.- 22.6 Graph Search.- 22.6.1 Recognition Model with Search.- 22.7 Semantic Issues and Industrial Applications.- 22.8 Problems with Noise.- 22.9 Aspects of Music.- 22.10Music reception.- 22.11Background Information.- 22.12Exercises.- xviii Contents.- 23 Different Speech Applications – Part A.- 23.1 Generalities.- 23.2 Example Applications.- 23.2.1 Experimental laboratory.- 23.2.2 Health care support (everyday actions).- 23.2.3 Diagnostic support for persons with possible dementia.- 23.2.4 Noise.- 23.3 Background Information.- 23.4 Exercises.- 24 Different Speech Applications – Part B.- 24.1 Introduction.- 24.2 Discrete-Time Signals.- 24.3 Speech Processing.- 24.3.1 Framing.- 24.3.2 Pre-emphasis.- 24.3.3 Windowing.- 24.3.4 Fourier Transform.- 24.3.5 Mel-Filtering.- 24.3.6 Mel-Frequency Cepstral Coeffcients.- 24.4 Speech Analysis and Sound Effects Laboratory (SASE_Lab).- 24.5 Wake-Up-Word Speech Recognition.- 24.5.1 Introduction.- 24.5.2 Wake-up-Word Paradigm.- 24.5.3 Wake-Up-Word: Definition.- 24.5.4 Wake-Up-Word System.- 24.5.5 Front-End of the Wake-Up-Word System.- 24.6 Conclusion.- 24.6.1 Wake-Up-Word: Tool Demo.- 24.6.2 Elevator Simulator.- 24.7 Background Information.- 24.8 Exercises.- 24.9 Speech Analysis and Sound E_ects Laboratory (SASE_Lab)" .- 25 Biomedical Signals: ECG, EEG.- 25.1 ECG signals.- 25.1.1 Bioelectric Signals.- 25.1.2 Noise.- 25.2 EEG Signals.- 25.2.1 General properties.- 25.2.2 Signal types and properties.- 25.2.3 Disadvantages.- 25.3 Neural Network use.- 25.4 Major Research Questions.- 25.5 Background Information.- Contents xix.- 25.6 Exercises.- 26 Seismic Signals.- 26.1 Generalities.- 26.2 Sources of seismic signals.- 26.3 Intermediate elements.- 26.4 Practical Data Sources.- 26.5 Major seismic problems.- 26.6 Noise.- 26.7 Background Information.- 26.8 Exercises.- 27 Radar Signals.- 27.1 Introduction.- 27.2 Radar Types and Applications.- 27.3 Doppler Equations, Ambiguity Function(AF) and Matched.- Filter.- 27.4 Moving Target Detection.- 27.5 Applications and Discussions.- 27.6 Examples.- 27.7 Background Information.- 27.8 Exercises.- 28 Visual Story Telling.- 28.1 Introduction.- 28.1.1 Common Visualization Approaches.- 28.2 Analytics and Visualization.- 28.2.1 Visualization.- 28.2.2 Visual Data Minin.- 28.3 Communication and Visualization.- 28.4 Background Information.- 28.5 Exercises.- 29 Digital Processes and Multimedia.- 29.1 Images.- 29.1.1 DigitalImage Processing.- 29.1.2 Images as Matrices.- 29.1.3 Gray Scale Images.- 29.2 Spatial Filtering.- 29.2.1 Linear Filtering of Images.- 29.2.2 Separable Filters.- 29.2.3 Mechanics of Linear Spatial Filtering Operation.- 29.3 Median Filtering.- 29.4 Color Equalization.- 29.4.1 Image Transformations.- 29.4.2 Examples of Image Transformation Matrixes.- xx Contents.- 29.5 Basic Image Statistics.- 29.6 Abstraction Levels of Images and its Representations.- 29.6.1 Lowest Level.- 29.6.2 Geometric Level.- 29.6.3 Domain Level.- 29.6.4 Segmentation.- 29.7 Background Information.- 29.8 Exercises.- 30 Visualizations of Emergency Operation Centre.- 30.1 Introduction.- 30.2 Communications in Emergency Situations.- 30.3 Emergency Scenario.- 30.3.1 Classification and EOC Scenario.- 30.4 Technical Aspects and Techniques.- 30.4.1 Classification.- 30.4.2 Clustering.- 30.5 Background Information.- 30.6 Exercises.- 31 Intelligent Interactive Communications.- 31.1 Introduction.- 31.2 Spoken Dialogue System.- 31.3 Gesture based Interaction.- 31.4 Object Recognition and Identification.- 31.5 Visual Story Telling.- 31.6 Virtual Environment for Personal Assistance.- 31.7 Sensor Fusion.- 31.8 Intelligent Human Machine for Communication.- Application Scenario.- 31.9 Background Information.- 31.10Exercises.- 32 Comparisons.- 32.1 Generalities.- 32.1.1 EEG and ECG.- 32.1.2 Speech and biomedical applications.- 32.1.3 Seismic and biomedical signals.- 32.1.4 Speech and Images.- 32.2 Overall.- 32.3 Background Information.- 32.3.1 General.- 32.4 Exercises.- Glossary.