Machine Learning and Metaheuristic Computation

Erik Cuevas, PhD, is a Full Professor in the Department of Electronics at the University of Guadalajara. He is a Member of the Mexican Academy of Sciences and the National System of Researchers. He has provided editorial services on several specialized journals. Jorge Galvez, PhD, is a Full Professor in the Department of Innovation Based on Information and Knowledge at the University of Guadalajara. He is a Member of the Mexican Academy of Sciences and the National System of Researchers. Omar Avalos, PhD, is a Professor in the Electronics and Computing Division of the University Center for Exact Sciences and Engineering at the University of Guadalajara. He is a Member of the Mexican Academy of Sciences and the National System of Researchers. Fernando Wario, PhD, is a Professor at the University of Guadalajara and an Associate Researcher at the Institute of Cognitive Sciences and Technologies (ISTC) in Rome, Italy. He is a Member of the Mexican Academy of Sciences and the National System of Researchers.

• About the Authors xiPreface xiiiAcknowledgments xviiIntroduction xix1 Fundamentals of Machine Learning 11.1 Introduction 11.2 Different Types of Machine Learning Approaches 41.3 Supervised Learning 61.4 Unsupervised Learning 81.5 Reinforcement Learning 101.6 Which Algorithm to Apply? 131.7 Recommendation to Build a Machine Learning Model 15References 192 Introduction to Metaheuristics Methods 212.1 Introduction 212.2 Classic Optimization Methods 232.3 Descending Gradient Method 242.4 Metaheuristic Methods 292.5 Exploitation and Exploration 352.6 Acceptance and Probabilistic Selection 372.7 Random Search 412.8 Simulated Annealing 47References 573 Fundamental Machine Learning Methods 593.1 Introduction 593.2 Regression 603.2.1 Explanatory Purpose 623.2.2 Predictive Purpose 623.3 Classification 713.3.1 Relationship Between Regression and Classification 723.3.2 Differences Between Regression and Classification 723.4 Decision Trees 733.4.1 Procedure of Classification 743.4.2 Determination of the Splitting Point 773.4.2.1 Gini Index 773.4.2.2 Entropy 783.4.3 Example of Classification 793.5 Bayesian Classification 863.5.1 Conditional Probability 873.5.2 Classification of Fraudulent Financial Reports 873.5.3 Practical Constraints by Using the Exact Bayes Method 903.5.4 Naive Bayes Method 903.5.5 Computational Experiment 923.6 k-Nearest Neighbors (k-NN) 993.6.1 k-NN for Classification 993.6.2 k-NN for Regression 1013.7 Clustering 1053.7.1 Similarity Indexes 1073.7.2 Methods for Clustering 1083.8 Hierarchical Clustering 1123.8.1 Implementation in MATLAB 1143.9 K-Means Algorithm 1223.9.1 Implementation of K-Means Method in MATLAB 1273.10 Expectation-Maximization Method 1303.10.1 Gaussian Mixture Models 1313.10.2 Maximum Likelihood Estimation 1313.10.3 EM in One Dimension 1323.10.3.1 Initialization 1323.10.3.2 Expectation 1323.10.3.3 Maximization 1333.10.4 Numerical Example 1333.10.5 EM in Several Dimensions 135References 1414 Main Metaheuristic Techniques 1454.1 Introduction 1454.1.1 Use of Metaphors 1454.1.2 Problems of the Use of Metaphors 1464.1.3 Metaheuristic Algorithms 1474.2 Genetic Algorithms 1484.2.1 Canonical Genetic Algorithm 1494.2.2 Selection Process 1524.2.3 Binary Crossover Process 1554.2.4 Binary Mutation Process 1564.2.5 Implementation of the Binary GA 1574.2.6 Genetic Algorithm Utilizing Real-Valued Parameters 1644.2.7 Crossover Operator for Real-Valued Parameters 1654.2.8 Mutation Operator for Real-Valued Parameters 1764.2.9 Computational Implementation of the GA with Real Parameters 1814.3 Particle Swarm Optimization (PSO) 1894.3.1 Strategy for Searching in Particle Swarm Optimization 1894.3.2 Analysis of the PSO Algorithm 1924.3.3 Inertia Weighting 1924.3.4 Particle Swarm Optimization Algorithm Using MATLAB 1934.4 Differential Evolution (DE) Algorithm 1964.4.1 The Search Strategy of DE 1974.4.2 The Mutation Operation in DE 2004.4.2.1 Mutation Rand/ 1 2014.4.2.2 Mutación Best/ 1 2014.4.2.3 Mutation Rand/ 2 2024.4.2.4 Mutation Best/ 2 2024.4.2.5 Mutation Current-to-Best/ 1 2034.4.3 The Crossover Operation in DE 2034.4.4 The Selection Operation in DE 2054.4.5 Implementation of DE in MATLAB 205References 2095 Metaheuristic Techniques for Fine-Tuning Parameter of Complex Systems 2115.1 Introduction 2115.2 Differential Evolution (DE) 2115.2.1 Mutation 2125.2.1.1 Mutation Best/ 1 2135.2.1.2 Mutation Rand/ 2 2135.2.1.3 Mutation Best/ 2 2135.2.1.4 Mutation Current-to-Best/ 1 2135.2.2 Crossover 2135.2.3 Selection 2145.3 Adaptive Network-Based Fuzzy Inference System (ANFIS) 2195.4 Differential Evolution for Fine-Tuning ANFIS Parameters Setting 220References 2366 Techniques of Machine Learning for Producing Metaheuristic Operators 2376.1 Introduction 2376.2 Hierarchical Clustering 2386.2.1 Agglomerative Hierarchical Clustering Algorithm 2396.3 Chaotic Sequences 2436.4 Cluster-Chaotic-Optimization (CCO) 2456.4.1 Initialization 2466.4.2 Clustering 2466.4.3 Intra-Cluster Procedure 2476.4.3.1 Local Attraction Movement 2476.4.3.2 Local Perturbation Strategy 2476.4.3.3 Extra-Cluster Procedure 2486.4.3.4 Global Attraction Movement 2496.4.3.5 Global Perturbation Strategy 2496.5 Computational Procedure 2506.6 Implementation of the CCO Algorithm in MATLAB 2506.7 Spring Design Optimization Problem Using the CCO Algorithm in MATLAB 258References 2677 Techniques of Machine Learning for Modifying the Search Strategy 2697.1 Introduction 2697.2 Self-Organization Map (SOM) 2707.2.1 Network Architecture 2727.2.2 Competitive Learning Model 2737.2.2.1 Competition Procedure 2737.2.2.2 Cooperation Procedure 2747.2.2.3 Synaptic Adaptation Procedure 2757.2.3 Self-Organization Map (SOM) Algorithm 2757.2.4 Application of Self-Organization Map (SOM) 2767.3 Evolutionary-SOM (EA-SOM) 2777.3.1 Initialization 2807.3.2 Training 2817.3.3 Knowledge Extraction 2817.3.4 Solution Production 2827.3.5 New Training Set Construction 2837.4 Computational Procedure 2837.5 Implementation of the EA-SOM Algorithm in MATLAB 2847.6 Gear Design Optimization Problem Using the EA-SOM Algorithm in MATLAB 289References 2948 Techniques of Machine Learning Mixed with Metaheuristic Methods 2978.1 Introduction 2978.2 Flower Pollination Algorithm (FPA) 2988.2.1 Global Rule and Lévy Flight 2988.2.2 Local Rule 2998.2.3 Elitist Selection Procedure 2998.3 Feedforward Neural Networks (FNNs) 3038.3.1 Perceptron 3058.3.2 Feedforward Neural Networks (FNNs) 3058.4 Training an FNN Using FPA 306References 3089 Metaheuristic Methods for Classification 3119.1 Introduction 3119.2 Crow Search Algorithm (CSA) 3119.3 CSA for Nearest-Neighbor Method (k-NN) 3159.4 CSA for Logistic Regression 3199.5 CSA for Fisher Linear Discriminant 3239.6 CSA for Naïve Bayes Classification 3269.7 CSA for Support Vector Machine 330References 33610 Metaheuristic Methods for Clustering 33910.1 Introduction 33910.2 Cuckoo Search Method (CSM) 34010.3 Search Strategy for CSM 34010.3.1 Initialization 34210.3.2 Lévy Flight 34210.3.3 Solution Replacement 34410.3.4 Elitist Selection 34410.4 Computational Procedure 34510.4.1 Metaheuristic Operators for CSM 34510.5 Implementation of the CSM in MATLAB 34710.6 Cuckoo Search Method for K-Means 35210.6.1 Implementation of KM algorithm in MATLAB 35410.6.2 Cuckoo Search Method for K-Means 35610.6.2.1 Implementation of CSM to KM Clustering in MATLAB 358References 36311 Metaheuristic Methods for Dimensional Reduction 36511.1 Introduction 36511.2 Ant Colony Optimization (ACO) 36511.2.1 Pheromone Representation 36611.2.2 Ant-Based Solution Construction 36711.2.3 Pheromone Update 36711.3 Dimensionality Reduction 37211.4 ACO for Feature Selection 373References 37512 Metaheuristic Methods for Regression 37712.1 Introduction 37712.2 Genetic Algorithm (GA) 37712.2.1 Computational Structure 37812.2.2 Initialization 37812.2.3 Selection Method 37812.2.3.1 Roulette Wheel Selection 37812.2.3.2 Stochastic Reminder Selection 37912.2.3.3 Rank-Based Selection 37912.2.3.4 Tournament Selection 38012.2.4 Crossover 38012.2.5 Mutation 38112.3 Neural Network Regression with Artificial Genetic 38612.4 Linear Regression Employing an Artificial Genetic 391References 396Index 397