Normalization Techniques in Deep Learning

Lei Huang, Ph.D., is an Associate Professor and Doctoral Supervisor in the School of Artificial Intelligence (Institute) at Beihang University, China.  His current research mainly focuses on normalization techniques (involving methods, theories, and applications) in training DNNs. He also has wide interests in deep learning theory (representation and optimization) and computer vision tasks. Dr. Huang has published over 40 peer-reviewed articles in top-tier machine learning and computer vision conferences and journals such as CVPR, NeurIPS, ICML, and IEEE TPAMI. He serves in program committees and as a reviewer for the top-tier conferences and journals in machine learning and computer vision.  Dr. Huang received his B.Sc. and Ph.D. degrees from Beihang University.

• Introduction.- Motivation and Overview of Normalization in DNNs.- A General View of Normalizing Activations.- A Framework for Normalizing Activations as Functions.- Multi-Mode and Combinational Normalization.- BN for More Robust Estimation.- Normalizing Weights.- Normalizing Gradients.- Analysis of Normalization.- Normalization in Task-specific Applications.- Summary and Discussion.