Virtual and Augmented Reality

ACHINTYA K. BHOWMIK, PHD, is Chief Technology Officer at Starkey, a global leader in hearing technology and sensory augmentation, and an Adjunct Professor at Stanford University, where he teaches human-computer interaction and intelligent systems. Previously, he led pioneering work in 3D sensing and perceptual computing at Intel. Dr. Bhowmik is a Fellow of the IEEE, SID, AAIA, and AIIA, and his innovations have been recognized with numerous honors, including TIME’s Best Inventions and the Red Dot Design Award. He has authored over 200 publications and multiple books on immersive and interactive technologies.

• Series Editor's Foreword xxiiiAbout the Author xxvPreface xxviiAcknowledgments xxixSection 1 Overview and Evolution 11 Overview of Virtual and Augmented Reality: Toward Immersive and Interactive Experiences 31.1 Introduction 31.2 Key Concepts in Immersive Technologies 41.3 Multisensory Immersion and Interaction 71.4 Goals of the Book 91.5 Overview of the Structure and Content 101.6 Conclusion 122 The Evolution of Virtual and Augmented Reality: Accomplishments and Challenges 172.1 Introduction 172.2 Virtual and Augmented Reality in Science Fiction 182.3 The Early Concepts Through to the 1950s and 1960s 222.4 The 1980s and 1990s: Commercial Exploration 272.5 The 2000s: Expansion into New Fields 312.6 The 2010s: Technological Innovations and Mainstream Adoption 332.7 The 2020s: Integration, Convergence, and Pushing the Frontiers 402.8 Challenges and Limitations 452.9 Summary 49Section 2 Sensory-Perceptual Immersion 573 Visual Immersion: Foundations of Human Vision 593.1 Introduction 593.2 Anatomy of the Human Eye 603.3 Visual Information Processing in the Brain 643.4 Field of View (FOV) for Human Vision 663.5 Perception of Depth and Space 673.6 Motion Perception 723.7 Color Vision 733.8 Brightness and Contrast Perception 743.9 Summary 764 Visual Immersion: Displays and Optics 794.1 Introduction 794.2 Display System Configurations 804.3 Key Attributes of Displays for VR and AR 844.4 Display Technologies 894.5 Optics for VR and AR Systems 1024.6 Advanced Topics and Future Trends 1074.7 Summary 1125 Visual Immersion: Creating Virtual Worlds with 3D Graphics 1195.1 Introduction 1195.2 Fundamentals of Graphics in VR and AR 1215.3 Rendering Pipeline 1245.4 Animation and Motion 1405.5 Advanced 3D Graphics Techniques 1425.6 Computational Hardware and Software Tools 1475.7 Future Trends and Innovations 1535.8 Summary 1546 Visual Immersion: Understanding and Augmenting the World with Computer Vision 1616.1 Introduction 1616.2 Fundamentals of Computer Vision 1626.3 3D Reconstruction 1816.4 Augmenting the Reality: Overlaying Virtual on Real 1826.5 Summary 1897 Auditory Immersion: Spatial Sound and 3D Audio 1957.1 Introduction 1957.2 Basic Principles of Sound Waves 1967.3 Fundamentals of Human Auditory Perception 1987.4 Sound Technologies in Virtual and Augmented Reality 2107.5 Summary 2238 Vestibular Immersion: Motion Sensing and Stimulation 2318.1 Introduction 2318.2 The Human Vestibular System 2338.3 Creating Vestibular Immersion 2368.4 Motion Tracking Techniques 2378.5 Technologies for Vestibular Stimulation 2538.6 Summary 2629 Somatosensory Immersion: Touch and Haptic Feedback 2739.1 Introduction 2739.2 Physiology of Somatosensory Perception 2759.3 Haptic Feedback for Immersive Systems 2799.4 Designing Haptic Feedback Systems 2929.5 Applications of Haptic Feedback Technologies 2959.6 Summary 29710 Olfactory Immersion: Smell Simulation 30310.1 Introduction 30310.2 Basics of Olfactory Perception 30410.3 Enhancing Virtual Experiences with Olfactory Feedback 30810.4 User Experience Considerations 31110.5 Future Directions and Innovations 31610.6 Summary 31911 Multisensory Integration: Creating Unified Immersive Experiences 32511.1 Introduction 32511.2 Foundations of Multisensory Integration 32611.3 Challenges in Multisensory Integration 33211.4 Synchronization Across Modalities 33411.5 Applications of Multisensory Integration 33811.6 Summary 341Section 3 Human Inputs and Interactions 34912 Hand Gesture Recognition: Intuitive Interfaces 35112.1 Introduction 35112.2 Physiology of the Human Hand and Gesture Categories 35212.3 Fundamentals of Hand Gesture Tracking 35512.4 Gesture Tracking System Architectures 35712.5 Key Components of Gesture Tracking Systems 36012.6 Gesture Recognition Algorithms 36412.7 Applications of Hand Gesture Tracking 37112.8 Challenges and Limitations 37412.9 Future Trends and Research Directions 37512.10 Summary 37613 Eye-Gaze Tracking: Enabling Adaptive Interactions 38513.1 Introduction 38513.2 Physiology of Eye Movement 38613.3 Types of Eye-Gaze Tracking Systems 39013.4 Technologies and Methods 39113.5 Calibration and Accuracy 39613.6 Integration with VR and AR Systems 39913.7 Applications of Eye-Gaze Tracking in VR and AR 40313.8 Challenges and Limitations 40513.9 Future Trends and Research Directions 40613.10 Summary 40714 Speech Recognition: Voice as a Natural Input Modality 41514.1 Introduction 41514.2 Fundamentals of Voice Recognition 41614.3 Types of Voice Recognition Systems 42014.4 Technologies and Methods 42114.5 Integration with VR and AR Systems 43714.6 Applications of Speech Recognition in VR and AR 43814.7 Challenges and Limitations 44014.8 Future Directions 44214.9 Summary 44315 Facial Expression Recognition: Avatars and Emotional Presence 45115.1 Introduction 45115.2 Physiology of the Human Facial Expression 45215.3 Technologies for Facial Expression Tracking 45415.4 Real-Time Expression Rendering in Avatars 46015.5 Personalization and User Representation 46415.6 Impact on Immersion and Presence 46615.7 Challenges and Limitations 46715.8 Summary 46916 Brain–Computer Interfaces: Direct Neural Interactions 47716.1 Introduction 47716.2 BCI Principles and Systems 47816.3 Hardware Components 48416.4 Software Components 48716.5 BCI Applications 49116.6 Challenges and Limitations 49416.7 Future Trends and Research Directions 49516.8 Summary 49617 Multimodal Interactions: Bridging Inputs for Holistic Experiences 50317.1 Introduction 50317.2 The Case for Multimodal Interactions 50417.3 Principles of Multimodal System Design 50617.4 Combining Interaction Modalities 50817.5 Key Technologies for Multimodal Systems 51117.6 Applications of Multimodal Interactions 51217.7 Challenges in Multimodal Integration 51417.8 Future Directions 51617.9 Summary 517Section 4 Systems and Applications 52518 System Architectures and Designs: Integration Challenges 52718.1 Introduction 52718.2 Integration Challenges and Usability Considerations 52818.3 Designing for Scalability and Collaboration 53818.4 Case Studies in System Integration 53918.5 Summary 54219 Applications of Virtual and Augmented Reality: Innovating Across Industries 54919.1 Introduction 54919.2 Healthcare and Medicine 55019.3 Education and Training 55319.4 Enterprise and Industrial Applications 55519.5 Entertainment and Media 55919.6 Retail and E-Commerce 56219.7 Military and Defense 56519.8 Arts, Culture, and Tourism 56819.9 Summary 572Section 5 Reflections and Outlook 58720 Accessibility and Inclusivity: Designing Safe and Equitable Systems 58920.1 Introduction 58920.2 Principles of Accessible and Inclusive Design 59020.3 Accessibility Features in Virtual and Augmented Reality Systems 59220.4 Inclusivity in Multicultural and Global Contexts 60020.5 Ethical Considerations and User Safety 60220.6 Summary 60621 The Simulation Hypothesis: Are We Already Living in a Virtual World? 61521.1 Introduction 61521.2 Origins and Theoretical Foundations 61621.3 Technological Feasibility 61721.4 Supports and Counterarguments 61821.5 Societal Implications 62021.6 Cultural and Psychological Impact 62121.7 Future Directions and Research 62221.8 Summary 62222 Toward the Future: Emerging Directions in Virtual and Augmented Reality 62722.1 Introduction 62722.2 Technology Trends Shaping the Future 62822.3 Expanding Applications 62922.4 Human-Centric Designs 63122.5 Challenges and Research Opportunities 63222.6 The Broader Societal and Cultural Impact 63422.7 A Vision for the Future 63522.8 Summary 636Index 639