3D Videocommunication

Dr Oliver Schreer, Heinrich-Hertz-Institute, & TU Berlin, Germany Oliver Schreer is Adjunct Professor at the Faculty of Electrical Engineering and Computer Science, Technical University Berlin. He lectures on Image Processing in Videocommunications and is a regular guest editor for the IEEE Transactions on Circuits, Systems and Video Technology. Dr Peter Kauff, Heinrich-Hertz-Institute, Berlin, Germany Peter Kauff is the head of the Immersive Media & 3D Video Group at Heinrich-Hertz-Institute (HHI), Fraunhofer Gesellschaft, Berlin. He has been involved in numerous German and European projects related to digital HDTV signal processing and coding, interactive MPEG-4-based services, and advanced 3D video processing for immersive tele-presence and immersive media. Professor Dr Thomas Sikora, Head of the Communication Systems Group, Technical University of Berlin, Berlin As the chairman of the ISO-MPEG video group (Moving Picture Experts Group), Dr Sikora was responsible for the development and standardization of the MPEG video coding algorithms. He frequently works as an industry consultant on issues related to interactive digital video. He is an appointed member of the Supervisory board of a number of German companies and international research organizations. He is an Associate Editor for IEEE Signal Processing Magazine and the EURASIP Signal Processing: Image Communication journal and currently serves as the Editor-in-Chief of the IEEE Transactions on Circuits and Systems for Video Technology.

List of Contributors xiii Symbols xix Abbreviations xxi Introduction 1 Oliver Schreer, Peter Kauff and Thomas Sikora Section I Applications of 3D Videocommunication 5 1 History of Telepresence 7 Wijnand A. IJsselsteijn 1.1 Introduction 7 1.2 The Art of Immersion: Barkers Panoramas 10 1.3 Cinerama and Sensorama 11 1.4 Virtual Environments 14 1.5 Teleoperation and Telerobotics 16 1.6 Telecommunications 18 1.7 Conclusion 19 References 20 2 3D TV Broadcasting 23 Christoph Fehn 2.1 Introduction 23 2.2 History of 3D TV Research 24 2.3 A Modern Approach to 3D TV 26 2.3.1 A Comparison with a Stereoscopic Video Chain 28 2.4 Stereoscopic View Synthesis 29 2.4.1 3D Image Warping 29 2.4.2 A Virtual Stereo Camera 30 2.4.3 The Disocclusion Problem 32 2.5 Coding of 3D Imagery 34 2.5.1 Human Factor Experiments 35 2.6 Conclusions 36 Acknowledgements 37 References 37 3 3D in Content Creation and Post-production 39 Oliver Grau 3.1 Introduction 39 3.2 Current Techniques for Integrating Real and Virtual Scene Content 41 3.3 Generation of 3D Models of Dynamic Scenes 44 3.4 Implementation of a Bidirectional Interface Between Real and Virtual Scenes 46 3.4.1 Head Tracking 49 3.4.2 View-dependent Rendering 50 3.4.3 Mask Generation 50 3.4.4 Texturing 51 3.4.5 Collision Detection 52 3.5 Conclusions 52 References 52 4 Free Viewpoint Systems 55 Masayuki Tanimoto 4.1 General Overview of Free Viewpoint Systems 55 4.2 Image Domain System 57 4.2.1 EyeVision 57 4.2.2 3D-TV 58 4.2.3 Free Viewpoint Play 59 4.3 Ray-space System 59 4.3.1 FTV (Free Viewpoint TV) 59 4.3.2 Birds-eye View System 60 4.3.3 Light Field Video Camera System 62 4.4 Surface Light Field System 64 4.5 Model-based System 65 4.5.1 3D Room 65 4.5.2 3D Video 66 4.5.3 Multi-texturing 67 4.6 Integral Photography System 68 4.6.1 NHK System 68 4.6.2 1D-II 3D Display System 70 4.7 Summary 70 References 71 5 Immersive Videoconferencing 75 Peter Kauff and Oliver Schreer 5.1 Introduction 75 5.2 The Meaning of Telepresence in Videoconferencing 76 5.3 Multi-party Communication Using the Shared Table Concept 79 5.4 Experimental Systems for Immersive Videoconferencing 83 5.5 Perspective and Trends 87 Acknowledgements 88 References 88 Section II 3D Data Representation and Processing 91 6 Fundamentals of Multiple-view Geometry 93 Spela Ivekovic, Andrea Fusiello and Emanuele Trucco 6.1 Introduction 93 6.2 Pinhole Camera Geometry 94 6.3 Two-view Geometry 96 6.3.1 Introduction 96 6.3.2 Epipolar Geometry 97 6.3.3 Rectification 102 6.3.4 3D Reconstruction 104 6.4 N-view Geometry 106 6.4.1 Trifocal Geometry 106 6.4.2 The Trifocal Tensor 108 6.4.3 Multiple-view Constraints 109 6.4.4 Uncalibrated Reconstruction from N views 110 6.4.5 Autocalibration 111 6.5 Summary 112 References 112 7 Stereo Analysis 115 Nicole Atzpadin and Jane Mulligan 7.1 Stereo Analysis Using Two Cameras 115 7.1.1 Standard Area-based Stereo Analysis 117 7.1.2 Fast Real-time Approaches 120 7.1.3 Post-processing 123 7.2 Disparity From Three or More Cameras 125 7.2.1 Two-camera versus Three-camera Disparity 127 7.2.2 Correspondence Search with Three Views 128 7.2.3 Post-processing 129 7.3 Conclusion 130 References 130 8 Reconstruction of Volumetric 3D Models 133 Peter Eisert 8.1 Introduction 133 8.2 Shape-from-Silhouette 135 8.2.1 Rendering of Volumetric Models 136 8.2.2 Octree Representation of Voxel Volumes 137 8.2.3 Camera Calibration from Silhouettes 139 8.3 Space-carving 140 8.4 Epipolar Image Analysis 143 8.4.1 Horizontal Camera Motion 143 8.4.2 Image Cube Trajectory Analysis 145 8.5 Conclusions 148 References 148 9 View Synthesis and Rendering Methods 151 Reinhard Koch and Jan-Friso Evers-Senne 9.1 The Plenoptic Function 152 9.1.1 Sampling the Plenoptic Function 152 9.1.2 Recording of the Plenoptic Sampl