Oliver Schreer, Fraunhofer Heinrich-Hertz-Institut, Berlin, GermanyDr. Schreer is Scientific Project Manager of the "Immersive Media & 3D Video" Group in the Image Processing Department at HHI, Berlin. He is responsible for the lead of European Research projects (e.g. FascinatE project) and coordinating research on 3D video processing and immersive media.Graham Thomas, BBC Research & Development, London, UKDr. Thomas is Principal Research Engineer and Section Lead of the Production Magic department at BBC R&D. He leads a team of engineers developing 3D image processing and graphics techniques for TV production. His work has led to many commercial products, e.g. the Piero sports graphics system, which won a Queen’s Award in 2011. He holds 20 patents and is Visiting Professor at University of Surrey.Ben Shirley, University of Salford, UKBen Shirley is Senior Lecturer at the University of Salford and Academic Lead for the Digital Innovation Cluster in the University’s College of Science & Technology at MediaCityUK. He has been principal investigator for a range of broadcast related research projects including leading Salford’s work in the EU FascinatE project. He teaches audio technology and broadcast engineering at undergraduate and postgraduate level.Georg Thallinger, Joanneum Research, Graz, AustriaDr. Thallinger is Head of the research group "Audiovisual Media" at JOANNEUM RESEARCH, Austria. He oversees a group active in the field of digital media with a focus on video and image processing as well as metadata for applications in media production and archiving, film restoration, media monitoring and security. Dr. Thallinger is involved in the EU FascinatE project.Jean-Francois Macq, Alcatel-Lucent, Antwerp, BelgiumDr. Macq is a Senior Research Engineer at Bell Labs, the research organization of Alcatel-Lucent. He has been responsible for several national and international research projects in the field of video coding and delivery, currently including the EU FascinatE project.Javier Ruiz-Hidalgo, Universitat Politècnica de Catalunya (UPC), Barcelona, SpainDr. Ruiz-Hidalgo is an Associate Professor at the Universitat Politècnica de Catalunya. He has been involved in various European Projects as a principal researcher from the Image Processing Group at UPC, including the EU FascinatE project.Omar Niamut, Netherlands Organization for Applied Scientific Research (TNO), Delft, The NetherlandsDr. Niamut is a Senior Research Scientist in the "Media Networks and Services" Group at TNO. He has advised the European Parliament on the harmonization of mobile TV and was editor of the technology report on mobile TV in the European Commission-assigned European Mobile Broadcast Council. He participated in ETSI TISPAN standardization with over 200 contributions on interactive services for next generation IPTV and advised the Singapore government on the use of IPTV standards. He has been involved in a number of European projects, including the Fascinate project. Dr. Niamut holds 15 patents.

Select Guide Rating

• List of Editors and Contributors xiiiList of Abbreviations xviiNotations xxiii1 Introduction 1Oliver Schreer, Jean-Franc¸ois Macq, Omar Aziz Niamut, Javier Ruiz-Hidalgo, Ben Shirley, Georg Thallinger and Graham Thomas2 State-of-the-Art and Challenges in Media Production, Broadcast and Delivery 5Graham Thomas, Arvid Engstr¨om, Jean-Franc¸ois Macq, Omar Aziz Niamut, Ben Shirley and Richard Salmon2.1 Introduction 52.2 Video Fundamentals and Acquisition Technology 72.2.1 How Real Should Video Look? 72.2.2 Fundamentals of Video 92.2.3 Camera Technology 142.2.4 Production for Formats with Differing Aspect Ratios 192.2.5 Stereoscopic 3D Video 202.2.6 Challenges for the Future 212.3 Audio Fundamentals and Acquisition Technology 212.3.1 Introduction 212.3.2 Fundamentals of Audio 212.3.3 Non-Live Production 272.3.4 Live Production 312.3.5 Surround Sound 342.3.6 Challenges for the Future 342.4 Live Programme Production 342.4.1 The Production Area and Roles in Production 352.4.2 The Production Workspace 372.4.3 Vision Mixing: Techniques for Directing, Selecting and Mixing Camera Feeds 382.4.4 Audio Mixing 402.4.5 Replay Operation in Live Television Production 402.4.6 Challenges for the Future 422.5 Coding and Delivery 432.5.1 Managed Delivery Networks 432.5.2 Unmanaged Delivery Networks 472.5.3 Codecs and Transport Protocols 482.5.4 Challenges for the Future 502.6 Display Technology 502.6.1 Plasma Displays – Developing the Flat Panel Display Market 512.6.2 LCD – the Current Dominant Technology 522.6.3 Projection Technologies 532.6.4 Other Technologies 542.6.5 Impact on Broadcasters 552.6.6 Challenges for the Future 562.7 Audio Reproduction Technology 562.7.1 Stereophonic Sound Systems 572.7.2 Holophonic Systems 592.7.3 Binaural Systems 612.7.4 Hybrid Systems 622.7.5 Challenges for the Future 622.8 Use of Archive Material 622.8.1 Video Format Conversion 632.8.2 Audio Format Conversion 642.8.3 Challenges for the Future 642.9 Concept of Format-Agnostic Media 652.9.1 Limitations of Current Production and Delivery Approaches 652.9.2 A New Approach: Format-Agnostic Media 652.9.3 Metadata for Enabling Rich User Interaction 662.9.4 A Format-Agnostic Media Production and Delivery System 672.10 Conclusion 68Notes 69References 693 Video Acquisition 74Oliver Schreer, Ingo Feldmann, Richard Salmon, Johannes Steurer and Graham Thomas3.1 Introduction 743.2 Ultra-High Definition Panoramic Video Acquisition 753.2.1 History of Panoramic Imaging 753.2.2 The Geometry of Two Views 793.2.3 Fundamentals of Panoramic Video Acquisition 823.2.4 Geometrical Constraints for Parallax Free Stitching of Two Images 843.2.5 Registration of Views 883.2.6 Stitching, Warping and Blending of Views 913.3 Use of Conventional Video Content to Enhance Panoramic Video 943.3.1 Calibration of Camera Position and Orientation 943.3.2 Photometric Matching of Panoramic and Broadcast Cameras 983.3.3 Blending and Switching Between Camera Views 1013.4 High Frame Rate Video 1023.4.1 Early Work on HDTV Frame Rates 1043.4.2 Issues with Conventional Frame Rates 1043.4.3 Practical Investigations into the Effects of High Frame Rates 1073.4.4 Future Frame Rates for TV Production and Distribution 1113.4.5 Consideration of Frame Rates and Motion Portrayal in Synthetic Production 1113.4.6 Conclusions on Frame Rates 1123.5 High Dynamic Range Video 1123.5.1 The Human Visual System in Natural Environments 1133.5.2 Conventional and HDR Video Cameras 1153.5.3 Conventional and HDR Displays 1173.5.4 HDR Video Formats 1193.5.5 Adaptive Tone-mapping for Format-Agnostic Video 1203.6 Conclusion 125Notes 126References 1264 Platform Independent Audio 130Ben Shirley, Rob Oldfield, Frank Melchior and Johann-Markus Batke4.1 Introduction 1304.2 Terms and Definitions 1324.2.1 Auditory Event and Sound Event 1324.2.2 Basic Room Acoustics Theory 1344.3 Definition of the Problem Space 1354.3.1 Reproduction Environment 1354.3.2 Reproduction Method 1374.3.3 Audio-visual Coherence 1414.3.4 User Interaction 1434.3.5 Example Scenario 1434.4 Scene Representation 1444.4.1 Components of a Virtual Sound Scene 1444.4.2 Representations of Virtual Sound Scenes 1464.4.3 Implementation Examples 1474.5 Scene Acquisition 1494.5.1 Capturing Discrete Audio Objects 1504.5.2 Capturing the Sound Field Component 1524.5.3 Capturing the Diffuse Field 1534.6 Scene Reproduction 1534.6.1 Scenario: Mobile Consumption Via Headphones 1534.6.2 Scenario: Interactive Multichannel Reproduction 1544.6.3 Scenario: Big Screen 1544.6.4 Scenario: Interactive and Free Viewpoint 3D 1554.7 Existing Systems 1564.7.1 Commercial Systems 1564.7.2 Research Projects 1574.7.3 Perceptive Media 1604.8 Conclusion 1614.8.1 Open Issues 162References 1625 Semi-Automatic Content Annotation 166Werner Bailer, Marco Masetti, Goranka Zori´c, Marcus Thaler and Georg Thallinger5.1 Introduction 1665.1.1 Requirements on Semi-automatic Annotation Tools 1675.1.2 Requirements on Metadata 1685.2 Metadata Models and Analysis Architectures 1705.2.1 Metadata Models 1705.2.2 Architectures for Audio-visual Analysis 1715.2.3 Storing MPEG-7 Metadata 1725.2.4 Bulk Loading Techniques for Massive MPEG-7 Metadata Storage 1755.2.5 An Example Architecture of a Semantic Layer Management System 1755.3 Domain-independent Saliency 1775.3.1 Spatio-temporal Visual Saliency 1775.3.2 Estimating Grid-based Saliency 1785.3.3 Salient Regions for Controlling Automated Shot Selection 1795.4 Person Detection and Tracking 1805.4.1 Person Detection 1815.4.2 Person Tracking 1825.4.3 Multicamera and Panoramic Environment 1845.4.4 GPU Accelerated Real-time Tracking Beyond HD 1875.5 Online Detection of Concepts and Actions 1895.5.1 Sequence-based Kernels 1905.5.2 Kernels for Online Detection 1935.5.3 Performance of Online Kernels 1945.6 Supporting Annotation for Automated Production 1955.6.1 User Preferences and Functionality Definition 1955.6.2 Design Overview and Principles 1965.6.3 Preconfiguration and Predefined Workspaces 1985.7 Conclusion 204References 2056 Virtual Director 209Rene Kaiser and Wolfgang Weiss6.1 Introduction 2096.1.1 What is a Virtual Director? 2106.1.2 Features Enabled by a Virtual Director 2116.1.3 Definitions 2126.1.4 Requirements for Virtual Director Technology 2136.1.5 Existing Implementations and Research Activities 2156.2 Implementation Approaches 2196.2.1 Behaviour Implementation Approaches 2206.2.2 Combining Rule Engines and Event Processing Technology 2236.3 Example Architecture and Workflow 2256.3.1 Workflow of the Production System 2256.3.2 Workflow of the Virtual Director 2266.3.3 Distributed Nature 2286.3.4 Sources of Knowledge 2286.4 Virtual Director Subprocesses 2306.4.1 Semantic Lifting 2306.4.2 Shot Candidate Identification 2316.4.3 Shot Framing 2336.4.4 Shot Prioritisation 2346.4.5 Decision Making 2356.5 Behaviour Engineering: Production Grammar 2376.5.1 Production Knowledge Elicitation Process 2376.5.2 Cinematic Techniques 2386.5.3 Audio Scripting 2416.5.4 Domain Model 2416.5.5 Limitations in Rule-based Behaviour Engineering 2426.6 Virtual Director: Example Prototype 2436.6.1 Architecture and Software Framework 2456.6.2 Production Scripts 2466.6.3 Behaviour Implementation 2476.6.4 Production Grammar Example 2486.7 Conclusion 2516.7.1 Summary 2516.7.2 Limitations 2516.7.3 Testing and Evaluation 2526.7.4 Research Roadmap 2536.7.5 Concluding Thoughts 255References 2567 Scalable Delivery of Navigable and Ultra-High Resolution Video 260Jean-Franc¸ois Macq, Patrice Rond˜ao Alface, Ray van Brandenburg, Omar Aziz Niamut, Martin Prins and Nico Verzijp7.1 Introduction 2607.2 Delivery of Format-Agnostic Content: Key Concepts and State-of-the-Art 2627.2.1 Delivery Agnostic to Content Formats, Device Capabilities and Network Bandwidth 2627.2.2 Delivery Agnostic to Video Timing – the Temporal Interactivity Case 2647.2.3 Delivery Agnostic to Video Reframing – the Spatial Interactivity Case 2667.3 Spatial Random Access in Video Coding 2677.3.1 Video Compression and Random Access – A Fundamental Trade-off 2687.3.2 Spatial Random Access by Tracking Coding Dependencies 2717.3.3 Spatial Random Access by Frame Tiling 2717.3.4 Multi-Resolution Tiling 2737.3.5 Overlapping Tiling for Low-Powered Devices 2747.4 Models for Adaptive Tile-based Representation and Delivery 2767.4.1 Saliency-based Adaptive Coding of Tiled Content 2777.4.2 Optimisation of Tile Selection Under Delay and Bandwidth Constraints 2807.5 Segment-based Adaptive Transport 2817.5.1 Video Streaming Over IP 2827.5.2 Tiled HTTP Adaptive Streaming Over the Internet 2857.5.3 Publish/Subscribe System for Interactive Video Streaming 2897.6 Conclusion 294References 2948 Interactive Rendering 298Javier Ruiz-Hidalgo, Malte Borsum, Axel Kochale and Goranka Zori´c8.1 Introduction 2988.2 Format-Agnostic Rendering 2998.2.1 Available Rendering Solutions in End Terminals 2998.2.2 Requirements for Format-Agnostic Video Rendering 3068.2.3 Description of a Technical Solution 3088.3 Device-less Interaction for Rendering Control 3118.3.1 Sensors for Gesture Recognition 3148.3.2 Gesture Analysis Techniques 3178.3.3 Recognition and Classification Techniques 3198.3.4 Remaining Challenges in Gesture Recognition Systems 3218.3.5 Description of a Technical Solution 3218.3.6 User Study of the Gesture Interface 3278.4 Conclusions 331References 3329 Application Scenarios and Deployment Domains 337Omar Aziz Niamut, Arvid Engstr¨om, Axel Kochale, Jean-Franc¸ois Macq, Graham Thomas and Goranka Zori´c9.1 Introduction 3379.2 Application Scenarios 3389.2.1 Digital Cinema: A Totally Immersive Experience 3389.2.2 Home Theatre: In Control 3399.2.3 Mobile: Navigation and Magnification 3399.3 Deployment in the Production Domain 3409.3.1 Outlook on New Production Practices 3409.3.2 Use of Format-Agnostic Technology to Aid Conventional Production 3429.3.3 Production of Format-Agnostic Content to Support End User Interaction 3449.4 Deployment in the Network Domain 3479.4.1 Network Requirements 3479.4.2 Impact of Application Scenarios 3489.5 Deployment in the Device Domain 3519.5.1 Device Capabilities 3519.5.2 User and Producer Expectations 3559.6 Deployment in the User Domain 3569.7 Conclusion 357References 357Index 359

Oliver Schreer, Fraunhofer Heinrich-Hertz-Institut, Berlin, GermanyDr. Schreer is Scientific Project Manager of the "Immersive Media & 3D Video" Group in the Image Processing Department at HHI, Berlin. He is responsible for the lead of European Research projects (e.g. FascinatE project) and coordinating research on 3D video processing and immersive media.Graham Thomas, BBC Research & Development, London, UKDr. Thomas is Principal Research Engineer and Section Lead of the Production Magic department at BBC R&D. He leads a team of engineers developing 3D image processing and graphics techniques for TV production. His work has led to many commercial products, e.g. the Piero sports graphics system, which won a Queen’s Award in 2011. He holds 20 patents and is Visiting Professor at University of Surrey.Ben Shirley, University of Salford, UKBen Shirley is Senior Lecturer at the University of Salford and Academic Lead for the Digital Innovation Cluster in the University’s College of Science & Technology at MediaCityUK. He has been principal investigator for a range of broadcast related research projects including leading Salford’s work in the EU FascinatE project. He teaches audio technology and broadcast engineering at undergraduate and postgraduate level.Georg Thallinger, Joanneum Research, Graz, AustriaDr. Thallinger is Head of the research group "Audiovisual Media" at JOANNEUM RESEARCH, Austria. He oversees a group active in the field of digital media with a focus on video and image processing as well as metadata for applications in media production and archiving, film restoration, media monitoring and security. Dr. Thallinger is involved in the EU FascinatE project.Jean-Francois Macq, Alcatel-Lucent, Antwerp, BelgiumDr. Macq is a Senior Research Engineer at Bell Labs, the research organization of Alcatel-Lucent. He has been responsible for several national and international research projects in the field of video coding and delivery, currently including the EU FascinatE project.Javier Ruiz-Hidalgo, Universitat Politècnica de Catalunya (UPC), Barcelona, SpainDr. Ruiz-Hidalgo is an Associate Professor at the Universitat Politècnica de Catalunya. He has been involved in various European Projects as a principal researcher from the Image Processing Group at UPC, including the EU FascinatE project.Omar Niamut, Netherlands Organization for Applied Scientific Research (TNO), Delft, The NetherlandsDr. Niamut is a Senior Research Scientist in the "Media Networks and Services" Group at TNO. He has advised the European Parliament on the harmonization of mobile TV and was editor of the technology report on mobile TV in the European Commission-assigned European Mobile Broadcast Council. He participated in ETSI TISPAN standardization with over 200 contributions on interactive services for next generation IPTV and advised the Singapore government on the use of IPTV standards. He has been involved in a number of European projects, including the Fascinate project. Dr. Niamut holds 15 patents.

Select Guide Rating

• List of Editors and Contributors xiiiList of Abbreviations xviiNotations xxiii1 Introduction 1Oliver Schreer, Jean-Franc¸ois Macq, Omar Aziz Niamut, Javier Ruiz-Hidalgo, Ben Shirley, Georg Thallinger and Graham Thomas2 State-of-the-Art and Challenges in Media Production, Broadcast and Delivery 5Graham Thomas, Arvid Engstr¨om, Jean-Franc¸ois Macq, Omar Aziz Niamut, Ben Shirley and Richard Salmon2.1 Introduction 52.2 Video Fundamentals and Acquisition Technology 72.2.1 How Real Should Video Look? 72.2.2 Fundamentals of Video 92.2.3 Camera Technology 142.2.4 Production for Formats with Differing Aspect Ratios 192.2.5 Stereoscopic 3D Video 202.2.6 Challenges for the Future 212.3 Audio Fundamentals and Acquisition Technology 212.3.1 Introduction 212.3.2 Fundamentals of Audio 212.3.3 Non-Live Production 272.3.4 Live Production 312.3.5 Surround Sound 342.3.6 Challenges for the Future 342.4 Live Programme Production 342.4.1 The Production Area and Roles in Production 352.4.2 The Production Workspace 372.4.3 Vision Mixing: Techniques for Directing, Selecting and Mixing Camera Feeds 382.4.4 Audio Mixing 402.4.5 Replay Operation in Live Television Production 402.4.6 Challenges for the Future 422.5 Coding and Delivery 432.5.1 Managed Delivery Networks 432.5.2 Unmanaged Delivery Networks 472.5.3 Codecs and Transport Protocols 482.5.4 Challenges for the Future 502.6 Display Technology 502.6.1 Plasma Displays – Developing the Flat Panel Display Market 512.6.2 LCD – the Current Dominant Technology 522.6.3 Projection Technologies 532.6.4 Other Technologies 542.6.5 Impact on Broadcasters 552.6.6 Challenges for the Future 562.7 Audio Reproduction Technology 562.7.1 Stereophonic Sound Systems 572.7.2 Holophonic Systems 592.7.3 Binaural Systems 612.7.4 Hybrid Systems 622.7.5 Challenges for the Future 622.8 Use of Archive Material 622.8.1 Video Format Conversion 632.8.2 Audio Format Conversion 642.8.3 Challenges for the Future 642.9 Concept of Format-Agnostic Media 652.9.1 Limitations of Current Production and Delivery Approaches 652.9.2 A New Approach: Format-Agnostic Media 652.9.3 Metadata for Enabling Rich User Interaction 662.9.4 A Format-Agnostic Media Production and Delivery System 672.10 Conclusion 68Notes 69References 693 Video Acquisition 74Oliver Schreer, Ingo Feldmann, Richard Salmon, Johannes Steurer and Graham Thomas3.1 Introduction 743.2 Ultra-High Definition Panoramic Video Acquisition 753.2.1 History of Panoramic Imaging 753.2.2 The Geometry of Two Views 793.2.3 Fundamentals of Panoramic Video Acquisition 823.2.4 Geometrical Constraints for Parallax Free Stitching of Two Images 843.2.5 Registration of Views 883.2.6 Stitching, Warping and Blending of Views 913.3 Use of Conventional Video Content to Enhance Panoramic Video 943.3.1 Calibration of Camera Position and Orientation 943.3.2 Photometric Matching of Panoramic and Broadcast Cameras 983.3.3 Blending and Switching Between Camera Views 1013.4 High Frame Rate Video 1023.4.1 Early Work on HDTV Frame Rates 1043.4.2 Issues with Conventional Frame Rates 1043.4.3 Practical Investigations into the Effects of High Frame Rates 1073.4.4 Future Frame Rates for TV Production and Distribution 1113.4.5 Consideration of Frame Rates and Motion Portrayal in Synthetic Production 1113.4.6 Conclusions on Frame Rates 1123.5 High Dynamic Range Video 1123.5.1 The Human Visual System in Natural Environments 1133.5.2 Conventional and HDR Video Cameras 1153.5.3 Conventional and HDR Displays 1173.5.4 HDR Video Formats 1193.5.5 Adaptive Tone-mapping for Format-Agnostic Video 1203.6 Conclusion 125Notes 126References 1264 Platform Independent Audio 130Ben Shirley, Rob Oldfield, Frank Melchior and Johann-Markus Batke4.1 Introduction 1304.2 Terms and Definitions 1324.2.1 Auditory Event and Sound Event 1324.2.2 Basic Room Acoustics Theory 1344.3 Definition of the Problem Space 1354.3.1 Reproduction Environment 1354.3.2 Reproduction Method 1374.3.3 Audio-visual Coherence 1414.3.4 User Interaction 1434.3.5 Example Scenario 1434.4 Scene Representation 1444.4.1 Components of a Virtual Sound Scene 1444.4.2 Representations of Virtual Sound Scenes 1464.4.3 Implementation Examples 1474.5 Scene Acquisition 1494.5.1 Capturing Discrete Audio Objects 1504.5.2 Capturing the Sound Field Component 1524.5.3 Capturing the Diffuse Field 1534.6 Scene Reproduction 1534.6.1 Scenario: Mobile Consumption Via Headphones 1534.6.2 Scenario: Interactive Multichannel Reproduction 1544.6.3 Scenario: Big Screen 1544.6.4 Scenario: Interactive and Free Viewpoint 3D 1554.7 Existing Systems 1564.7.1 Commercial Systems 1564.7.2 Research Projects 1574.7.3 Perceptive Media 1604.8 Conclusion 1614.8.1 Open Issues 162References 1625 Semi-Automatic Content Annotation 166Werner Bailer, Marco Masetti, Goranka Zori´c, Marcus Thaler and Georg Thallinger5.1 Introduction 1665.1.1 Requirements on Semi-automatic Annotation Tools 1675.1.2 Requirements on Metadata 1685.2 Metadata Models and Analysis Architectures 1705.2.1 Metadata Models 1705.2.2 Architectures for Audio-visual Analysis 1715.2.3 Storing MPEG-7 Metadata 1725.2.4 Bulk Loading Techniques for Massive MPEG-7 Metadata Storage 1755.2.5 An Example Architecture of a Semantic Layer Management System 1755.3 Domain-independent Saliency 1775.3.1 Spatio-temporal Visual Saliency 1775.3.2 Estimating Grid-based Saliency 1785.3.3 Salient Regions for Controlling Automated Shot Selection 1795.4 Person Detection and Tracking 1805.4.1 Person Detection 1815.4.2 Person Tracking 1825.4.3 Multicamera and Panoramic Environment 1845.4.4 GPU Accelerated Real-time Tracking Beyond HD 1875.5 Online Detection of Concepts and Actions 1895.5.1 Sequence-based Kernels 1905.5.2 Kernels for Online Detection 1935.5.3 Performance of Online Kernels 1945.6 Supporting Annotation for Automated Production 1955.6.1 User Preferences and Functionality Definition 1955.6.2 Design Overview and Principles 1965.6.3 Preconfiguration and Predefined Workspaces 1985.7 Conclusion 204References 2056 Virtual Director 209Rene Kaiser and Wolfgang Weiss6.1 Introduction 2096.1.1 What is a Virtual Director? 2106.1.2 Features Enabled by a Virtual Director 2116.1.3 Definitions 2126.1.4 Requirements for Virtual Director Technology 2136.1.5 Existing Implementations and Research Activities 2156.2 Implementation Approaches 2196.2.1 Behaviour Implementation Approaches 2206.2.2 Combining Rule Engines and Event Processing Technology 2236.3 Example Architecture and Workflow 2256.3.1 Workflow of the Production System 2256.3.2 Workflow of the Virtual Director 2266.3.3 Distributed Nature 2286.3.4 Sources of Knowledge 2286.4 Virtual Director Subprocesses 2306.4.1 Semantic Lifting 2306.4.2 Shot Candidate Identification 2316.4.3 Shot Framing 2336.4.4 Shot Prioritisation 2346.4.5 Decision Making 2356.5 Behaviour Engineering: Production Grammar 2376.5.1 Production Knowledge Elicitation Process 2376.5.2 Cinematic Techniques 2386.5.3 Audio Scripting 2416.5.4 Domain Model 2416.5.5 Limitations in Rule-based Behaviour Engineering 2426.6 Virtual Director: Example Prototype 2436.6.1 Architecture and Software Framework 2456.6.2 Production Scripts 2466.6.3 Behaviour Implementation 2476.6.4 Production Grammar Example 2486.7 Conclusion 2516.7.1 Summary 2516.7.2 Limitations 2516.7.3 Testing and Evaluation 2526.7.4 Research Roadmap 2536.7.5 Concluding Thoughts 255References 2567 Scalable Delivery of Navigable and Ultra-High Resolution Video 260Jean-Franc¸ois Macq, Patrice Rond˜ao Alface, Ray van Brandenburg, Omar Aziz Niamut, Martin Prins and Nico Verzijp7.1 Introduction 2607.2 Delivery of Format-Agnostic Content: Key Concepts and State-of-the-Art 2627.2.1 Delivery Agnostic to Content Formats, Device Capabilities and Network Bandwidth 2627.2.2 Delivery Agnostic to Video Timing – the Temporal Interactivity Case 2647.2.3 Delivery Agnostic to Video Reframing – the Spatial Interactivity Case 2667.3 Spatial Random Access in Video Coding 2677.3.1 Video Compression and Random Access – A Fundamental Trade-off 2687.3.2 Spatial Random Access by Tracking Coding Dependencies 2717.3.3 Spatial Random Access by Frame Tiling 2717.3.4 Multi-Resolution Tiling 2737.3.5 Overlapping Tiling for Low-Powered Devices 2747.4 Models for Adaptive Tile-based Representation and Delivery 2767.4.1 Saliency-based Adaptive Coding of Tiled Content 2777.4.2 Optimisation of Tile Selection Under Delay and Bandwidth Constraints 2807.5 Segment-based Adaptive Transport 2817.5.1 Video Streaming Over IP 2827.5.2 Tiled HTTP Adaptive Streaming Over the Internet 2857.5.3 Publish/Subscribe System for Interactive Video Streaming 2897.6 Conclusion 294References 2948 Interactive Rendering 298Javier Ruiz-Hidalgo, Malte Borsum, Axel Kochale and Goranka Zori´c8.1 Introduction 2988.2 Format-Agnostic Rendering 2998.2.1 Available Rendering Solutions in End Terminals 2998.2.2 Requirements for Format-Agnostic Video Rendering 3068.2.3 Description of a Technical Solution 3088.3 Device-less Interaction for Rendering Control 3118.3.1 Sensors for Gesture Recognition 3148.3.2 Gesture Analysis Techniques 3178.3.3 Recognition and Classification Techniques 3198.3.4 Remaining Challenges in Gesture Recognition Systems 3218.3.5 Description of a Technical Solution 3218.3.6 User Study of the Gesture Interface 3278.4 Conclusions 331References 3329 Application Scenarios and Deployment Domains 337Omar Aziz Niamut, Arvid Engstr¨om, Axel Kochale, Jean-Franc¸ois Macq, Graham Thomas and Goranka Zori´c9.1 Introduction 3379.2 Application Scenarios 3389.2.1 Digital Cinema: A Totally Immersive Experience 3389.2.2 Home Theatre: In Control 3399.2.3 Mobile: Navigation and Magnification 3399.3 Deployment in the Production Domain 3409.3.1 Outlook on New Production Practices 3409.3.2 Use of Format-Agnostic Technology to Aid Conventional Production 3429.3.3 Production of Format-Agnostic Content to Support End User Interaction 3449.4 Deployment in the Network Domain 3479.4.1 Network Requirements 3479.4.2 Impact of Application Scenarios 3489.5 Deployment in the Device Domain 3519.5.1 Device Capabilities 3519.5.2 User and Producer Expectations 3559.6 Deployment in the User Domain 3569.7 Conclusion 357References 357Index 359

Oliver Schreer, Fraunhofer Heinrich-Hertz-Institut, Berlin, GermanyDr. Schreer is Scientific Project Manager of the "Immersive Media & 3D Video" Group in the Image Processing Department at HHI, Berlin. He is responsible for the lead of European Research projects (e.g. FascinatE project) and coordinating research on 3D video processing and immersive media.Graham Thomas, BBC Research & Development, London, UKDr. Thomas is Principal Research Engineer and Section Lead of the Production Magic department at BBC R&D. He leads a team of engineers developing 3D image processing and graphics techniques for TV production. His work has led to many commercial products, e.g. the Piero sports graphics system, which won a Queen’s Award in 2011. He holds 20 patents and is Visiting Professor at University of Surrey.Ben Shirley, University of Salford, UKBen Shirley is Senior Lecturer at the University of Salford and Academic Lead for the Digital Innovation Cluster in the University’s College of Science & Technology at MediaCityUK. He has been principal investigator for a range of broadcast related research projects including leading Salford’s work in the EU FascinatE project. He teaches audio technology and broadcast engineering at undergraduate and postgraduate level.Georg Thallinger, Joanneum Research, Graz, AustriaDr. Thallinger is Head of the research group "Audiovisual Media" at JOANNEUM RESEARCH, Austria. He oversees a group active in the field of digital media with a focus on video and image processing as well as metadata for applications in media production and archiving, film restoration, media monitoring and security. Dr. Thallinger is involved in the EU FascinatE project.Jean-Francois Macq, Alcatel-Lucent, Antwerp, BelgiumDr. Macq is a Senior Research Engineer at Bell Labs, the research organization of Alcatel-Lucent. He has been responsible for several national and international research projects in the field of video coding and delivery, currently including the EU FascinatE project.Javier Ruiz-Hidalgo, Universitat Politècnica de Catalunya (UPC), Barcelona, SpainDr. Ruiz-Hidalgo is an Associate Professor at the Universitat Politècnica de Catalunya. He has been involved in various European Projects as a principal researcher from the Image Processing Group at UPC, including the EU FascinatE project.Omar Niamut, Netherlands Organization for Applied Scientific Research (TNO), Delft, The NetherlandsDr. Niamut is a Senior Research Scientist in the "Media Networks and Services" Group at TNO. He has advised the European Parliament on the harmonization of mobile TV and was editor of the technology report on mobile TV in the European Commission-assigned European Mobile Broadcast Council. He participated in ETSI TISPAN standardization with over 200 contributions on interactive services for next generation IPTV and advised the Singapore government on the use of IPTV standards. He has been involved in a number of European projects, including the Fascinate project. Dr. Niamut holds 15 patents.

Select Guide Rating

• List of Editors and Contributors xiiiList of Abbreviations xviiNotations xxiii1 Introduction 1Oliver Schreer, Jean-Franc¸ois Macq, Omar Aziz Niamut, Javier Ruiz-Hidalgo, Ben Shirley, Georg Thallinger and Graham Thomas2 State-of-the-Art and Challenges in Media Production, Broadcast and Delivery 5Graham Thomas, Arvid Engstr¨om, Jean-Franc¸ois Macq, Omar Aziz Niamut, Ben Shirley and Richard Salmon2.1 Introduction 52.2 Video Fundamentals and Acquisition Technology 72.2.1 How Real Should Video Look? 72.2.2 Fundamentals of Video 92.2.3 Camera Technology 142.2.4 Production for Formats with Differing Aspect Ratios 192.2.5 Stereoscopic 3D Video 202.2.6 Challenges for the Future 212.3 Audio Fundamentals and Acquisition Technology 212.3.1 Introduction 212.3.2 Fundamentals of Audio 212.3.3 Non-Live Production 272.3.4 Live Production 312.3.5 Surround Sound 342.3.6 Challenges for the Future 342.4 Live Programme Production 342.4.1 The Production Area and Roles in Production 352.4.2 The Production Workspace 372.4.3 Vision Mixing: Techniques for Directing, Selecting and Mixing Camera Feeds 382.4.4 Audio Mixing 402.4.5 Replay Operation in Live Television Production 402.4.6 Challenges for the Future 422.5 Coding and Delivery 432.5.1 Managed Delivery Networks 432.5.2 Unmanaged Delivery Networks 472.5.3 Codecs and Transport Protocols 482.5.4 Challenges for the Future 502.6 Display Technology 502.6.1 Plasma Displays – Developing the Flat Panel Display Market 512.6.2 LCD – the Current Dominant Technology 522.6.3 Projection Technologies 532.6.4 Other Technologies 542.6.5 Impact on Broadcasters 552.6.6 Challenges for the Future 562.7 Audio Reproduction Technology 562.7.1 Stereophonic Sound Systems 572.7.2 Holophonic Systems 592.7.3 Binaural Systems 612.7.4 Hybrid Systems 622.7.5 Challenges for the Future 622.8 Use of Archive Material 622.8.1 Video Format Conversion 632.8.2 Audio Format Conversion 642.8.3 Challenges for the Future 642.9 Concept of Format-Agnostic Media 652.9.1 Limitations of Current Production and Delivery Approaches 652.9.2 A New Approach: Format-Agnostic Media 652.9.3 Metadata for Enabling Rich User Interaction 662.9.4 A Format-Agnostic Media Production and Delivery System 672.10 Conclusion 68Notes 69References 693 Video Acquisition 74Oliver Schreer, Ingo Feldmann, Richard Salmon, Johannes Steurer and Graham Thomas3.1 Introduction 743.2 Ultra-High Definition Panoramic Video Acquisition 753.2.1 History of Panoramic Imaging 753.2.2 The Geometry of Two Views 793.2.3 Fundamentals of Panoramic Video Acquisition 823.2.4 Geometrical Constraints for Parallax Free Stitching of Two Images 843.2.5 Registration of Views 883.2.6 Stitching, Warping and Blending of Views 913.3 Use of Conventional Video Content to Enhance Panoramic Video 943.3.1 Calibration of Camera Position and Orientation 943.3.2 Photometric Matching of Panoramic and Broadcast Cameras 983.3.3 Blending and Switching Between Camera Views 1013.4 High Frame Rate Video 1023.4.1 Early Work on HDTV Frame Rates 1043.4.2 Issues with Conventional Frame Rates 1043.4.3 Practical Investigations into the Effects of High Frame Rates 1073.4.4 Future Frame Rates for TV Production and Distribution 1113.4.5 Consideration of Frame Rates and Motion Portrayal in Synthetic Production 1113.4.6 Conclusions on Frame Rates 1123.5 High Dynamic Range Video 1123.5.1 The Human Visual System in Natural Environments 1133.5.2 Conventional and HDR Video Cameras 1153.5.3 Conventional and HDR Displays 1173.5.4 HDR Video Formats 1193.5.5 Adaptive Tone-mapping for Format-Agnostic Video 1203.6 Conclusion 125Notes 126References 1264 Platform Independent Audio 130Ben Shirley, Rob Oldfield, Frank Melchior and Johann-Markus Batke4.1 Introduction 1304.2 Terms and Definitions 1324.2.1 Auditory Event and Sound Event 1324.2.2 Basic Room Acoustics Theory 1344.3 Definition of the Problem Space 1354.3.1 Reproduction Environment 1354.3.2 Reproduction Method 1374.3.3 Audio-visual Coherence 1414.3.4 User Interaction 1434.3.5 Example Scenario 1434.4 Scene Representation 1444.4.1 Components of a Virtual Sound Scene 1444.4.2 Representations of Virtual Sound Scenes 1464.4.3 Implementation Examples 1474.5 Scene Acquisition 1494.5.1 Capturing Discrete Audio Objects 1504.5.2 Capturing the Sound Field Component 1524.5.3 Capturing the Diffuse Field 1534.6 Scene Reproduction 1534.6.1 Scenario: Mobile Consumption Via Headphones 1534.6.2 Scenario: Interactive Multichannel Reproduction 1544.6.3 Scenario: Big Screen 1544.6.4 Scenario: Interactive and Free Viewpoint 3D 1554.7 Existing Systems 1564.7.1 Commercial Systems 1564.7.2 Research Projects 1574.7.3 Perceptive Media 1604.8 Conclusion 1614.8.1 Open Issues 162References 1625 Semi-Automatic Content Annotation 166Werner Bailer, Marco Masetti, Goranka Zori´c, Marcus Thaler and Georg Thallinger5.1 Introduction 1665.1.1 Requirements on Semi-automatic Annotation Tools 1675.1.2 Requirements on Metadata 1685.2 Metadata Models and Analysis Architectures 1705.2.1 Metadata Models 1705.2.2 Architectures for Audio-visual Analysis 1715.2.3 Storing MPEG-7 Metadata 1725.2.4 Bulk Loading Techniques for Massive MPEG-7 Metadata Storage 1755.2.5 An Example Architecture of a Semantic Layer Management System 1755.3 Domain-independent Saliency 1775.3.1 Spatio-temporal Visual Saliency 1775.3.2 Estimating Grid-based Saliency 1785.3.3 Salient Regions for Controlling Automated Shot Selection 1795.4 Person Detection and Tracking 1805.4.1 Person Detection 1815.4.2 Person Tracking 1825.4.3 Multicamera and Panoramic Environment 1845.4.4 GPU Accelerated Real-time Tracking Beyond HD 1875.5 Online Detection of Concepts and Actions 1895.5.1 Sequence-based Kernels 1905.5.2 Kernels for Online Detection 1935.5.3 Performance of Online Kernels 1945.6 Supporting Annotation for Automated Production 1955.6.1 User Preferences and Functionality Definition 1955.6.2 Design Overview and Principles 1965.6.3 Preconfiguration and Predefined Workspaces 1985.7 Conclusion 204References 2056 Virtual Director 209Rene Kaiser and Wolfgang Weiss6.1 Introduction 2096.1.1 What is a Virtual Director? 2106.1.2 Features Enabled by a Virtual Director 2116.1.3 Definitions 2126.1.4 Requirements for Virtual Director Technology 2136.1.5 Existing Implementations and Research Activities 2156.2 Implementation Approaches 2196.2.1 Behaviour Implementation Approaches 2206.2.2 Combining Rule Engines and Event Processing Technology 2236.3 Example Architecture and Workflow 2256.3.1 Workflow of the Production System 2256.3.2 Workflow of the Virtual Director 2266.3.3 Distributed Nature 2286.3.4 Sources of Knowledge 2286.4 Virtual Director Subprocesses 2306.4.1 Semantic Lifting 2306.4.2 Shot Candidate Identification 2316.4.3 Shot Framing 2336.4.4 Shot Prioritisation 2346.4.5 Decision Making 2356.5 Behaviour Engineering: Production Grammar 2376.5.1 Production Knowledge Elicitation Process 2376.5.2 Cinematic Techniques 2386.5.3 Audio Scripting 2416.5.4 Domain Model 2416.5.5 Limitations in Rule-based Behaviour Engineering 2426.6 Virtual Director: Example Prototype 2436.6.1 Architecture and Software Framework 2456.6.2 Production Scripts 2466.6.3 Behaviour Implementation 2476.6.4 Production Grammar Example 2486.7 Conclusion 2516.7.1 Summary 2516.7.2 Limitations 2516.7.3 Testing and Evaluation 2526.7.4 Research Roadmap 2536.7.5 Concluding Thoughts 255References 2567 Scalable Delivery of Navigable and Ultra-High Resolution Video 260Jean-Franc¸ois Macq, Patrice Rond˜ao Alface, Ray van Brandenburg, Omar Aziz Niamut, Martin Prins and Nico Verzijp7.1 Introduction 2607.2 Delivery of Format-Agnostic Content: Key Concepts and State-of-the-Art 2627.2.1 Delivery Agnostic to Content Formats, Device Capabilities and Network Bandwidth 2627.2.2 Delivery Agnostic to Video Timing – the Temporal Interactivity Case 2647.2.3 Delivery Agnostic to Video Reframing – the Spatial Interactivity Case 2667.3 Spatial Random Access in Video Coding 2677.3.1 Video Compression and Random Access – A Fundamental Trade-off 2687.3.2 Spatial Random Access by Tracking Coding Dependencies 2717.3.3 Spatial Random Access by Frame Tiling 2717.3.4 Multi-Resolution Tiling 2737.3.5 Overlapping Tiling for Low-Powered Devices 2747.4 Models for Adaptive Tile-based Representation and Delivery 2767.4.1 Saliency-based Adaptive Coding of Tiled Content 2777.4.2 Optimisation of Tile Selection Under Delay and Bandwidth Constraints 2807.5 Segment-based Adaptive Transport 2817.5.1 Video Streaming Over IP 2827.5.2 Tiled HTTP Adaptive Streaming Over the Internet 2857.5.3 Publish/Subscribe System for Interactive Video Streaming 2897.6 Conclusion 294References 2948 Interactive Rendering 298Javier Ruiz-Hidalgo, Malte Borsum, Axel Kochale and Goranka Zori´c8.1 Introduction 2988.2 Format-Agnostic Rendering 2998.2.1 Available Rendering Solutions in End Terminals 2998.2.2 Requirements for Format-Agnostic Video Rendering 3068.2.3 Description of a Technical Solution 3088.3 Device-less Interaction for Rendering Control 3118.3.1 Sensors for Gesture Recognition 3148.3.2 Gesture Analysis Techniques 3178.3.3 Recognition and Classification Techniques 3198.3.4 Remaining Challenges in Gesture Recognition Systems 3218.3.5 Description of a Technical Solution 3218.3.6 User Study of the Gesture Interface 3278.4 Conclusions 331References 3329 Application Scenarios and Deployment Domains 337Omar Aziz Niamut, Arvid Engstr¨om, Axel Kochale, Jean-Franc¸ois Macq, Graham Thomas and Goranka Zori´c9.1 Introduction 3379.2 Application Scenarios 3389.2.1 Digital Cinema: A Totally Immersive Experience 3389.2.2 Home Theatre: In Control 3399.2.3 Mobile: Navigation and Magnification 3399.3 Deployment in the Production Domain 3409.3.1 Outlook on New Production Practices 3409.3.2 Use of Format-Agnostic Technology to Aid Conventional Production 3429.3.3 Production of Format-Agnostic Content to Support End User Interaction 3449.4 Deployment in the Network Domain 3479.4.1 Network Requirements 3479.4.2 Impact of Application Scenarios 3489.5 Deployment in the Device Domain 3519.5.1 Device Capabilities 3519.5.2 User and Producer Expectations 3559.6 Deployment in the User Domain 3569.7 Conclusion 357References 357Index 359