Advances in Computer Vision and Machine Intelligence – serie

It has long been a dream to realize machines with flexible visual perception capability. Research on digital image processing by computers was initiated about 30 years ago, and since then a wide variety of image processing algorithms have been devised. Using such image processing algorithms and advanced hardware technologies, many practical ma­ chines with visual recognition capability have been implemented and are used in various fields: optical character readers and design chart readers in offices, position-sensing and inspection systems in factories, computer tomography and medical X-ray and microscope examination systems in hospitals, and so on. Although these machines are useful for specific tasks, their capabilities are limited. That is, they can analyze only simple images which are recorded under very carefully adjusted photographic conditions: objects to be recognized are isolated against a uniform background and under well-controlled artificial lighting. In the late 1970s, many image understanding systems were de­ veloped to study the automatic interpretation of complex natural scenes. They introduced artificial intelligence techniques to represent the knowl­ edge about scenes and to realize flexible control structures. The first author developed an automatic aerial photograph interpretation system based on the blackboard model (Naga1980). Although these systems could analyze fairly complex scenes, their capabilities were still limited; the types of recognizable objects were limited and various recognition vii viii Preface errors occurred due to noise and the imperfection of segmentation algorithms.

This book gathers together information concerning the interaction of hu­ man stereopsis with various stereoscopic viewing devices, especially those used in teleoperator systems. The book is not concerned with machine vi­ sion systems. In these systems, data analogous to human binocular visual information is gathered and analyzed by some device for use in decision making or control, often without the intervention of a human. This subject presents problems of considerable complexity; it has generated many inge­ nious solutions and has been the inspiration of much work of fundamental importance. But the problems are quite different from those encountered in the design of systems intended to exploit human stereopsis, and there is surprisingly little cross-fertilization between the two fields. 1. 1. SCOPE AND STRUCTURE OF THIS BOOK The book surveys the known properties of the human unaided binocu­ lar system, and where possible gives the magnitude of parameters that are of use in designing technical systems involving a human operator. Chapter 2 summarizes the human stereoscopic vision literature including the depth distortions of unaided stereoscopic viewing. Chapter 3 describes a variety of 3-D image viewing techniques and deals with the performance limits of human stereopsis assisted by simple stereoscopic viewing devices. Chapter 4 extends this treatment to television binocular viewing devices, and shows 1 2 Chapter 1 that the nature of the depth distortion is changed. Chapter 5 analyzes the geometry of single camera stereoscopic systems, and discusses the advan­ tages and disadvantages of such systems.

Image motion processing is important to machine vision systems because it can lead to the recovery of 3D structure and motion. Author Amar Mitiche offers a comprehensive mathematical treatment of this key subject in visual systems research. Mitiche examines the interpretation of point correspondences as well as the interpretation of straight line correspondences and optical flow. In addition, the author considers interpretation by knowledge-based systems and presents the relevant mathematical basis for 3D interpretation.

Figure 1.1. An outdoor scene "A bus is passing three cars which are parking between trees at the side of the road. Houses having two storeys are lined up at the street. 3 4 Introduction Figure 1.2. An assembly scene There seems to be a small open place between the group of houses in the foreground and the store in the background". In such or a similar way the content of the natural scene shown above can be described. It is quite easy to give such a short description. The problem is somewhat more complex for the second image. First of all, it can be stated that the image does not show an everyday scene. It appears as a kind of man made surrounding. But everyone can accept the following statements about this image: 1. The image shows a snapshot of an assembly line. 2. The robot in front is screwing. 3. There is no person in the working area of the robots. 4. All objects on the conveyor belt are worked on by robots. There are no free objects on the belt.

In this groundbreaking new volume, computer researchers discuss the development of technologies and specific systems that can interpret data with respect to domain knowledge. Although the chapters each illuminate different aspects of image interpretation, all utilize a common approach - one that asserts such interpretation must involve perceptual learning in terms of automated knowledge acquisition and application, as well as feedback and consistency checks between encoding, feature extraction, and the known knowledge structures in a given application domain. The text is profusely illustrated with numerous figures and tables to reinforce the concepts discussed.

This comprehensive reference introduces the intricate science of implementing image modelling and retrieval based on the shape, on the colour and texture, or on spatial arrangement of imaged objects.

DEFECT PROPORTION OF DETECTION INITIAL RATE DETECTION RATE INSPECTOR 3 COMPLEXITY OF TIMES PAN OF PERFORMING o~ ________________________ o~ ______________________ __ -;. INSPECTION TASK -;. VISUAL INSPECTION Fagure 1. Trends in relations between the complexity of inspection tasks, defect detection rates (absolute and relative), and inspection time. Irrespective of the necessities described above, and with the excep­ tion of specific generic application systems (e.g., bare-board PCB inspection, wafer inspection, solder joint inspection, linewidth measure­ ment), vision systems are still not found frequently in today's electronics factories. Besides cost, some major reasons for this absence are: 1. The detection robustness or accuracy is still insufficient. 2. The total inspection time is often too high, although this can frequently be attributed to mechanical handling or sensing. 3. There are persistent gaps among process engineers, CAD en­ gineers, manufacturing engineers, test specialists, and computer vision specialists, as problems dominate the day-to-day interac­ tions and prevent the establishment of trust. 4. Computer vision specialists sometimes still believe that their contributions are universal, so that adaptation to each real problem becomes tedious, or stumbles over the insufficient availabIlity of multidisciplinary expertise. Whether we like it or not, we must still use appropriate sensors, lighting, and combina­ tions of algorithms for each class of applications; likewise, we cannot design mechanical handling, illumination, and sensing in isolation from each other.

Computer vision deals with the problem of manipulating information contained in large quantities of sensory data, where raw data emerge from the transducing 6 7 sensors at rates between 10 to 10 pixels per second. Conventional general­ purpose computers are unable to achieve the computation rates required to op­ erate in real time or even in near real time, so massively parallel systems have been used since their conception in this important practical application area. The development of massively parallel computers was initially character­ ized by efforts to reach a speedup factor equal to the number of processing elements (linear scaling assumption). This behavior pattern can nearly be achieved only when there is a perfect match between the computational struc­ ture or data structure and the system architecture. The theory of hierarchical modular systems (HMSs) has shown that even a small number of hierarchical levels can sizably increase the effectiveness of very large systems. In fact, in the last decade several hierarchical architectures that support capabilities which can overcome performances gained with the assumption of linear scaling have been proposed. Of these architectures, the most commonly considered in com­ puter vision is the one based on a very large number of processing elements (PEs) embedded in a pyramidal structure. Pyramidal architectures supply the same image at different resolution lev­ els, thus ensuring the use of the most appropriate resolution for the operation, task, and image at hand.

With the recent advances in multimedia technology, on-line libraries of digital images are assuming an ever increasing relevance within a wide range of information systems. Effective access to such archives requires on external textual keywords that conventional searching techniques based are complemented by content-based queries addressing appearing visual features of searched data. Central to this retrieval approach is the creation of models, which permit to abstract images into some space of features and support indexing and comparison of visual contents. Depending on the specific characteris­ tics of the images at hand, such models can rely on different facets of the informative contents of visual data: color and texture distribution, shape of appearing objects, spatial arrangement. This book introduces and exemplifies objectives and research themes in image modeling and retrieval. In the introductory chapter, the problem of image modeling and retrieval is motivated and discussed, and major entry-pointers to the literature are provided. Afterwards, different model­ ing approaches are addressed in six chapters contributed by major research groups in the field: modeling based on object shape is addressed in chapter 2 by F. Korn, N. Sidiropoulos, C. Faloutsos, E. Siegel, and Z. Protopapas, and in chapter 3 by R. Mehrotra and J. E. Gary; modeling based on color and texture distribution is addressed in chapter 4 by G. D. Finlayson, S. S. Chat­ terjee, and B. V. Funt, and in chapter 5 by I. Gagliardi, A.

Image motion processing is important to machine vision systems because it can lead to the recovery of 3D structure and motion. Author Amar Mitiche offers a comprehensive mathematical treatment of this key subject in visual systems research. Mitiche examines the interpretation of point correspondences as well as the interpretation of straight line correspondences and optical flow. In addition, the author considers interpretation by knowledge-based systems and presents the relevant mathematical basis for 3D interpretation.

It has long been a dream to realize machines with flexible visual perception capability. Research on digital image processing by computers was initiated about 30 years ago, and since then a wide variety of image processing algorithms have been devised. Using such image processing algorithms and advanced hardware technologies, many practical ma­ chines with visual recognition capability have been implemented and are used in various fields: optical character readers and design chart readers in offices, position-sensing and inspection systems in factories, computer tomography and medical X-ray and microscope examination systems in hospitals, and so on. Although these machines are useful for specific tasks, their capabilities are limited. That is, they can analyze only simple images which are recorded under very carefully adjusted photographic conditions: objects to be recognized are isolated against a uniform background and under well-controlled artificial lighting. In the late 1970s, many image understanding systems were de­ veloped to study the automatic interpretation of complex natural scenes. They introduced artificial intelligence techniques to represent the knowl­ edge about scenes and to realize flexible control structures. The first author developed an automatic aerial photograph interpretation system based on the blackboard model (Naga1980). Although these systems could analyze fairly complex scenes, their capabilities were still limited; the types of recognizable objects were limited and various recognition vii viii Preface errors occurred due to noise and the imperfection of segmentation algorithms.

This book gathers together information concerning the interaction of hu­ man stereopsis with various stereoscopic viewing devices, especially those used in teleoperator systems. The book is not concerned with machine vi­ sion systems. In these systems, data analogous to human binocular visual information is gathered and analyzed by some device for use in decision making or control, often without the intervention of a human. This subject presents problems of considerable complexity; it has generated many inge­ nious solutions and has been the inspiration of much work of fundamental importance. But the problems are quite different from those encountered in the design of systems intended to exploit human stereopsis, and there is surprisingly little cross-fertilization between the two fields. 1. 1. SCOPE AND STRUCTURE OF THIS BOOK The book surveys the known properties of the human unaided binocu­ lar system, and where possible gives the magnitude of parameters that are of use in designing technical systems involving a human operator. Chapter 2 summarizes the human stereoscopic vision literature including the depth distortions of unaided stereoscopic viewing. Chapter 3 describes a variety of 3-D image viewing techniques and deals with the performance limits of human stereopsis assisted by simple stereoscopic viewing devices. Chapter 4 extends this treatment to television binocular viewing devices, and shows 1 2 Chapter 1 that the nature of the depth distortion is changed. Chapter 5 analyzes the geometry of single camera stereoscopic systems, and discusses the advan­ tages and disadvantages of such systems.

In this groundbreaking new volume, computer researchers discuss the development of technologies and specific systems that can interpret data with respect to domain knowledge. Although the chapters each illuminate different aspects of image interpretation, all utilize a common approach - one that asserts such interpretation must involve perceptual learning in terms of automated knowledge acquisition and application, as well as feedback and consistency checks between encoding, feature extraction, and the known knowledge structures in a given application domain. The text is profusely illustrated with numerous figures and tables to reinforce the concepts discussed.