Javaan Singh Chahl, Lakhmi C Jain, Akiko Mizutani and Mika Sato-Ilic (Eds.) Innovations in Intelligent Machines - Studies in Computational Intelligence, Volume 70 Editor-in-chief Prof Janusz Kacprzyk Systems Research Institute Polish Academy of Sciences ul Newelska 01-447 Warsaw Poland E-mail: kacprzyk@ibspan.waw.pl Further volumes of this series can be found on our homepage: springer.com Vol 60 Vladimir G Ivancevic and Tijana T Ivacevic Computational Mind: A Complex Dynamics Perspective, 2007 ISBN 978-3-540-71465-1 Vol 49 Keshav P Dahal, Kay Chen Tan, Peter I Cowling (Eds.) Evolutionary Scheduling, 2007 ISBN 978-3-540-48582-7 Vol 61 Jacques Teller, John R Lee and Catherine Roussey (Eds.) Ontologies for Urban Development, 2007 ISBN 978-3-540-71975-5 Vol 50 Nadia Nedjah, Leandro dos Santos Coelho, Luiza de Macedo Mourelle (Eds.) Mobile Robots: The Evolutionary Approach, 2007 ISBN 978-3-540-49719-6 Vol 62 Lakhmi C Jain, Raymond A Tedman and Debra K Tedman (Eds.) Evolution of Teaching and Learning Paradigms in Intelligent Environment, 2007 ISBN 978-3-540-71973-1 Vol 51 Shengxiang Yang, Yew Soon Ong, Yaochu Jin Honda (Eds.) Evolutionary Computation in Dynamic and Uncertain Environment, 2007 ISBN 978-3-540-49772-1 Vol 63 Wlodzislaw Duch and Jacek Ma´ dziuk (Eds.) n Challenges for Computational Intelligence, 2007 ISBN 978-3-540-71983-0 Vol 52 Abraham Kandel, Horst Bunke, Mark Last (Eds.) Applied Graph Theory in Computer Vision and Pattern Recognition, 2007 ISBN 978-3-540-68019-2 Vol 64 Lorenzo Magnani and Ping Li (Eds.) Model-Based Reasoning in Science, Technology, and Medicine, 2007 ISBN 978-3-540-71985-4 Vol 53 Huajin Tang, Kay Chen Tan, Zhang Yi Neural Networks: Computational Models and Applications, 2007 ISBN 978-3-540-69225-6 Vol 65 S Vaidya, L C Jain and H Yoshida (Eds.) Advanced Computational Intelligence Paradigms in Healthcare-2, 2007 ISBN 978-3-540-72374-5 Vol 54 Fernando G Lobo, Cl´ udio F Lima a and Zbigniew Michalewicz (Eds.) Parameter Setting in Evolutionary Algorithms, 2007 ISBN 978-3-540-69431-1 Vol 66 Lakhmi C Jain, Vasile Palade and Dipti Srinivasan (Eds.) Advances in Evolutionary Computing for System Design, 2007 ISBN 978-3-540-72376-9 Vol 55 Xianyi Zeng, Yi Li, Da Ruan and Ludovic Koehl (Eds.) Computational Textile, 2007 ISBN 978-3-540-70656-4 Vol 56 Akira Namatame, Satoshi Kurihara and Hideyuki Nakashima (Eds.) Emergent Intelligence of Networked Agents, 2007 ISBN 978-3-540-71073-8 Vol 57 Nadia Nedjah, Ajith Abraham and Luiza de Macedo Mourella (Eds.) Computational Intelligence in Information Assurance and Security, 2007 ISBN 978-3-540-71077-6 Vol 58 Jeng-Shyang Pan, Hsiang-Cheh Huang, Lakhmi C Jain and Wai-Chi Fang (Eds.) Intelligent Multimedia Data Hiding, 2007 ISBN 978-3-540-71168-1 Vol 59 Andrzej P Wierzbicki and Yoshiteru Nakamori (Eds.) Creative Environments, 2007 ISBN 978-3-540-71466-8 Vol 67 Vassilis G Kaburlasos and Gerhard X Ritter (Eds.) Computational Intelligence Based on Lattice Theory, 2007 ISBN 978-3-540-72686-9 Vol 68 Cipriano Galindo, Juan-Antonio Fern´ ndez-Madrigal and Javier Gonzalez a A Multi-Hierarchical Symbolic Model of the Environment for Improving Mobile Robot Operation, 2007 ISBN 978-3-540-72688-3 Vol 69 Falko Dressler and Iacopo Carreras (Eds.) Advances in Biologically Inspired Information Systems: Models, Methods, and Tools, 2007 ISBN 978-3-540-72692-0 Vol 70 Javaan Singh Chahl, Lakhmi C Jain, Akiko Mizutani and Mika Sato-Ilic (Eds.) Innovations in Intelligent Machines-1, 2007 ISBN 978-3-540-72695-1 Javaan Singh Chahl Lakhmi C Jain Akiko Mizutani Mika Sato-Ilic (Eds.) Innovations in Intelligent Machines - With 146 Figures and 10 Tables Dr Javaan Singh Chahl Dr Akiko Mizutani Defence Science and Technology Organisation Edinburgh South Australia Australia Odonatrix Pty Ltd Adelaide South Australia Australia Prof Lakhmi C Jain Prof Mika Sato-Ilic University of South Australia Mawson Lakes Campus Adelaide, South Australia Australia E-mail:- Lakhmi.jain@unisa.edu.au Faculty of Systems and Information Engineering University of Tsukuba Japan Library of Congress Control Number: 2007927247 ISSN print edition: 1860-949X ISSN electronic edition: 1860-9503 ISBN 978-3-540-72695-1 Springer Berlin Heidelberg New York This work is subject to copyright All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilm or in any other way, and storage in data banks Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer-Verlag Violations are liable to prosecution under the German Copyright Law Springer is a part of Springer Science+Business Media springer.com c Springer-Verlag Berlin Heidelberg 2007 The use of general descriptive names, registered names, trademarks, etc in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use Cover design: deblik, Berlin A Typesetting by the SPi using a Springer LTEX macro package Printed on acid-free paper SPIN: 11588450 89/SPi 543210 Foreword Innovations in Intelligent Machines is a very timely volume that takes a fresh look on the recent attempts of instilling human-like intelligence into computer-controlled devices By contrast to the machine intelligence research of the last two decades, the recent work in this area recognises explicitly the fact that human intelligence is not purely computational but that it also has an element of empirical validation (interaction with the environment) Also, recent research recognises that human intelligence does not always prevent one from making errors but it equips one with the ability to learn from mistakes The latter is the basic premise for the development of the collaborative (swarm) intelligence that demonstrates the value of the virtual experience pool assembled from cases of successful and unsuccessful execution of a particular algorithm The editors are to be complemented for their vision of designing a framework within which they ask some fundamental questions about the nature of intelligence in general and intelligent machines in particular and illustrate answers to these questions with specific practical system implementations in the consecutive chapters of the book Chapter addresses the cost effectiveness of “delegating” operator’s intelligence to on-board computers so as to achieve single operator control of multiple unmanned aerial vehicles (UAV) The perspective of cost effectiveness allows one to appreciate the distinction between the optimal (algorithmic) and the intelligent (non-algorithmic, empirical) decision-making, which necessarily implies some costs In this context the decision to use or not to use additional human operators can be seen as the assessment of the “value” of the human intelligence in performing a specific task The challenge of the development of collaborative (swarm) intelligence and its specific application to UAV path planning over the terrain with complex topology is addressed in Chapters and The authors of these chapters propose different technical solutions based on the application of game theory, negotiation techniques and neural networks but they reach the same conclusions that the cooperative behaviour of individual UAVs, exchanging VI Foreword information about their successes and failures, underpins the development of human-like intelligence This insight is further developed in Chapter where the authors look at the evolution-based dynamic path planning Chapter emphasises the importance of physical constraints on the UAVs in accomplishing a specific task To re-phrase it in slightly more general terms, it highlights the fact that algorithmic information processing may be numerically correct but it may not be physically very meaningful if the laws of physics are not taken fully into account This is exactly where the importance of empirical verification comes to fore in intelligent decision-making The practice of processing uncertain information at various levels of abstraction (granulation) is now well recognised as a characteristic feature of human information processing By discussing the state estimation of UAVs based on information provided by low fidelity sensors, Chapter provides a reference material for dealing with uncertain data Discussion of the continuousdiscrete extended Kalman filter placed in the context of intelligent machines underlines the importance of information abstraction (granulation) Chapters and share a theme of enhancement of sensory perception of intelligent machines Given that the interaction with the environment is a key component of intelligent machines, the development of sensors providing omni directional vision is a promising way to achieving enhanced levels of intelligence Also the ability to achieve, through appropriate sensor design, long distance (low accuracy) and short distance (high accuracy) vision correlates closely with the multi-resolution (granular) information processing by humans The book is an excellent compilation of leading-edge contributions in the area of intelligent machines and it is likely to be on the essential reading list of those who are keen to combine theoretical insights with practical applications Andrzej Bargiela Professor of Computer Science University of Nottingham, UK Preface Advanced computational techniques for decision making on unmanned systems are starting to be factored into major policy directives such as the United States Department of Defence UAS Roadmap Despite the expressed need for the elusive characteristic of “autonomy”, there are no existing systems that are autonomous by any rigorous definition Through the use of sophisticated algorithms, residing in every software subsystem (state estimation, navigation, control and so on) it is conceivable that a degree of true autonomy might emerge The science required to achieve robust behavioural modules for autonomous systems is sampled in this book There are a host of technologies that could be implemented on current operational systems Many of the behaviours described are present in fielded systems albeit in an extremely primitive form For example, waypoint navigation as opposed to path planning, so the prospects of upgrading current implementations are good if hurdles such as airworthiness can be overcome We can confidently predict that within a few years the types of behaviour described herein will be commonplace on both large and small unmanned systems This research book includes a collection of chapters on the state of art in the area of intelligent machines We believe that this research will provide a sound basis to make autonomous systems human-like We are grateful to the authors and reviewers for their vision and contribution The editorial assistance provided by Springer-Verlag is acknowledged Editors Contents Foreword V Preface VII Intelligent Machines: An Introduction Lakhmi C Jain, Anas Quteishat, and Chee Peng Lim Introduction Learning in Intelligent Machines Application of Intelligent Machines 3.1 Unmanned Aerial Vehicle (UAV) 3.2 Underwater Robot 3.3 Space Vehicle 3.4 Humanoid Robot 3.5 Other Attempts in Intelligent Machines Chapters Included in this Book Summary References 1 3 4 7 Predicting Operator Capacity for Supervisory Control of Multiple UAVs M.L Cummings, Carl E Nehme, Jacob Crandall, and Paul Mitchell Introduction Previous Experimental Multiple UAV studies Predicting Operator Capacity through Temporal Constraints 3.1 Wait Times 3.2 Experimental Analysis of the Fan-out Equations 3.3 Linking Fan-out to Operator Performance 3.4 The Overall Cost Function 3.5 The Human Model 3.6 Optimization through Simulated Annealing 3.7 Results of Simulation 11 11 12 14 15 16 24 25 27 28 29 X Contents Meta-Analysis of the Experimental and Modeling Prediction methods 33 Conclusions 36 References 36 Team, Game, and Negotiation based Intelligent Autonomous UAV Task Allocation for Wide Area Applications P.B Sujit, A Sinha, and D Ghose Introduction Existing Literature Task Allocation Using Team Theory 3.1 Basics of Team Theory 3.2 Problem Formulation 3.3 Team Theoretic Solution 3.4 Simulation Results Task Allocation using Negotiation 4.1 Problem Formulation 4.2 Decision-making 4.3 Simulation Results Search using Game Theoretic Strategies 5.1 N-person Game Model 5.2 Solution Concepts 5.3 Simulation Results Conclusions References 39 39 41 42 42 43 45 47 50 50 53 58 61 62 63 69 72 72 UAV Path Planning Using Evolutionary Algorithms Ioannis K Nikolos, Eleftherios S Zografos, and Athina N Brintaki Introduction 1.1 Basic Definitions 1.2 Cooperative Robotics 1.3 Path Planning for Single and Multiple UAVs 1.4 Outline of the Current Work B-Spline and Evolutionary Algorithms Fundamentals 2.1 B-Spline Curves 2.2 Fundamentals of Evolutionary Algorithms (EAs) 2.3 The Solid Boundary Representation Off-line Path Planner for a Single UAV Coordinated UAV Path Planning 4.1 Constraints and Objectives 4.2 Path Modeling Using B-Spline Curves 4.3 Objective Function Formulation The Optimization Procedure 5.1 Differential Evolution Algorithm 5.2 Radial Basis Function Network for DE Assistance 77 77 77 79 80 85 86 86 88 89 90 92 92 93 94 97 97 99 256 J Gaspar et al − − − − − − − Fig 16 A real world experiment combining Visual Path Following for door traversal and Topological Navigation for long-distance goals Odometry results before (a) and after (b) the addition of ground truth measurements robot heading is easily specified by clicking on the desired direction of travel in the panoramic image, and the desired (x, y) locations are specified by clicking in the bird’s-eye view Using 3D models further improves the visualization of the scene A unique feature of such a representation is that the user can tell the robot to arrive to a given destination at a certain orientation simply by rotating the 3D model Beyond the benefits of immersion, it allows to group the information of many views and get a global view of the environment In order to build the 3D scene models, we propose Interactive Scene Reconstruction, a method based on the complimentary nature of Human and Robot perceptions While Humans have an immediate qualitative understanding of the scene encompassing co-planarity and co-linearity properties of a number of points of the scene, Robots equipped with omnidirectional cameras can take precise azimuthal and elevation measurements Interactive scene reconstruction has recently drawn lots of attention Debevec et al in [22], propose an interactive scene reconstruction approach for modelling and rendering architectural scenes They derive a geometric model combining edge lines observed in the images with geometrical properties known a priori This approach is advantageous relative to building a CAD model from scratch, as some information comes directly from the images In addition, it is simpler than a conventional structure from motion problem because, instead of reconstructing points, it deals with reconstructing scene parameters, which is a much lower dimension and better conditioned problem Toward Robot Perception through Omnidirectional Vision 257 In [79] Sturm uses an omnidirectional camera based on a parabolic mirror and a telecentric lens for reconstructing a 3D scene The user specifies relevant points and planes grouping those points The directions of the planes are computed e.g from vanishing points, and the image points are back-projected to obtain parametric representations where the points move on the 3D projection rays The points and the planes, i.e their distances to the viewer, are simultaneously reconstructed by minimizing a cost functional based on the distances from the points to the planes We build 3D models using omnidirectional images and some limited user input, as in Sturm’s work However our approach is based on a different reconstruction method and the omnidirectional camera is a generalised single projection centre camera modelled by the Unified Projection Model [37] The reconstruction method is that proposed by Grossmann for conventional cameras [43], applied to single projection centre omnidirectional cameras for which a back-projection model was obtained The back-projection transforms the omnidirectional camera to a (very wide field of view) pin-hole camera The user input is of geometrical nature, namely alignment and coplanarity properties of points and lines After backprojection, the data is arranged according to the geometrical constraints, resulting in a linear problem whose solution can be found in a single step 4.1 Interactive Scene Reconstruction We now present the method for interactively building a 3D model of the environment The 3D information is obtained from co-linearity and co-planarity properties of the scene The texture is then extracted from the images to obtain a realistic virtual environment The 3D model is a Euclidean reconstruction of the scene As such, it may be translated and rotated for visualization and many models can be joined into a single representation of the environment As in other methods [50, 79], the reconstruction algorithm presented here works in structured environments, in which three orthogonal directions, “x”, “y” and “z” shape the scene The operator specifies in an image the location of 3D points of interest and indicates properties of alignment and planarity In this section, we present a method based on [42] In all, the information specified by the operator consists of: – Image points corresponding to 3D points that will be reconstructed, usually on edges of the floor and of walls – Indications of “x−”, “y−” and “z =constant” planes as and of alignments of points along the x, y and z directions This typically includes the floor and vertical walls – Indications of points that form 3D surfaces that should be visualized as such 258 J Gaspar et al The remainder of this section shows how to obtain a 3D reconstruction from this information Using Back-projection to form Perspective Images In this section, we derive a transformation, applicable to single projection centre omnidirectional cameras that obtain images as if acquired by perspective projection cameras This is interesting as it provides a way to utilize methodologies for perspective cameras directly with omnidirectional cameras In particular, the interactive scene reconstruction method (described in the following sections) follows this approach of using omnidirectional cameras transformed to perspective cameras The acquisition of correct perspective images, independent of the scenario, requires that the vision sensor be characterised by a single projection centre [2] The unified projection model has, by definition, this property but, due to the intermediate mapping over the sphere, the obtained images are in general not perspective In order to obtain correct perspective images, the spherical projection must be first reversed from the image plane to the sphere surface and then, re-projected to the desired plane from the sphere centre We term this reverse projection back-projection The back-projection of an image pixel (u, v), obtained through spherical projection, yields a 3D direction k · (x, y, z) given by the following equations derived from Eq (1): a = (l + m), b = (u2 + v ) la − sign(a) a2 + (1 − l2 )b u x = y v a2 + b (25) z = ± − x2 − y √ where z is negative if |a| /l > b, and positive otherwise It is assumed, without loss of generality, that (x, y, z) is lying on the surface of the unit sphere Figure 17 illustrates the back-projection Given an omnidirectional image we use back-projection to map image points to the surface of a sphere centred at the camera viewpoint10 At this point, it is worth noting that the set M = {P : P = (x, y, z)} interpreted as points of the projective plane, already define a perspective image By rotating and scaling the set M one obtains specific viewing directions and 10 The omnidirectional camera utilized here is based on a spherical mirror and therefore does not have a single projection centre However, as the scene depth is large as compared to the sensor size, the sensor approximates a single projection centre system (details in [33]) Hence it is possible to find the parameters of the corresponding unified projection model system and use Eq (25) Toward Robot Perception through Omnidirectional Vision 259 Fig 17 (Top) original omnidirectional image and back-projection to a spherical surface centred at the camera viewpoint (Below) Examples of perspective images obtained from the omnidirectional image focal lengths Denoting the transformation of coordinates from the omnidirectional camera to a desired (rotated) perspective camera by R then the new perspective image {p : p = (u, v, 1)} becomes: p = λKRP (26) where K contains intrinsic parameters and λ is a scaling factor This is the pin-hole camera projection model [25], when the origin of the coordinates is the camera centre Figure 17 shows some examples of perspective images obtained from the omnidirectional image The perspective images illustrate the selection of the viewing direction Aligning the Data with the Reference Frame In the reconstruction algorithm we use the normalised perspective projection model [25], by choosing K = I3×3 in Eqs (25) and (26): p = λRP (27) in which p = [u v 1]T is the image point, in homogeneous coordinates and P = [x y z]T is the 3D point The rotation matrix R is chosen to align the camera frame with the reference (world) frame Since the z axis is vertical, the matrix R takes the form: ⎡ ⎤ cos(θ) sin(θ) R = ⎣ − sin(θ) cos(θ) ⎦ , (28) 0 260 J Gaspar et al where θ is the angle formed by the x axis of the camera and that of the world coordinate system This angle will be determined from the vanishing points [14] of these directions A vanishing point is the intersection in the image of the projections of parallel 3D lines If one has the images of two or more lines parallel to a given 3D direction, it is possible to determine its vanishing point [79] In our case, information provided by the operator allows for the determination of alignments of points along the x and y directions It is thus possible to compute the vanishing points of these directions and, from there, the angle θ between the camera and world coordinate systems Reconstruction Algorithm Having determined the projection matrix R in Eq (27), we proceed to estimate the position of the 3D points P This will be done by using the image points p to linearly constrain the unknown quantities From the projection equation, one has p × RP = 03 , which is equivalently written (29) Sp RP = 03 , where Sp is the Rodrigues matrix associated with the cross product with vector p Writing this equation for each of the N unknown 3D points gives the linear system: ⎤ ⎤⎡ ⎡ P1 Sp1 R ⎥ ⎢ P2 ⎥ ⎢ Sp2 R ⎥ ⎥⎢ ⎢ (30) ⎥ ⎢ ⎥ = A.P = 03N ⎢ ⎦⎣ ⎦ ⎣ SpN R PN where A is block diagonal and P contains the 3N coordinates that we wish to estimate: Since only two equations from the set defined by Eq (29) are independent, the co-rank of A is equal to the number of points N The indeterminacy in this system of equations corresponds to the unknown depth at which each points lies, relatively to the camera This indeterminacy is removed by the planarity and alignment information given by the operator For example, when two points belong to a z = constant plane, their z coordinates are necessarily equal and there is thus a single unknown quantity, rather than two Equation (30) is modified to take this information into account by replacing the columns of A (resp rows of P) corresponding to the two unknown z coordinates by a single column (resp row) that is the sum of the two Alignment information likewise states the equality of two pairs of unknowns Each item of geometric information provided by the user is used to transform the linear system in Equation (30) into a smaller system involving only distinct quantities: Toward Robot Perception through Omnidirectional Vision A P = 03N 261 (31) This system is solved in the total least-squares [39] sense by assigning to P the singular vector of A corresponding to the smallest singular value The original vector of coordinates P is obtained from P by performing the inverse of the operations that led from Eq (30) to Eq (31) The reconstruction algorithm is easily extended to the case of multiple cameras The orientation of the cameras is estimated from vanishing points as above and the projection model becomes: p = λ(RP − Rt) (32) where t is the position of the camera It is zero for the first camera and is one of t1 tj if j additional cameras are present Considering for example that there are two additional cameras and following the same procedure as for a single image, similar A and P are defined for each camera The problem has six new degrees of freedom corresponding to the two unknown translations t1 and t2 : ⎡ ⎤ ⎡ ⎤ P1 ⎢ P2 ⎥ A1 ⎢ ⎥ ⎣ ⎦ ⎢ P3 ⎥ = A2 −A2 12 (33) ⎢ ⎥ A3 −A3 13 ⎣ t1 ⎦ t2 where 12 and 13 are matrices to stack the blocks of A2 and A3 As before co-linearity and co-planarity information is used to obtain a reduced system Note that columns corresponding to different images may be combined, for example if a 3D point is tracked or if a line or plane spans multiple images The reduced system is solved in the total least-squares sense and the 3D points P are retrieved as in the single-view case The detailed reconstruction method is given in [42] Results Our reconstruction method provides estimates of 3D points in the scene In order to visualise these estimates, facets are added to connect some of the 3D points, as indicated by the user Texture is extracted from the omnidirectional images and a complete textured 3D model is obtained Figure 18 shows an omnidirectional image and the superposed user input This input consists of the 16 points shown, knowledge that sets of points belong to constant x, y or z planes and that other sets belong to lines parallel to the x, y or z axes The table on the side of the images shows all the user-defined data Planes orthogonal to the x and y axes are in light gray and white respectively, and one horizontal plane is shown in dark gray (the topmost horizontal plane is not shown as it would occlude the other planes) 262 J Gaspar et al Fig 18 Interactive modelling based on co-planarity and co-linearity properties using a single omnidirectional image (Top) Original image with superposed points and lines localised by the user Planes orthogonal to the x, y and z axis are shown in light gray, white, and dark gray respectively (Table) The numbers are the indexes shown on the image (Below) Reconstruction result and view of the textured mapped 3D model Figure 18 shows the resulting texture-mapped reconstruction This result shows the effectiveness of omnidirectional imaging to visualize the immediate vicinity of the sensor It is interesting to note that just a few omnidirectional images are sufficient for building the 3D model (the example shown utilized a single image), as opposed to a larger number of “normal” images that would be required to reconstruct the same scene [50, 79] 4.2 Human Robot Interface based on 3D World Models Now that we have the 3D scene model, we can build the Human Robot interface In addition to the local headings or poses, the 3D model allows us to specify complete missions The human operator selects the start and end locations in the model, and can indicate points of interest for the robot to undertake specific tasks See Fig 19 Given that the targets are specified on interactive models, i.e models built and used on the user side, they need to be translated as tasks that the robot understands The translation depends on the local world models and navigation sequences the robot has in its database Most of the world that the robot knows is in the form of a topological map In this case the targets are images that the robot has in its image database The images used to build Toward Robot Perception through Omnidirectional Vision 263 Fig 19 Tele-operation interface based on 3D models: (top) tele-operator view, (middle) robot view and (bottom) world view the interactive model are nodes of the topological map Thus, a fraction of a distance on an interactive model is translated as the same fraction on a link of the topological map At some points there are precise navigation requirements Many of these points are identified in the topological map and will be invoked automatically when travelling between nodes Therefore, many of the Visual Path Following tasks performed not need to be explicitly defined by the user However, should the user desires, he may add new Visual Path Following tasks In that case, the user chooses landmarks, navigates in the interactive model and then asks the robot to follow the same trajectory Interactive modelling offers a simple procedure for building a 3D model of the scene where a vehicle may operate Even though the models not contain very fine details, they can provide the remote user of the robot with a sufficiently rich description of the environment The user can instruct the robot to move to desired position, simply by manipulating the model to reach the desired view point Such simple scene models can be transmitted even with low bandwidth connections Conclusion The challenge of developing perception as a key competence of vision-based mobile robots is of fundamental importance to their successful application in the real world Vision provides information on world structure and compares favourably with other sensors due to the large amount of rich data available 264 J Gaspar et al In terms of perception, omnidirectional vision has the additional advantage of providing output views (images) with simple geometries Our sensors output Panoramic and Bird’s Eye views that are images as obtained by cylindrical retinas or pin-hole cameras imaging the ground plane Panoramic and Bird’s Eye views are useful for navigation, namely for servoing tasks, as they make localisation a simple 2D rigid transformation estimation problem Successful completion of the door crossing experiment, for example, relied on the tracking of features surrounding the sensor Such experiments are not possible with limited field of view (conventional) cameras Even cameras equipped with panand-tilt mounting would be unable to perform the many separate landmark trackings shown in our experiments Designing navigation modalities for the task at hand is easier and more effective when compared to designing a single complex navigation mode [8] Therefore, in this work, emphasis was placed on building appropriate representations rather than always relying upon highly accurate information about the environment The decision to use this representation was partly inspired by the way in which humans and animals model spatial knowledge Our combined navigation modalities, Visual Path Following and Topological Navigation, constituted an effective approach to tasks containing both short paths to follow with high precision and long paths to follow qualitatively Interactive Scene Reconstruction was shown to be an effective method of obtaining 3D scene models, as compared to conventional reconstruction methods For example, the model of the corridor corner, in Sect.4, was built from a single image This constitutes a very difficult task for automatic reconstruction due to the low texture These 3D models formed the basis for the human-robot interface Unlike many other works, a unique feature of this representation was that the user could specify a given destination, at a certain orientation, simply by rotating the 3D model When considering the system as a whole, (i) our approach to visual perception was found to be useful and convenient because it provided world-structure information for navigation, tailored to the task at hand, (ii) the navigation modalities fulfilled the purpose of semi-autonomous navigation by providing autonomy while naturally combining with the human-robot interface, (iii) the human-robot interface provided intuitive way to set high level tasks, by combining limited user input with the simple output of the sensor (images) In the future, omnidirectional vision will certainly have many developments Many current catadioptric setups assume a rigid mounting of the mirror on the camera Pan-tilt-zoom cameras have been demonstrated to be convenient for surveillance tasks, because of providing a large (virtual) field-of-view while having good resolution when zooming at regions of interest [75] Adding convex mirrors will allow enlarging the field-of-view and achieving faster pan-tilt motions, obtaining the so termed active omnidirectional camera Networking cameras poses new calibration challenges resulting from the mixture of various camera types, the overlapping (or not) of the fieldsof-view, the different requirements of calibration quality (many times can be Toward Robot Perception through Omnidirectional Vision 265 reduced just to a topological connection between cameras) and the type of calibration data used (as simple as static background or as dynamic as people moving) [76] As suggested by the title, we believe there is a large amount of work still to be done before we have a full and true understanding of perception We believe that key challenges can be addressed by building artificial vision systems In the future our understanding of perception will allow for robots with visual perception systems, robust enough to cope with new and novel environments Then, as happened with computers, almost every person will have their very own robot, or what we may term the personal service robot References S Baker and S K Nayar, A theory of catadioptric image formation, Proc Int Conf Computer Vision (ICCV’97), January 1998, pp 35–42 A theory of single-viewpoint catadioptric image formation, International Journal of Computer Vision 35 (1999), no 2, 175–196 R Benosman and S B Kang (eds.), Panoramic vision, Springer Verlag, 2001 M Betke and L Gurvits, Mobile robot localization using landmarks, IEEE Trans on Robotics and Automation 13 (1997), no 2, 251–263 J Borenstein, H R Everett, and Liqiang Feng, Navigating mobile robots: Sensors and techniques, A K Peters, Ltd., Wellesley, MA, 1996 (also: Where am I? Systems and Methods for Mobile Robot Positioning, ftp://ftp.eecs.umich edu/people/johannb/pos96rep.pdf) G Borgefors, Hierarchical chamfer matching: A parametric edge matching algorithm, IEEE Transactions on Pattern Analysis and Machine Intelligence 10 (1988), no 6, 849–865 R Brooks, Visual map making for a mobile robot, Proc IEEE Conf on Robotics and Automation, 1985 R A Brooks, A robust layered control system for a mobile robot, IEEE Transactions on Robotics and Automation (1986), 14–23 A Bruckstein and T Richardson, Omniview cameras with curved surface mirrors, Proceedings of the IEEE Workshop on Omnidirectional Vision at CVPR 2000, June 2000, First published in 1996 as a Bell Labs Technical Memo, pp 79–86 10 D Burschka, J Geiman, and G Hager, Optimal landmark configuration for vision-based control of mobile robots, Proc IEEE Int Conf on Robotics and Automation, 2003, pp 3917–3922 11 Z L Cao, S J Oh, and E.L Hall, Dynamic omni-directional vision for mobile robots, Journal of Robotic Systems (1986), no 1, 5–17 12 J S Chahl and M V Srinivasan, Reflective surfaces for panoramic imaging, Applied Optics 36 (1997), no 31, 8275–8285 13 P Chang and M Herbert, Omni-directional structure from motion, Proceedings of the 1st International IEEE Workshop on Omni-directional Vision (OMNIVIS’00) at CVPR 2000, June 2000 14 R Collins and R Weiss, Vanishing point calculation as a statistical inference on the unit sphere, Int Conf on Computer Vision (ICCV), 1990, pp 400–403 266 J Gaspar et al 15 T Conroy and J Moore, Resolution invariant surfaces for panoramic vision systems, IEEE ICCV’99, 1999, pp 392–397 16 Olivier Cuisenaire, Distance transformations: Fast algorithms and applications to medical image processing, Ph.D thesis, U Catholique de Louvain, October 1999 17 K Daniilidis (ed.), 1st international ieee workshop on omnidirectional vision at cvpr 2000, June 2000 18 ——, Page of omnidirectional vision hosted by the grasp laboratory, http://www cis.upenn.edu/∼kostas/omni.html, 2005 19 P David, D DeMenthon, and R Duraiswami, Simultaneous pose and correspondence determination using line features, Proc IEEE Conf Comp Vision Patt Recog., 2003 20 A Davison, Real-time simultaneous localisation and mapping with a single camera, IEEE Int Conf on Computer Vision, 2003, pp 1403–1410 vol 21 C Canudas de Wit, H Khennouf, C Samson, and O J Sordalen, Chap.5: Nonlinear control design for mobile robots, Nonlinear control for mobile robots (Yuan F Zheng, ed.), World Scientific series in Robotics and Intelligent Systems, 1993 22 P E Debevec, C J Taylor, and J Malik, Modeling and rendering architecture from photographs: a hybrid geometry and image-based approach, SIGGRAPH, 1996 23 S Derrien and K Konolige, Approximating a single viewpoint in panoramic imaging devices, Proceedings of the 1st International IEEE Workshop on Omnidirectional Vision at CVPR 2000, June 2000, pp 85–90 24 G DeSouza and A Kak, Vision for mobile robot navigation: A survey, IEEE Transactions on Pattern Analysis and Machine Intelligence 24 (2002), no 2, 237–267 25 O Faugeras, Three-dimensional computer vision - a geometric viewpoint, MIT Press, 1993 26 Mark Fiala, Panoramic computer vision, Ph.D thesis, University of Alberta, 2002 27 S Fleck, F Busch, P Biber, H Andreasson, and W Straber, Omnidirectional 3d modeling on a mobile robot using graph cuts, Proc IEEE Int Conf on Robotics and Automation, 2005, pp 1760–1766 28 J Foote and D Kimber, Flycam: Practical panoramic video and automatic camera control, Proc of the IEEE Int Conference on Multimedia and Expo, vol III, August 2000, pp 1419–1422 29 S Gaechter and T Pajdla, Mirror design for an omnidirectional camera with a uniform cylindrical projection when using svavisca sensor, Tech report, Czech Tech Univ - Faculty of Electrical Eng ftp://cmp.felk.cvut.cz/ pub/cmp/articles/pajdla/Gaechter-TR-2001-03.pdf, March 2001 30 S Gaechter, T Pajdla, and B Micusik, Mirror design for an omnidirectional camera with a space variant imager, IEEE Workshop on Omnidirectional Vision Applied to Robotic Orientation and Nondestructive Testing, August 2001, pp 99–105 31 J Gaspar, Omnidirectional vision for mobile robot navigation, Ph.D thesis, Instituto Superior T´cnico, Dept Electrical Engineering, Lisbon - Portugal, e 2003 Toward Robot Perception through Omnidirectional Vision 267 32 J Gaspar, C Decc´, J Okamoto Jr, and J Santos-Victor, Constant resolution o omnidirectional cameras, 3rd International IEEE Workshop on Omni-directional Vision at ECCV, 2002, pp 27–34 33 J Gaspar, E Grossmann, and J Santos-Victor, Interactive reconstruction from an omnidirectional image, 9th International Symposium on Intelligent Robotic Systems (SIRS’01), July 2001 34 J Gaspar and J Santos-Victor, Visual path following with a catadioptric panoramic camera, Int Symp Intelligent Robotic Systems, July 1999, pp 139–147 35 J Gaspar, N Winters, and J Santos-Victor, Vision-based navigation and environmental representations with an omni-directional camera, IEEE Transactions on Robotics and Automation 16 (2000), no 6, 890–898 36 D Gavrila and V Philomin, Real-time object detection for smart vehicles, IEEE, Int Conf on Computer Vision (ICCV), 1999, pp 87–93 37 C Geyer and K Daniilidis, A unifying theory for central panoramic systems and practical applications, ECCV 2000, June 2000, pp 445–461 38 ——, Catadioptric projective geometry, International Journal of Computer Vision 43 (2001), 223–243 39 Gene H Golub and Charles F Van Loan, Matrix computations, third ed., Johns Hopkins Studies in the Mathematical Sciences, The Johns Hopkins University Press, 1996 MR 417 720 40 P Greguss, Panoramic imaging block for 3d space, US patent 4,566,763, January 1986, Hungarian Patent granted in 1983 41 P Greguss (ed.), Ieee icar 2001 workshop on omnidirectional vision applied to robotic orientation and non-destructive testing, August 2001 42 E Grossmann, D Ortin, and J Santos-Victor, Algebraic aspects of reconstruction of structured scenes from one or more views, British Machine Vision Conference, BMVC2001, September 2001, pp 633–642 43 Etienne Grossmann, Maximum likelihood 3d reconstruction from one or more uncalibrated views under geometric constraints, Ph.D thesis, Instituto Superior T´cnico, Dept Electrical Engineering, Lisbon–Portugal, 2002 e 44 E Hecht and A Zajac, Optics, Addison Wesley, 1974 45 R Hicks, The page of catadioptric sensor design, http://www.math.drexel edu/∼ahicks/design/, 2004 46 R Hicks and R Bajcsy, Catadioptric sensors that approximate wide-angle perspective projections, Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR’00), June 2000, pp 545–551 47 A Howard, M.J Mataric, and G Sukhatme, Putting the ‘i’ in ‘team’: an egocentric approach to cooperative localization, IEEE Int Conf on Robotics and Automation, 2003 48 D Huttenlocher, G Klanderman, and W Rucklidge, Comparing images using the hausdorff distance, IEEE Transactions on Pattern Analysis and Machine Intelligence 15 (1993), no 9, 850–863 49 D Huttenlocher, R Lilien, and C Olsen, View-based recognition using an eigenspace approximation to the hausdorff measure, IEEE Transactions on Pattern Analysis and Machine Intelligence 21 (1999), no 9, 951–956 50 S B Kang and R Szeliski, 3d scene data recovery using omnidirectional multibaseline stereo, CVPR, 1996, pp 364–370 268 J Gaspar et al 51 N Karlsson, E Di Bernardo, J Ostrowski, L Goncalves, P Pirjanian, and M Munich, The vslam algorithm for robust localization and mapping, Proc IEEE Int Conf on Robotics and Automation, 2005, pp 24–29 52 A Kosaka and A Kak, Fast vision-guided mobile robot navigation using modelbased reasoning and prediction of uncertainties, CVGIP: Image Understanding 56 (1992), no 3, 271–329 53 J J Leonard and H F Durrant-Whyte, Mobile robot localization by tracking geometric beacons, IEEE Trans on Robotics and Automation (1991), no 3, 376–382 54 R Lerner, E Rivlin, and I Shimshoni, Landmark selection for task-oriented navigation, Proc Int Conf on Intelligent Robots and Systems, 2006, pp 2785–2791 55 LIRA-Lab, Document on specification, Tech report, Esprit Project n 31951– SVAVISCA - available at http://www.lira.dist.unige.it - SVAVISCA– GIOTTO Home Page, May 1999 56 A Majumder, W Seales, G Meenakshisundaram, and H Fuchs, Immersive teleconferencing: A new algorithm to generate seamless panoramic video imagery, Proceedings of the 7th ACM Conference on Multimedia, 1999 57 D Marr, Vision, W.H Freeman, 1982 58 B McBride, Panoramic cameras time line, http://panphoto.com/TimeLine html 59 B Micusik and T Pajdla, Structure from motion with wide circular field of view cameras, IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI) 28 (2006), no 7, 1135–1149 60 K Miyamoto, Fish-eye lens, Journal of the Optical Society of America 54 (1964), no 8, 1060–1061 61 L Montesano, J Gaspar, J Santos-Victor, and L Montano, Cooperative localization by fusing vision-based bearing measurements and motion, Int Conf on Intelligent Robotics and Systems, 2005, pp 2333–2338 62 H Murase and S K Nayar, Visual learning and recognition of 3d objects from appearance, International Journal of Computer Vision 14 (1995), no 1, 5–24 63 V Nalwa, A true omni-directional viewer, Technical report, Bell Laboratories, February 1996 64 S K Nayar, Catadioptric image formation, Proc of the DARPA Image Understanding Workshop, May 1997, pp 1431–1437 65 ——, Catadioptric omnidirectional camera, Proc IEEE Conf Computer Vision and Pattern Recognition, June 1997, pp 482–488 66 S K Nayar and V Peri, Folded catadioptric cameras, Proceedings of the IEEE Computer Vision and Pattern Recognition Conference, June 1999 67 E Oja, Subspace methods for pattern recognition, Research Studies Press, 1983 68 M Ollis, H Herman, and S Singh, Analysis and design of panoramic stereo using equi-angular pixel cameras, Tech report, Carnegie Mellon University Robotics Institute, TR CMU-RI-TR-99-04, 1999, comes from web 69 T Pajdla and V Hlavac, Zero phase representation of panoramic images for image based localization, 8th Inter Conf on Computer Analysis of Images and Patterns CAIP’99, 1999 70 V Peri and S K Nayar, Generation of perspective and panoramic video from omnidirectional video, Proc DARPA Image Understanding Workshop, 1997, pp 243–246 Toward Robot Perception through Omnidirectional Vision 269 71 R Pless, Using many cameras as one, Proc CVPR, 2003, pp II: 587–593 72 D Rees, Panoramic television viewing system, us patent 505 465, postscript file, April 1970 73 W Rucklidge, Efficient visual recognition using the hausdorff distance, Lecture Notes in Computer Science, vol 1173, Springer-Verlag, 1996 74 J Shi and C Tomasi, Good features to track, Proc of the IEEE Int Conference on Computer Vision and Pattern Recognition, June 1994, pp 593–600 75 S Sinha and M Pollefeys, Towards calibrating a pan-tilt-zoom camera network, OMNIVIS’04, workshop on Omnidirectional Vision and Camera Networks (held with ECCV 2004), 2004 76 S.N Sinha and M Pollefeys, Synchronization and calibration of camera networks from silhouettes, International Conference on Pattern Recognition (ICPR’04), vol 1, 23–26 Aug 2004, pp 116–119 Vol 77 T Sogo, H Ishiguro, and M Treivedi, Real-time target localization and tracking by n-ocular stereo, Proceedings of the 1st International IEEE Workshop on Omni-directional Vision (OMNIVIS’00) at CVPR 2000, June 2000 78 M Spetsakis and J Aloimonos, Structure from motion using line correspondences, International Journal of Computer Vision (1990), no 3, 171–183 79 P Sturm, A method for 3d reconstruction of piecewise planar objects from single panoramic images, 1st International IEEE Workshop on Omnidirectional Vision at CVPR, 2000, pp 119–126 80 P Sturm and S Ramalingam, A generic concept for camera calibration, Proceedings of the European Conference on Computer Vision, Prague, Czech Republic, vol 2, Springer, May 2004, pp 1–13 81 W Sturzl, H Dahmen, and H Mallot, The quality of catadioptric imaging application to omnidirectional stereo, European Conference on Computer Vision, 2004, pp LNCS 3021:614–627 82 T Svoboda, T Pajdla, and V Hlav´ˇ, Epipolar geometry for panoramic camac eras, Proc European Conf Computer Vision, July 1998, pp 218–231 83 R Talluri and J K Aggarwal, Mobile robot self-location using model-image feature correspondence, IEEE Transactions on Robotics and Automation 12 (1996), no 1, 63–77 84 G Thomas, Real-time panospheric image dewarping and presentation for remote mobile robot control, Journal of Advanced Robotics 17 (2003), no 4, 359–368 85 S Thrun and A Bucken, Integrating grid-based and topological maps for mobile robot navigation, Proceedings of the 13th National Conference on Artifical Intelligence (AAAI’96), 1996 86 S Watanabe, Karhunen-lo`ve expansion and factor analysis, Transactions of the e 4th Prague Conference on Information Theory, Statistical Decision Functions and Random Processes, 1965, pp 635–660 87 R Wehner and S Wehner, Insect navigation: use of maps or ariadne’s thread?, Ethology, Ecology, Evolution (1990), 27–48 88 N Winters, A holistic approach to mobile robot navigation using omnidirectional vision, Ph.D thesis, University of Dublin, Trinity College, 2002 89 N Winters, J Gaspar, G Lacey, and J Santos-Victor, Omni-directional vision for robot navigation, 1st International IEEE Workshop on Omni-directional Vision at CVPR, 2000, pp 21–28 90 N Winters and J Santos-Victor, Omni-directional visual navigation, 7th International Symposium on Intelligent Robotics Systems (SIRS’99), July 1999, pp 109–118 270 J Gaspar et al 91 N Winters and G Lacey, Overview of tele-operation for a mobile robot, TMR Workshop on Computer Vision and Mobile Robots (CVMR’98), September 1999 92 N Winters and J Santos-Victor, Omni-directional visual navigation, Proc Int Symp on Intelligent Robotic Systems, July 1999, pp 109–118 93 P Wunsch and G Hirzinger, Real-time visual tracking of 3-d objects with dynamic handling of occlusion, IEEE Int Conf on Robotics and Automation, April 1997, pp 2868–2873 94 Y Yagi, Omnidirectional sensing and its applications, IEICE Transactions on Information and Systems (1999), no E82-D-3, 568–579 95 Y Yagi, Y Nishizawa, and M Yachida, Map-based navigation for mobile robot with omnidirectional image sensor COPIS, IEEE Trans Robotics and Automation 11 (1995), no 5, 634–648 96 K Yamazawa, Y Yagi, and M Yachida, Obstacle detection with omnidirectional image sensor hyperomni vision, IEEE ICRA, 1995, pp 1062–1067 97 J Zheng and S Tsuji, Panoramic representation for route recognition by a mobile robot, International Journal of Computer Vision (1992), no 1, 55–76 ... Innovations in Intelligent Machines- 1, 2007 ISBN 978-3-540-72695 -1 Javaan Singh Chahl Lakhmi C Jain Akiko Mizutani Mika Sato-Ilic (Eds. ) Innovations in Intelligent Machines - With 14 6 Figures and 10 Tables... Jain et al. : Intelligent Machines: An Introduction, Studies in Computational Intelligence (SCI) 70, 1? ??9 (2007) c Springer-Verlag Berlin Heidelberg 2007 www.springerlink.com L.C Jain et al When... Proceedings of the 20 01 IEEE Mountain Workshop on Soft Computing in Industrial Applications, pp 10 9? ?11 4, 20 01 12 J Pearl, Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference,