Advances in Pattern Recognition For other titles published in this series, go to http://www.springer.com/4205 Branislav Kisa˘canin Shuvra S Bhattacharyya Sek Chai • Editors Embedded Computer Vision 123 Editors Branislav Kisa˘canin, PhD Texas Instruments Dallas, TX, USA Shuvra S Bhattacharyya, PhD University of Maryland College Park, MD, USA Sek Chai, PhD Motorola Schaumburg, IL, USA Series editor Professor Sameer Singh, PhD Research School of Informatics, Loughborough University, Loughborough, UK Advances in Pattern Recognition Series ISSN 1617-7916 ISBN 978-1-84800-303-3 e-ISBN 978-1-84800-304-0 DOI 10.1007/978-1-84800-304-0 British Library Cataloguing in Publication Data A catalogue record for this book is available from the British Library Library of Congress Control Number: 2008935617 c Springer-Verlag London Limited 2009 Apart from any fair dealing for the purposes of research or private study, or criticism or review, as permitted under the Copyright, Designs and Patents Act 1988, this publication may only be reproduced, stored or transmitted, in any form or by any means, with the prior permission in writing of the publishers, or in the case of reprographic reproduction in accordance with the terms of licences issued by the Copyright Licensing Agency Enquiries concerning reproduction outside those terms should be sent to the publishers The use of registered names, trademarks, etc in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant laws and regulations and therefore free for general use The publisher makes no representation, express or implied, with regard to the accuracy of the information contained in this book and cannot accept any legal responsibility or liability for any errors or omissions that may be made Printed on acid-free paper Springer Science+Business Media springer.com To Saˇska, Milena, and Nikola BK To Milu, Arpan, and Diya SSB To Ying and Aaron SC Foreword As a graduate student at Ohio State in the mid-1970s, I inherited a unique computer vision laboratory from the doctoral research of previous students They had designed and built an early frame-grabber to deliver digitized color video from a (very large) electronic video camera on a tripod to a mini-computer (sic) with a (huge!) disk drive—about the size of four washing machines They had also designed a binary image array processor and programming language, complete with a user’s guide, to facilitate designing software for this one-of-a-kind processor The overall system enabled programmable real-time image processing at video rate for many operations I had the whole lab to myself I designed software that detected an object in the field of view, tracked its movements in real time, and displayed a running description of the events in English For example: “An object has appeared in the upper right corner It is moving down and to the left Now the object is getting closer The object moved out of sight to the left”—about like that The algorithms were simple, relying on a sufficient image intensity difference to separate the object from the background (a plain wall) From computer vision papers I had read, I knew that vision in general imaging conditions is much more sophisticated But it worked, it was great fun, and I was hooked A lot has changed since! Dissertation after dissertation, the computer vision research community has contributed many new techniques to expand the scope and reliability of real-time computer vision systems Cameras changed from analog to digital and became incredibly small At the same time, computers shrank from minicomputers to workstations to personal computers to microprocessors to digital signal processors to programmable digital media systems on a chip Disk drives became very small and are starting to give way to multi-gigabyte flash memories Many computer vision systems are so small and embedded in other systems that we don’t even call them “computers” anymore We call them automotive vision sensors, such as lane departure and blind spot warning sensors We call them smart cameras and digital video recorders for video surveillance We call them mobile phones (which happen to have embedded cameras and 5+ million lines of wideranging software), and so on vii viii Foreword Today that entire computer vision laboratory of the 1970s is upstaged by a battery-powered camera phone in my pocket So we are entering the age of “embedded vision.” Like optical character recognition and industrial inspection (machine vision) applications previously became sufficiently useful and cost-effective to be economically important, diverse embedded vision applications are emerging now to make the world a safer and better place to live We still have a lot of work to do! In this book we look at some of the latest techniques from universities and companies poking outside the envelope of what we already knew how to build We see emphasis on tackling important problems for society We see engineers evaluating many of the trade-offs needed to design cost-effective systems for successful products Should I use this processor or design my own? How many processors I need? Which algorithm is sufficient for a given problem? Can I re-design my algorithm to use a fixed-point processor? I see all of the chapters in this book as marking the embedded vision age The lessons learned that the authors share will help many of us to build better vision systems, align new research with important needs, and deliver it all in extraordinarily small systems May 2008 Bruce Flinchbaugh Dallas, TX Preface Embedded Computer Vision We are witnessing a major shift in the way computer vision applications are implemented, even developed The most obvious manifestation of this shift is in the platforms that computer vision algorithms are running on: from powerful workstations to embedded processors As is often the case, this shift came about at the intersection of enabling technologies and market needs In turn, a new discipline has emerged within the imaging/vision community to deal with the new challenges: embedded computer vision (ECV) Building on synergistic advances over the past decades in computer vision algorithms, embedded processing architectures, integrated circuit technology, and electronic system design methodologies, ECV techniques are increasingly being deployed in a wide variety of important applications They include high volume, cost-centric consumer applications, as well as accuracy- and performance-centric, mission-critical systems For example, in the multi-billion dollar computer and video gaming industry, the Sony EyeToyTMcamera, which includes processing to detect color and motion, is reaching out to gamers to play without any other interfaces Very soon, new camera-based games will detect body gestures based on movements of the hands, arms, and legs, to enhance the user experience These games are built upon computer vision research on articulated body pose estimation and other kinds of motion capture analysis As a prominent example outside of the gaming industry, the rapidly expanding medical imaging industry makes extensive use of ECV techniques to improve the accuracy of medical diagnoses, and to greatly reduce the side effects of surgical and diagnostic procedures Furthermore, ECV techniques can help address some of society’s basic needs for safety and security They are well suited for automated surveillance applications, which help to protect against malicious or otherwise unwanted intruders and activities, as well as for automotive safety applications, which aim to assist the driver and improve road safety ix x Preface Some well-established products and highly publicized technologies may be seen as early examples of ECV Two examples are the optical mouse (which uses a hardware implementation of an optical flow algorithm), and NASA’s Martian rovers, Spirit and Opportunity (which used computer vision on a processor of very limited capabilities during the landing, and which have a capability for vision-based self-navigation) In addition to the rapidly increasing importance and variety of ECV applications, this domain of embedded systems warrants specialized focus because ECV applications have a number of distinguishing requirements compared to general-purpose systems and other embedded domains For example, in low- to middle-end generalpurpose systems, and in domains of embedded computing outside of ECV, performance requirements are often significantly lower than what we encounter in ECV Cost and power consumption considerations are important for some areas of ECV, as they are in other areas of consumer electronics However, in some areas of ECV, such as medical imaging and surveillance, considerations of real-time performance and accuracy dominate Performance in turn is strongly related to considerations of buffering efficiency and memory management due to the large volumes of pixel data that must be processed in ECV systems This convergence of high-volume, multidimensional data processing; real-time performance requirements; and complex trade-offs between achievable accuracy and performance gives rise to some of the key distinguishing aspects in the design and implementation of ECV systems These aspects have also helped to influence the evolution of some of the major classes of embedded processing devices and platforms—including field programmable gate arrays (FPGAs), programmable digital signal processors (DSPs), graphics processing units (GPUs), and various kinds of heterogeneous embedded multiprocessor devices—that are relevant to the ECV domain Target Audience This book is written for researchers, practitioners, and managers of innovation in the field of ECV The researchers are those interested in advancing theory and application conception For this audience, we present the state of the art of the field today, and provide insight about where major applications may go in the near future The practitioners are those involved in the implementation, development, and deployment of products For this audience, we provide the latest approaches and methodologies to designing on the different processing platforms for ECV Lastly, the managers are those tasked with leading the product innovation in a corporation For this audience, we provide an understanding of the technology so that necessary resources and competencies can be put in place to effectively develop a product based on computer vision For designers starting in this field, we provide in this book a historical perspective on early work in ECV that is a necessary foundation for their work For those in the midst of development, we have compiled a list of recent research from industry Preface xi and academia In either case, we hope to give a well-rounded discussion of future developments in ECV, from implementation methodology to applications The book can also be used to provide an integrated collection of readings for specialized graduate courses or professionally oriented short courses on ECV The book could, for example, help to complement a project-oriented emphasis in such a course with readings that would help to give a broader perspective on both the state of the art and evolution of the field Organization of the Book Each chapter in this book is a stand-alone exposition of a particular topic The chapters are grouped into three parts: Part I: Introduction, which comprises three introductory chapters: one on hardware and architectures for ECV, another on design methodologies, and one that introduces the reader to video analytics, possibly the fastest growing area of application of ECV Part II: Advances in Embedded Computer Vision, which contains seven chapters on the state-of-the art developments in ECV These chapters explore advantages of various architectures, develop high-level software frameworks, and develop algorithmic alternatives that are close in performance to standard approaches, yet computationally less expensive We also learn about issues of implementation on a fixed-point processor, presented on an example of an automotive safety application Part III: Looking Ahead, which consists of three forward-looking chapters describing challenges in mobile environments, video analytics, and automotive safety applications Overview of Chapters Each chapter mimics the organization of the book They all provide introduction, results, and challenges, but to a different degree, depending on whether they were written for Part I, II, or III Here is a summary of each chapter’s contribution: Part I: Introduction • Chapter 1: Hardware Considerations for Embedded Vision Systems by Mathias K¨olsch and Steven Butner This chapter is a gentle introduction into the complicated world of processing architectures suitable for vision: DSPs, FPGAs, SoCs, ASICs, GPUs, and GPPs The authors argue that in order to better understand the trade-offs involved in choosing the right architecture for a particular application, one needs to understand the entire real-time vision pipeline Following the pipeline, they discuss all of its parts, tracing the information flow from photons on the front end to the high-level output produced by the system at the back end 13 Challenges of Embedded Computer Vision in Automotive Safety Systems 267 Fig 13.10 [Left image] Shows examples of the default vehicle regions that are used for classification The default vehicle regions are obtained by placing a box centrally over each of the dark points in Fig 13.9 The boxes placed use the expected width at that row as their width and get their height using a default aspect ratio (that of a car) [Right image] Shows the final bounding boxes after refinement Note the improvement in accuracy of the bounding boxes speed up the execution is to run the algorithm on a subsampled version of the input image Since the cueing algorithm uses features that are reasonably independent of the size of the input image it is a good candidate for running at different scales, without requiring modification Table 13.1 gives a summary of the time taken for the different steps of the cueing algorithm when run at different resolutions These runs were performed on a PC running at 2.4 GHz The reported times are the total times taken by the various parts of the cueing algorithm when processing a full frame These values were obtained by averaging over multiple frames Table 13.1 Time taken for the various cueing steps Average times for the various steps over a full frame Resolution 320×240 160×120 Step Time Time Edge detection ms ms Edge post-processing ms [...]... Methodology for Embedded Computer Vision Systems by Sankalita Saha and Shuvra S Bhattacharyya In this chapter the authors provide a broad overview of literature regarding design methodologies for embedded computer vision • Chapter 3: We Can Watch It for You Wholesale by Alan J Lipton In this chapter the reader is taken on a tour of one of the fastest growing application areas in embedded computer vision video... of Computer Vision Applications by Changsong Shen, James J Little, and Sidney Fels In this chapter the authors present their work on a unified software architecture, OpenVL, which addresses a variety of problems faced by designers of embedded vision systems, such as hardware acceleration, reusability, and scalability Part III: Looking Ahead • Chapter 11: Mobile Challenges for Embedded Computer Vision. .. christian.zinner@arcs.ac.at List of Contributors Part I Introduction Chapter 1 Hardware Considerations for Embedded Vision Systems Mathias K¨olsch and Steven Butner Abstract Image processing and computer vision do not start with a frame in the frame buffer Embedded vision systems need to consider the entire real-time vision pipeline from image acquisition to result output, including the operations that are to be... foreseeing a need for such processing With the help of these highly optimized processing capabilities, advanced computer vision applications are already finding their way into consumer products The layout of this chapter largely mirrors the computer vision pipeline, from photon to embedded computer vision output First, we take a look at digital image sensors, followed by a description of interfaces between... commercial side of our field Part II: Advances in Embedded Computer Vision • Chapter 4: Using Robust Local Features on DSP-based Embedded Systems by Clemens Arth, Christian Leistner, and Horst Bischof In this chapter the authors present their work on robust local feature detectors and their suitability for embedded implementation They also describe their embedded implementation on a DSP platform and their... Vision for an Analytics-Powered Future 254 12.6 Summary 254 References 255 13 Challenges of Embedded Computer Vision in Automotive Safety Systems 257 Yan Zhang, Arnab S Dhua, Stephen J Kiselewich, and William A Bauson 13.1 Computer Vision. .. methods between and among these components are discussed, and specific organizational approaches are compared and contrasted 1.1 The Real-Time Computer Vision Pipeline You might be faced with the task of building an embedded system that performs one or multiple computer vision tasks One of your first and hardest questions would probably be what hardware components you should use There is a bewildering array... What is considered an embedded system? An embedded computer comprises one or multiple processors that serve a dedicated purpose in a larger system, for example, video processors in television sets and DVD players Their advantages lie in their suitability to the task, resulting in greater speed, lower power consumption, reduced cost, and faster startup than general-purpose systems Embedded systems are... Looking Ahead 11 Mobile Challenges for Embedded Computer Vision 219 Sek Chai 11.1 Introduction 219 11.2 In Search of the Killer Applications 221 11.2.1 Image Finishing 221 11.2.2 Video Codec 222 11.2.3 Computer Vision ... fast growing area, both technical and nontechnical • Chapter 13: Challenges of Embedded Computer Vision in Automotive Safety Systems by Yan Zhang, Arnab S Dhua, Stephen J Kiselewich, and William A Bauson This chapter provides a gentle introduction into the numerous techniques that will one day have to be implemented on an embedded platform in order to help improve automotive safety The system described ... the imaging /vision community to deal with the new challenges: embedded computer vision (ECV) Building on synergistic advances over the past decades in computer vision algorithms, embedded processing... advanced computer vision applications are already finding their way into consumer products The layout of this chapter largely mirrors the computer vision pipeline, from photon to embedded computer vision. .. Considerations for Embedded Vision Systems Mathias K¨olsch and Steven Butner Abstract Image processing and computer vision not start with a frame in the frame buffer Embedded vision systems need