DESIGN FOR EMBEDDED IMAGE PROCESSING ON FPGAS DESIGN FOR EMBEDDED IMAGE PROCESSING ON FPGAS Donald G. Bailey Massey University, New Zealand This edition first published 2011 Ó 2011 John Wiley & Sons (Asia) Pte Ltd Registered office John Wiley & Sons (Asia) Pte Ltd, 1 Fusionopolis Walk, #07-01 Solaris South Tower, Singapore 138628 For details of our global editorial offices, for customer services and for information about how to apply for permission to reuse the copyright material in this book please see our website at www.wiley.com. All Rights Reserved. No part of this publication may be reproduced, stored in a retrieval system or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as expressly permitted by law, without either the prior written permission of the Publisher, or authorization through payment of the appropriate photocopy fee to the Copyright Clearance Center. Requests for permission should be addressed to the Publisher, John Wiley & Sons (Asia) Pte Ltd, 1 Fusionopolis Walk, #07-01 Solaris South Tower, Singapore 138628, tel: 65-66438000, fax: 65-66438008, email: enquiry@wiley.com. Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic books. Designations used by companies to distinguish their products are often claimed as trademarks. All brand names and product names used in this book are trade names, service marks, trademarks or registered trademarks of their respective owners. The Publisher is not associated with any product or vendor mentioned in this book. This publication is designed to provide accurate and authoritative information in regard to the subject matter covered. It is sold on the understanding that the Publisher is not engaged in rendering professional services. If professional advice or other expert assistance is required, the services of a competent professional should be sought. Library of Congress Cataloging-in-Publication Data Bailey, Donald G. (Donald Graeme), 1962- Design for embedded image processing on FPGAs / Donald G. Bailey. p. cm. Includes bibliographical references and index. ISBN 978-0-470-82849-6 (hardback) 1. Embedded computer systems. 2. Field programmable gate arrays. I. Title. TK7895.E42B3264 2011 621.39’9–dc22 2011002991 Print ISBN: 978-0-470-82849-6 ePDF ISBN: 978-0-470-82850-2 oBook ISBN: 978-0-470-82851-9 ePub ISBN: 978-0-470-82852-6 Mobi ISBN: 978-1-118-07331-5 Set in 9/11 pt Times New Roman by Thomson Digital, Noida, India Contents Preface xi Acknowledgements xvii 1 Image Processing 1 1.1 Basic Definitions 2 1.2 Image Formation 3 1.3 Image Processing Operations 7 1.4 Example Application 9 1.5 Real-Time Image Processing 11 1.6 Embedded Image Processing 12 1.7 Serial Processing 12 1.8 Parallelism 14 1.9 Hardware Image Processing Systems 18 2 Field Programmable Gate Arrays 21 2.1 Programmable Logic 21 2.1.1 FPGAs vs. ASICs 24 2.2 FPGAs and Image Processing 25 2.3 Inside an FPGA 26 2.3.1 Logic 27 2.3.2 Interconnect 28 2.3.3 Input and Output 29 2.3.4 Clocking 30 2.3.5 Configuration 31 2.3.6 Power Consumption 32 2.4 FPGA Families and Features 33 2.4.1 Xilinx 33 2.4.2 Altera 38 2.4.3 Lattice Semiconductor 44 2.4.4 Achronix 46 2.4.5 SiliconBlue 47 2.4.6 Tabula 47 2.4.7 Actel 48 2.4.8 Atmel 49 2.4.9 QuickLogic 50 2.4.10 MathStar 50 2.4.11 Cypress 51 2.5 Choosing an FPGA or Development Board 51 3 Languages 53 3.1 Hardware Description Languages 56 3.2 Software-Based Languages 61 3.2.1 Structural Approaches 63 3.2.2 Augmented Languages 64 3.2.3 Native Compilation Techniques 69 3.3 Visual Languages 72 3.3.1 Behavioural 73 3.3.2 Dataflow 73 3.3.3 Hybrid 74 3.4 Summary 77 4 Design Process 79 4.1 Problem Specification 79 4.2 Algorithm Development 81 4.2.1 Algorithm Development Process 82 4.2.2 Algorithm Structure 83 4.2.3 FPGA Development Issues 86 4.3 Architecture Selection 86 4.3.1 System Level Architecture 87 4.3.2 Computational Architecture 89 4.3.3 Partitioning between Hardware and Software 93 4.4 System Implementation 96 4.4.1 Mapping to FPGA Resources 97 4.4.2 Algorithm Mapping Issues 100 4.4.3 Design Flow 101 4.5 Designing for Tuning and Debugging 102 4.5.1 Algorithm Tuning 102 4.5.2 System Debugging 104 5 Mapping Techniques 107 5.1 Timing Constraints 107 5.1.1 Low Level Pipelining 107 5.1.2 Process Synchronisation 110 5.1.3 Multiple Clock Domains 111 5.2 Memory Bandwidth Constraints 113 5.2.1 Memory Architectures 113 5.2.2 Caching 116 5.2.3 Row Buffering 117 5.2.4 Other Memory Structures 118 vi Contents 5.3 Resource Constraints 122 5.3.1 Resource Multiplexing 122 5.3.2 Resource Controllers 125 5.3.3 Reconfigurability 130 5.4 Computational Techniques 132 5.4.1 Number Systems 132 5.4.2 Lookup Tables 138 5.4.3 CORDIC 142 5.4.4 Approximations 150 5.4.5 Other Techniques 152 5.5 Summary 154 6 Point Operations 155 6.1 Point Operations on a Single Image 155 6.1.1 Contrast and Brightness Adjustment 155 6.1.2 Global Thresholding and Contouring 159 6.1.3 Lookup Table Implementation 162 6.2 Point Operations on Multiple Images 163 6.2.1 Image Averaging 164 6.2.2 Image Subtraction 166 6.2.3 Image Comparison 170 6.2.4 Intensity Scaling 171 6.2.5 Masking 173 6.3 Colour Image Processing 175 6.3.1 False Colouring 175 6.3.2 Colour Space Conversion 176 6.3.3 Colour Thresholding 192 6.3.4 Colour Correction 193 6.3.5 Colour Enhancement 197 6.4 Summary 197 7 Histogram Operations 199 7.1 Greyscale Histogram 199 7.1.1 Data Gathering 201 7.1.2 Histogram Equalisation 206 7.1.3 Automatic Exposure 210 7.1.4 Threshold Selection 211 7.1.5 Histogram Similarity 219 7.2 Multidimensional Histograms 219 7.2.1 Triangular Arrays 220 7.2.2 Multidimensional Statistics 222 7.2.3 Colour Segmentation 226 7.2.4 Colour Indexing 229 7.2.5 Texture Analysis 231 Contents vii 8 Local Filters 233 8.1 Caching 233 8.2 Linear Filters 239 8.2.1 Noise Smoothing 239 8.2.2 Edge Detection 241 8.2.3 Edge Enhancement 243 8.2.4 Linear Filter Techniques 243 8.3 Nonlinear Filters 248 8.3.1 Edge Orientation 250 8.3.2 Non-maximal Suppression 251 8.3.3 Zero-Crossing Detection 252 8.4 Rank Filters 252 8.4.1 Rank Filter Sorting Networks 255 8.4.2 Adaptive Histogram Equalisation 260 8.5 Colour Filters 261 8.6 Morphological Filters 264 8.6.1 Binary Morphology 264 8.6.2 Greyscale Morphology 269 8.6.3 Colour Morphology 270 8.7 Adaptive Thresholding 271 8.7.1 Error Diffusion 271 8.8 Summary 273 9 Geometric Transformations 275 9.1 Forward Mapping 276 9.1.1 Separable Mapping 277 9.2 Reverse Mapping 282 9.3 Interpolation 285 9.3.1 Bilinear Interpolation 286 9.3.2 Bicubic Interpolation 288 9.3.3 Splines 290 9.3.4 Interpolating Compressed Data 292 9.4 Mapping Optimisations 292 9.5 Image Registration 294 9.5.1 Feature-Based Methods 295 9.5.2 Area-Based Methods 299 9.5.3 Applications 305 10 Linear Transforms 309 10.1 Fourier Transform 310 10.1.1 Fast Fourier Transform 311 10.1.2 Filtering 318 10.1.3 Inverse Filtering 320 10.1.4 Interpolation 321 10.1.5 Registration 322 viii Contents 10.1.6 Feature Extraction 323 10.1.7 Goertzel’s Algorithm 324 10.2 Discrete Cosine Transform 325 10.3 Wavelet Transform 328 10.3.1 Filter Implementations 330 10.3.2 Applications of the Wavelet Transform 335 10.4 Image and Video Coding 336 11 Blob Detection and Labelling 343 11.1 Bounding Box 343 11.2 Run-Length Coding 346 11.3 Chain Coding 347 11.3.1 Sequential Implementation 347 11.3.2 Single Pass Algorithms 348 11.3.3 Feature Extraction 350 11.4 Connected Component Labelling 352 11.4.1 Random Access Algorithms 353 11.4.2 Multiple-Pass Algorithms 353 11.4.3 Two-Pass Algorithms 354 11.4.4 Single-Pass Algorithms 356 11.4.5 Multiple Input Labels 358 11.4.6 Further Optimisations 358 11.5 Distance Transform 359 11.5.1 Morphological Approaches 360 11.5.2 Chamfer Distance 360 11.5.3 Separable Transform 362 11.5.4 Applications 365 11.5.5 Geodesic Distance Transform 365 11.6 Watershed Transform 366 11.6.1 Flow Algorithms 366 11.6.2 Immersion Algorithms 367 11.6.3 Applications 369 11.7 Hough Transform 370 11.7.1 Line Hough Transform 371 11.7.2 Circle Hough Transform 373 11.7.3 Generalised Hough Transform 374 11.8 Summary 375 12 Interfacing 377 12.1 Camera Input 378 12.1.1 Camera Interface Standards 378 12.1.2 Deinterlacing 383 12.1.3 Global and Rolling Shutter Correction 384 12.1.4 Bayer Pattern Processing 384 Contents ix 12.2 Display Output 387 12.2.1 Display Driver 387 12.2.2 Display Content 390 12.3 Serial Communication 393 12.3.1 PS2 Interface 393 12.3.2 I 2 C 395 12.3.3 SPI 397 12.3.4 RS-232 397 12.3.5 USB 398 12.3.6 Ethernet 398 12.3.7 PCI Express 399 12.4 Memory 400 12.4.1 Static RAM 400 12.4.2 Dynamic RAM 401 12.4.3 Flash Memory 402 12.5 Summary 402 13 Testing, Tuning and Debugging 405 13.1 Design 405 13.1.1 Random Noise Sources 406 13.2 Implementation 409 13.2.1 Common Implementation Bugs 410 13.3 Tuning 412 13.4 Timing Closure 412 14 Example Applications 415 14.1 Coloured Region Tracking 415 14.2 Lens Distortion Correction 418 14.2.1 Characterising the Distortion 419 14.2.2 Correcting the Distortion 421 14.3 Foveal Sensor 424 14.3.1 Foveal Mapping 425 14.3.2 Using the Sensor 429 14.4 Range Imaging 429 14.4.1 Extending the Unambiguous Range 431 14.5 Real-Time Produce Grading 433 14.5.1 Software Algorithm 434 14.5.2 Hardware Implementation 436 14.6 Summary 439 References 441 Index 475 x Contents Preface I think it is useful to provide a little background as to why and how this book came into being. This will perhaps provide some insight into the way the material is structured, and why it is presented in the way that it is. Background Firstly, a little bit of history. I have an extensive background in image processing, particularly in the areas of image analysis, machine vision and robot vision, all strongly application-orientated areas. With over 25 years of applying image processing techniques to a wide range of problems, I have gained considerable experience in algorithm development. This is not only at the image processing application level but also at the image processing operation level. My approach to an application has usually been more pragmatic than theoretical – I have focussed on developing image processing algorithms that solved the problem at hand. Often this involved assembling sequences of existing image processing operations, but occasionally it required developing new algorithms and techniques to solve particular aspects of the problem. Through work on machine vision and robotics applications, I have become aware of some of the limitations of software-based solutions, particularly in terms of speed and algorithm efficiency. This led naturally to considering FPGAs as an implementation platform for embedded imaging applications. Many image processing operations are inherently parallel and FPGAs provide program- mable hardware, also inherently parallel. Therefore, it should be as simple as mapping one onto the other, right? Well, when I started implementing image processing algorithms on FPGAs, I had lots of ideas, but very little knowledge. I very soon found that therewere a lot of tricks that were needed to create an efficient design. Consequently, my students and I learned many of these the hard way, through trial and error. With my basic training as an electronics engineer, I was readily able to adapt to the hardware mindset. I have since discovered through observing my students, both at the undergraduate and postgraduate level, that this is perhaps the biggest hurdle to an efficient implementation. Image processing is traditionally thought of as a software domain task, whereas FPGA-based design is firmly in the hardware domain. To bridge the gap, it is necessary to think of algorithms not on their own but more in terms of their underlying computational architecture. Implementing an image processing algorithm (or indeed any algorithm) on an FPGA, therefore, consists of determining the underlying architecture of an algorithm, mapping that architecture onto the resources available within an FPGA and finally mapping the algorithm onto the hardware architecture. Unfortunately, there is very little material available to help those new to the area to get started. Even this insight into the process is not actually stated anywhere, although it is implicitly followed (whether consciously or not) by most people working in this area. [...]... inspection, to medical imaging Design for Embedded Image Processing on FPGAs, First Edition Donald G Bailey Ó 2011 John Wiley & Sons (Asia) Pte Ltd Published 2011 by John Wiley & Sons (Asia) Pte Ltd 2 Design for Embedded Image Processing on FPGAs 1.1 Basic Definitions More formally, an image is a spatial representation of an object, scene or other phenomenon (Haralick and Shapiro, 1991) Examples of images... are preprocessing operations These are image- to -image transformations, with the purpose of enhancing the relevant information within the image, while suppressing any irrelevant information Examples of preprocessing operations are distortion correction, contrast enhancement and filtering for noise reduction or edge detection Segmentation operations such as thresholding, colour detection, region growing... efficiently implementing common functions The next section of this book describes the implementation of many common image processing operations Some of the design decisions and alternative ways of mapping the operations onto FPGAs are considered While reasonably comprehensive, particularly for low level image- to -image transformations, it is impossible to cover every possible design The examples discussed... process are used to derive a three-dimensional representation based on one or more two-dimensional images of a scene The use of models implicitly provides an interpretation of the contents of the images obtained The fields are sometimes distinguished based on application: Machine vision is using image processing as part of the control system for a machine (Schaffer, 1984) Images are captured and analysed,... applications It involves both image reconstruction to create meaningful images from the raw data gathered from the sensors, and image analysis to extract useful information from the images Image and video coding focuses on the compression of an image or image sequence so that it occupies less storage space or takes less time to transmit from one location to another Compression is possible because many images... image super-resolution (reconstructing a high resolution image from a series of low resolution images) and tomography (reconstructing a cross-section of an object from a series of projections) Image analysis refers specifically to using computers to extract data from images The result is usually some form of measurement In the past, this was almost exclusively two-dimensional imaging, Image Processing. .. component Consequently, an embedded system is usually designed to perform one specific task, or a small range of specific tasks (Catsoulis, 2005), often with real-time constraints An obvious example of an embedded image processing system is a digital camera There the imaging functions include exposure and focus control, displaying a preview, and managing image compression and decompression Embedded vision... machine vision system for real-time produce grading Preface xvi Filter I Q k I 1 0 || B A Q T B A A-B A>B Image processing operation I Register x Counter k Constant | | Function block Frame buffer Single bit signal Multi-bit signal (a number) Signal concatenation Signal splitting Multiplexer Figure P.1 Conventions used in this book Top left: representation of an image processing operation; middle left:... set to minimise the classification errors (Bailey, 1985) This example is typical of a real-time inspection application It illustrates the different stages of the processing, and demonstrates how the image and information are transformed as the image is processed Image Processing 11 This application has mainly low and intermediate level operations; the high level classification step is relatively trivial... execution time This characteristic rules out certain classes of operation level algorithms from real-time processing In particular, operations that are based on iterative or recursive algorithms can only be used if they can be guaranteed to converge satisfactorily within a predefined number of iterations, for the whole range of inputs that may be encountered 12 Design for Embedded Image Processing on FPGAs . competent professional should be sought. Library of Congress Cataloging-in-Publication Data Bailey, Donald G. (Donald Graeme), 196 2- Design for embedded image processing on FPGAs / Donald G. Bailey. p DESIGN FOR EMBEDDED IMAGE PROCESSING ON FPGAS DESIGN FOR EMBEDDED IMAGE PROCESSING ON FPGAS Donald G. Bailey Massey University, New Zealand This edition first published 2011 Ó. ISBN: 97 8-0 -4 7 0-8 284 9-6 ePDF ISBN: 97 8-0 -4 7 0-8 285 0-2 oBook ISBN: 97 8-0 -4 7 0-8 285 1-9 ePub ISBN: 97 8-0 -4 7 0-8 285 2-6 Mobi ISBN: 97 8-1 -1 1 8-0 733 1-5 Set in 9/11 pt Times New Roman by Thomson Digital, Noida,