Mobile 3D Graphics with OpenGL ES and M3G Kari Pulli Tomi Aarnio Ville Miettinen Kimmo Roimela Jani Vaarala AMSTERDAM • BOSTON • HEIDELBERG • LONDON NEW YORK • OXFORD • PARIS • SAN DIEGO SAN FRANCISCO • SINGAPORE • SYDNEY • TOKYO Morgan Kaufmann is an imprint of Elsevier Acquisitions Editor Tiffany Gasbarrini Publishing Services Manager George Morrison Senior Production Editor Paul Gottehrer Cover Design Eric DeCicco Composition diacriTech Interior printer Maple-Vail Book Manufacturing Group Cover printer Phoenix Color Corp. Morgan Kaufmann Publishers is an imprint of Elsevier. 30 Corporate Drive, Suite 400, Burlington, MA 01803, USA This book is printed on acid-free paper. c  2008 by Elsevier Inc. All rights reserved. Designations used by companies to distinguish their products are often claimed as trademarks or registered trademarks. In all instances in which Morgan Kaufmann Publishers is aware of a claim, the product names appear in initial capital or all capital letters. Readers, however, should contact the appropriate companies for more complete information regarding trademarks and registration. No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means—electronic, mechanical, photocopying, scanning, or otherwise—without prior written permission of the publisher. Permissions may be sought directly from Elsevier’s Science & Technology Rights Department in Oxford, UK: phone: (+44) 1865 843830, fax: (+44) 1865 853333, E-mail: permissions@elsevier.com. You may also complete your request online via the Elsevier homepage (http://elsevier.com), by selecting “Support & Contact” then “Copyright and Permission” and then “Obtaining Permissions.” Library of Congress Cataloging-in-Publication Data Application submitted ISBN: 978-0-12-373727-4 For information on all Morgan Kaufmann publications, visit our Web site at www.mkp.com or www.books.elsevier.com Printed in the United States of America 0708091011 54321 Contents Preface xiii About the Authors xv CHAPTER 1. INTRODUCTION 1 1.1 About this Book 2 1.1.1 Typographic Conventions 3 1.2 Graphics on Handheld Devices 3 1.2.1 Device Categories 4 1.2.2 Display Technology 5 1.2.3 Processing Power 6 1.2.4 Graphics Hardware 8 1.2.5 Execution Environments 9 1.3 Mobile Graphics Standards 12 1.3.1 Fighting the Fragmentation 12 1.3.2 Design Principles 14 1.3.3 OpenGL ES 18 1.3.4 M3G 19 1.3.5 Related Standards 21 PART I ANATOMY OF A GRAPHICS ENGINE CHAPTER 2. LINEAR ALGEBRA FOR 3D GRAPHICS 27 2.1 Coordinate Systems 27 2.1.1 Vectors and Points 29 2.1.2 Vector Products 29 2.1.3 Homogeneous Coordinates 31 2.2 Matrices 31 2.2.1 Matrix Products 32 2.2.2 Identity and Inverse 33 2.2.3 Compound Transformations 33 2.2.4 Transforming Normal Vectors 34 v vi CONTENTS 2.3 Affine Transformations 35 2.3.1 Types of Affine Transformations 35 2.3.2 Transformation Around a Pivot 39 2.3.3 Example: Hierarchical Modeling 39 2.4 Eye Coordinate System 42 2.5 Projections 44 2.5.1 Near and Far Planes and the Depth Buffer 45 2.5.2 A General View Frustum 47 2.5.3 Parallel Projection 50 2.6 Viewport and 2D Coordinate Systems 51 CHAPTER 3. LOW-LEVEL RENDERING 55 3.1 Rendering Primitives 57 3.1.1 Geometric Primitives 57 3.1.2 Raster Primitives 60 3.2 Lighting 61 3.2.1 Color 61 3.2.2 Normal Vectors 63 3.2.3 Reflection Models and Materials 64 3.2.4 Lights 68 3.2.5 Full Lighting Equation 70 3.3 Culling and Clipping 70 3.3.1 Back-Face Culling 71 3.3.2 Clipping and View-Frustum Culling 71 3.4 Rasterization 73 3.4.1 Texture Mapping 74 3.4.2 Interpolating Gradients 82 3.4.3 Texture-Based Lighting 83 3.4.4 Fog 88 3.4.5 Antialiasing 90 3.5 Per-Fragment Operations 92 3.5.1 Fragment Tests 92 3.5.2 Blending 95 3.5.3 Dithering, Logical Operations, and Masking 99 3.6 Life Cycle of a Frame 100 3.6.1 Single versus Double Buffering 101 3.6.2 Complete Graphics System 101 3.6.3 Synchronization Points 102 CHAPTER 4. ANIMATION 105 4.1 Keyframe Animation 105 4.1.1 Interpolation 106 4.1.2 Quaternions 111 CONTENTS vii 4.2 Deforming Meshes 113 4.2.1 Morphing 113 4.2.2 Skinning 114 4.2.3 Other Dynamic Deformations 116 CHAPTER 5. SCENE MANAGEMENT 117 5.1 Triangle Meshes 118 5.2 Scene Graphs 120 5.2.1 Application Area 120 5.2.2 Spatial Data Structure 121 5.2.3 Content Creation 123 5.2.4 Extensibility 125 5.2.5 Class Hierarchy 125 5.3 Retained Mode Rendering 128 5.3.1 Setting Up the Camera and Lights 129 5.3.2 Resolving Rendering State 130 5.3.3 Finding Potentially Visible Objects 130 5.3.4 Sorting and Rendering 132 CHAPTER 6. PERFORMANCE AND SCALABILITY 133 6.1 Scalability 134 6.1.1 Special Effects 135 6.1.2 Tuning Down the Details 136 6.2 Performance Optimization 136 6.2.1 Pixel Pipeline 137 6.2.2 Ver tex Pipeline 139 6.2.3 Application Code 140 6.2.4 Profiling OpenGL ES Based Applications 141 6.2.5 Checklists 142 6.3 Changing and Querying the State 145 6.3.1 Optimizing State Changes 146 6.4 Model Data 146 6.4.1 Ver tex Data 147 6.4.2 Triangle Data 148 6.5 Transformation Pipeline 148 6.5.1 Object Hierarchies 148 6.5.2 Rendering Order 149 6.5.3 Culling 150 6.6 Lighting 151 6.6.1 Precomputed Illumination 151 viii CONTENTS 6.7 Textures 152 6.7.1 Texture Storage 152 PART II OPENGL ES AND EGL CHAPTER 7. INTRODUCING OPENGL ES 157 7.1 Khronos Group and OpenGL ES 157 7.2 Design Principles 158 7.3 Resources 159 7.3.1 Documentation 160 7.3.2 Technical Support 160 7.3.3 Implementations 160 7.4 API Overview 161 7.4.1 Profiles and Versions 161 7.4.2 OpenGL ES 1.0 in a Nutshell 161 7.4.3 New Features in OpenGL ES 1.1 164 7.4.4 Extension Mechanism 165 7.4.5 OpenGL ES Extension Pack 166 7.4.6 Utility APIs 166 7.4.7 Conventions 167 7.5 Hello, OpenGL ES! 170 CHAPTER 8. OPENGL ES TRANSFORMATION AND LIGHTING 173 8.1 Drawing Primitives 173 8.1.1 Primitive Types 174 8.1.2 Specifying Vertex Data 177 8.1.3 Drawing the Primitives 179 8.1.4 Ver tex Buffer Objects 180 8.2 Vertex Transformation Pipeline 183 8.2.1 Matrices 183 8.2.2 Transforming Normals 185 8.2.3 Texture Coordinate Transformation 186 8.2.4 Matrix Stacks 188 8.2.5 Viewport Transformation 188 8.2.6 User Clip Planes 189 8.3 Colors and Lighting 189 8.3.1 Specifying Colors and Materials 189 8.3.2 Lights 190 8.3.3 Two-Sided Lighting 192 8.3.4 Shading 193 8.3.5 Lighting Example 193 CONTENTS ix CHAPTER 9. OPENGL ES RASTERIZATION AND FRAGMENT PROCESSING 195 9.1 Back-Face Culling 195 9.2 Texture Mapping 196 9.2.1 Texture Objects 196 9.2.2 Specifying Texture Data 197 9.2.3 Texture Filtering 202 9.2.4 Texture Wrap Modes 205 9.2.5 Basic Texture Functions 205 9.2.6 Multi-Texturing 206 9.2.7 Texture Combiners 207 9.2.8 Point Sprite Texturing 209 9.2.9 Implementation Differences 209 9.3 Fog 210 9.4 Antialiasing 211 9.4.1 Edge Antialiasing 211 9.4.2 Multisampling 212 9.4.3 Other Antialiasing Approaches 213 9.5 Pixel Tests 214 9.5.1 Scissoring 214 9.5.2 Alpha Test 214 9.5.3 Stencil Test 215 9.5.4 Depth Testing 218 9.6 Applying Fragments to the Color Buffer 218 9.6.1 Blending 219 9.6.2 Dithering 220 9.6.3 Logic Ops 220 9.6.4 Masking Frame Buffer Channels 220 CHAPTER 10. MISCELLANEOUS OPENGL ES FEATURES 223 10.1 Frame Buffer Operations 223 10.1.1 Clearing the Buffers 223 10.1.2 Reading Back the Color Buffer 224 10.1.3 Flushing the Command Stream 225 10.2 State Queries 225 10.2.1 Static State 226 10.2.2 Dynamic State Queries 227 10.3 Hints 233 10.4 Extensions 234 10.4.1 Querying Extensions 234 10.4.2 Query Matrix 234 10.4.3 Matr ix Palette 235 10.4.4 Draw Texture 238 10.4.5 Using Extensions 238 x CONTENTS CHAPTER 11. EGL 241 11.1 API Overview 242 11.2 Configuration 244 11.3 Surfaces 248 11.4 Contexts 252 11.5 Extensions 253 11.6 Rendering into Textures 254 11.7 Writing High-Performance EGL Code 255 11.8 Mixing OpenGL ES and 2D Rendering 257 11.8.1 Method 1: Window Surface is in Control 257 11.8.2 Method 2: Pbuffer Surfaces and Bitmaps 258 11.8.3 Method 3: Pixmap Surfaces 258 11.9 Optimizing Power Usage 259 11.9.1 Power Management Implementations 259 11.9.2 Optimizing the Active Mode 261 11.9.3 Optimizing the Idle Mode 262 11.9.4 Measuring Power Usage 262 11.10 Example on EGL Configuration Selection 264 PART III M3G CHAPTER 12. INTRODUCING M3G 269 12.1 Overview 270 12.1.1 Mobile Java 270 12.1.2 Features and Structure 272 12.1.3 Hello, World 276 12.2 Design Principles and Conventions 277 12.2.1 High Abstraction Level 278 12.2.2 No Events or Callbacks 279 12.2.3 Robust Arithmetic 280 12.2.4 Consistent Methods 281 12.2.5 Parameter Passing 282 12.2.6 Numeric Values 283 12.2.7 Enumerations 284 12.2.8 Error Handling 284 12.3 M3G 1.1 285 12.3.1 Pure 3D Rendering 285 12.3.2 Rotation Interpolation 285 12.3.3 PNG and JPEG Loading 286 12.3.4 New Getters 287 12.3.5 Other Changes 288 CONTENTS xi CHAPTER 13. BASIC M3G CONCEPTS 289 13.1 Graphics3D 290 13.1.1 Render Targets 290 13.1.2 Viewport 293 13.1.3 Rendering 294 13.1.4 Static Properties 296 13.2 Image2D 297 13.3 Matrices and Transformations 300 13.3.1 Transform 300 13.3.2 Transformable 303 13.4 Object3D 306 13.4.1 Animating 306 13.4.2 Iterating and Cloning 306 13.4.3 Tags and Annotations 308 13.5 Importing Content 311 13.5.1 Loader 311 13.5.2 The File Format 313 CHAPTER 14. LOW-LEVEL MODELING IN M3G 319 14.1 Building meshes 319 14.1.1 VertexArray 319 14.1.2 VertexBuffer 320 14.1.3 IndexBuffer and Rendering Primitives 323 14.1.4 Example 325 14.2 Adding Color and Light: Appearance 326 14.2.1 PolygonMode 327 14.2.2 Material 328 14.2.3 Texture2D 329 14.2.4 Fog 332 14.2.5 CompositingMode 333 14.3 Lights and Camera 337 14.3.1 Camera 337 14.3.2 Light 339 14.4 2D Primitives 343 14.4.1 Background 343 14.4.2 Sprite3D 346 CHAPTER 15. THE M3G SCENE GRAPH 349 15.1 Scene Graph Basics: Node, Group, and World 349 15.2 Mesh Objects 351 15.3 Transforming Objects 354 15.3.1 Camera, Light, and Viewing Transformations 355 15.3.2 Node Alignment 356 xii CONTENTS 15.4 Layering and Multi-Pass Effects 360 15.5 Picking 362 15.6 Optimizing Performance 364 15.6.1 Visibility Optimization 365 15.6.2 Scope Masks 365 CHAPTER 16. ANIMATION IN M3G 367 16.1 Keyframe Animation: KeyframeSequence 367 16.2 Animation Targets: AnimationTrack 372 16.3 Timing and Speed: AnimationController 374 16.4 Animation Execution 377 16.5 Advanced Animation 378 16.5.1 Deformable Meshes 378 16.5.2 Animation Blending 385 16.5.3 Creating Discontinuities 387 16.5.4 Dynamic Animation 388 PART IV APPENDIX A FIXED-POINT MATHEMATICS 393 A.1 Fixed-Point Methods in C 395 A.1.1 Basic Operations 395 A.1.2 Shared Exponents 397 A.1.3 Trigonometric Operations 399 A.2 Fixed-Point Methods in Assembly Language 400 A.3 Fixed-Point Methods in Java 405 B JAVA PERFORMANCE TUNING 407 B.1 Virtual Machines 408 B.2 Bytecode Optimization 409 B.3 Garbage Collection 410 B.4 Memory Accesses 411 B.5 Method Calls 413 C GLOSSARY 415 Bibliography 419 Index 425 [...]... smart phones, and both OpenGL ES and M3G are typically available for 3D content 1.2.2 DISPLAY TECHNOLOGY The evolution of mobile phones coincides with the evolution of digital photography Digital cameras started the demand for small, cost-efficient, low-power, high-quality displays Mobile phones were able to leverage that demand, and soon small-display technology was being driven by mobile phones and, eventually,... version, OpenGL ES 2.0, provides a completely revamped API, and support for a high-level shading language (GLSL ES) : it replaces several stages of the traditional fixed-function graphics pipeline with programmable vertex and fragment shaders, and is therefore not backward-compatible with the 1.x series The 1.x and 2.x generations of OpenGL ES continue to coexist, together providing 3D graphics capabilities... SECTION 1.3 MOBILE GRAPHICS STANDARDS 21 and improving performance on the mass-market devices that do not have programmable graphics hardware, or any graphics hardware at all 1.3.5 RELATED STANDARDS There are several mobile graphics and multimedia standards closely related to OpenGL ES and M3G This book concentrates only on graphics APIs, but for sound and multimedia in general, you can refer to standards... extended with new functions and tokens in GL10 and GL11, for OpenGL ES versions 1.0 and 1.1, respectively Several OpenGL ES extensions are also exposed in the API, so features beyond the core functionality can be accessed Being a Java API, JSR 239 extends the error handling from native OpenGL ES with additional exceptions to catch out-of-bounds array accesses and other potential risks to system security and. .. these advances have resulted in a truly mobile computer As a side effect, real-time, interactive 3D graphics has become feasible and increasingly desirable for the masses 1.1 ABOUT THIS BOOK This book is about writing real-time 3D graphics applications for mobile devices We assume the reader has some background in mathematics, programming, and computer graphics, but not necessarily in mobile devices... embedded devices from wristwatches to smart phones, modern games consoles, and beyond All OpenGL ES 2.x devices are expected to ship with ES 1.1 drivers Details of the 2.x standard are beyond the scope of this book GLSL ES is closely related to the OpenGL Shading Language, well described by Rost [Ros04] A companion API called EGL, described in Chapter 11, handles the integration of OpenGL ES into the... Java 3D and OpenInventor It consists of nodes that encapsulate 3D graphics elements The nodes can be connected to form a scene graph representing the graphics objects and their relationships M3G is designed so that it can be efficiently implemented on top of an OpenGL ES renderer Standardized high-level APIs have never been as popular on desktop as low-level ones The main reason is that a high-level... and several appendices Part I gives an introduction to the 3D graphics concepts that are needed to understand OpenGL ES and M3G, which are then covered in Parts II and III, respectively The use of each API is demonstrated with hands-on code examples The appendices provide additional information and optimization tips for both C/C++ and Java developers as well as a glossary of acronyms and terms used in... necessarily in mobile devices The 3D graphics capabilities of mobile devices are exposed through two standardized application programming interfaces (APIs): OpenGL ES, typically accessed through C or C++, and M3G, for mobile Java We introduce the latter standard in terms of the former As OpenGL ES is utilized as the fundamental building block in many real-world M3G implementations, expressing this relationship... includes proprietary OpenGL ES bindings 2D vector graphics The variety of screen resolutions on mobile devices creates a problem for 2D content If graphics are rendered and distributed as bitmaps, chances are that the resolution of the content is different from the screen resolution of the output device Resampling the images to different resolutions often degrades the quality—text especially becomes blurry . definition of OpenGL ES standard. He also headed the team that implemented and integrated Nokia’s first OpenGL ES and EGL solution. xv Acknowledgments The creation and adoption of OpenGL ES and M3G was. real-time 3D graphics applications for mobile devices. We assume the reader has some background in mathematics, programming, and computer graphics, but not necessarily in mobile devices. The 3D graphics. phones, and both OpenGL ES and M3G are typically available for 3D content. 1.2.2 DISPLAY TECHNOLOGY The evolution of mobile phones coincides with the evolution of digital photography. Digi- tal