BOOKS FOR PROFESSIONALS BY PROFESSIONALS ® COMPANION eBOOK US $39.99 Shelve in Mobile Computing User level: Intermediate–Advanced www.apress.com L earn how to create augmented reality apps that unleash the full potential of iOS with Pro iOS 5 Augmented Reality. This book shows you how to use the on-board sensors and camera of your iOS device to enhance the environment around you with integrated facial recognition and social media functionality. Pro iOS 5 Augmented Reality first details the differences in hardware sensors, cameras and more between the iPhone 4 and iPhone 4S, iPod touch, iPad and iPad 2. It then walks you through the foundations of building an augmented reality application for the iPhone or iPad. From using MapKit, to the accelerom- eter and compass, to integrating facial recognition and Facebook data, you’ll learn the building blocks of creating augmented reality applications with the help of engaging case studies. With Pro iOS 5 Augmented Reality you’ll learn how to: • Use MapKit and integrate it into your app • Play and record sound within an augmented reality app • Use the iPhone or iPad camera and video • Program using the accelerometer, gyroscope, and compass • Use cocos2d to overlay a heads-up display on the camera view • Integrate facial recognition into your app • Build an augmented reality feature-rich enterprise game and Facebook apps After reading Pro iOS 5 Augmented Reality, you’ll be able to build augmented reality rich media apps or integrate all the best augmented reality techniques and tools into your existing apps. T urn your ideas for cool, useful apps into reality with Beginning Android 3. This book shows you how to use Android to build Java-based mobile applica- tions for a wide range of phones and devices. Author, Android columnist and developer Mark L. Murphy first explains how to install all the tools you need to become an Android developer. He then walks you through the fundamentals of Android development with details on UI lay- outs, Android’s menu system, widgets and multi-touch. Beginning Android 3 builds your skill-set one app at a time. Once you’ve mastered the basics with the creation of simple apps, it helps you build more advanced apps using Android’s location sensors, rotation detection and local database features. With Beginning Android 3, you’ll learn how to: • Create Flash games and user interfaces using both the Android widget framework and the built-in WebKit-powered Web browser components • Utilize the distinctive capabilities of the Android engine • Use and create Android applications incorporating activities, services, content providers and broadcast receivers • Support Android 3 and earlier, including multiple versions • Build and experience the array of new WebM video and other multimedia APIs. Beginning Android 3 helps you develop the skills that enable your apps to stand out from the crowd, ensuring a great start to your Android coding career. Pro iOS 5 Augmented Reality Kyle Roche Enhance your virtual world with the power of iOS augmented reality Roche Pro iOS 5 Augmented Reality Companion eBook Available SOURCE CODE ONLINE For your convenience Apress has placed some of the front matter material after the index. Please use the Bookmarks and Contents at a Glance links to access them. iii Contents at a Glance Contents v About the Author ix About the Technical Reviewers x Acknowledgments xi Preface xii ■Chapter 1: Introduction 1 ■Chapter 2: Hardware Comparison 15 ■Chapter 3: Using Location Services 31 ■Chapter 4: iOS Sensors 63 ■Chapter 5: Sound and User Feedback 87 ■Chapter 6: Camera and Video Capture 101 ■Chapter 7: Using cocos2D for AR 123 ■Chapter 8: Building a cocos2D AR Game 141 ■Chapter 9: Third-Party Augmented Reality Toolkits 181 ■Chapter 10: Building a Marker-Based AR Application with OpenGL ES 211 ■Chapter 11: Building a Social AR Application 225 ■Chapter 12: Facial-Recognition Techniques 263 ■Chapter 13: Building a Facial Recognition AR App 297 Index 333 1 Chapter Introduction Welcome to Pro iOS 5 Augmented Reality. Augmented reality (AR) has existed in sci-fi movies for decades, is used in the military for head-up displays (HUDs), and until recently, has been a thing of the future. With the upswing in mobile applications since the introduction of the iPhone and the Android operating system, applications such as Layar (www.layar.com), Metaio’s Junaio (www.junaio.com), and Wikitude (www.wikitude.com) have put augmented reality in the hands of the everyday consumer. In this book, I’ll walk you through how to create your own augmented reality applications for iOS. Time magazine named augmented reality among the top-ten technology trends for 2010. Time barely scratched the surface on the potential applications of AR. They selected a few vendor application platforms, such as Layar, and also discussed some more day-to-day applications, such as that employed by the United States Postal Service (USPS). Augmented Reality in the Real and Cyber World The USPS introduced an augmented reality application to its web site in 2010. If you’ve ever mailed something from the post office, you can attest to the fact that quickly selecting a box that fits your needs without holding up the line is a near impossible task. Either you’re stuck wasting a lot of space with a bigger box or you’re holding up the 20 people behind you while you jam all your items into the box that almost fits everything. The USPS took a shot at making this easier, without requiring you to leave your home or office. Basically, you go to the USPS web site (www.prioritymail.com) and use the Virtual Box Simulator and your webcam to try out different box sizes before you head out for the post office. It works like this: Print out a special icon (the USPS eagle) so the simulator knows where to put the hologram of the virtual box. See Figure 1–1. 1. Make sure your webcam is enabled. 1 CHAPTER 1: Introduction 2 2. Launch the Virtual Box Simulator. Put the printed image in the view of the webcam and the simulator puts a hologram of different options for shipping containers around the image. See Figure 1–2. Figure 1–1. This eagle icon is printed and used by the USPS to augment your camera’s view with a simulated shipping container. There are a few basic principles to follow when creating icons or markers for recognition. For traditional markers, you want high-contrast objects that carry a certain uniqueness and aren’t found in common scenarios. In fact, random images are often more effective. Also, you want to use images that have a certain rotation and aren’t symmetrical either horizontally or vertically. This helps the AR program recognize orientation and adjust accordingly. The USPS marker is a good example of these principles. Figure 1–2. The hologram is overlaying the printed icon. CHAPTER 1: Introduction 3 Notice in Figure 1–2 that the simulator allows you to adjust transparency, move your to- be-shipped item on different angles and rotations, and experience exactly which shipping container you need to ship your materials. The USPS uses the marker and some sort of recognition algorithm to find it in the live camera view, track its orientation, and augment the picture with the current box you’ve selected. Pop Culture There are hundreds of other applications for AR in advertising, real estate, the automotive industry, and especially in consumer spending. Although statistics suggest that well over half the population of the United States has tried online shopping, the revenue accounts for only eight percent of consumer spending, according to Wikipedia. Obviously, there are various theories as to why the traction hasn’t taken more market share. Among them are the basic concerns about privacy and security online, but there are equally as many theories on the lack of physical interaction accounting for an unknown product. In some cases, such as with clothing, you just need to see and feel what you’re buying. Sometime in late 2010, we started seeing multiple AR experiences penetrate the retail market. Growing up in the late ’70s, I recall Jane Jetson trying out new hairstyles with the push of a button, or Luke Skywalker listening to the brief about the approach methods for the Death Star over a holographic 3D display. This type of experience is now available for consumers. From trying on new clothing and accessories, to finding out where your grocer’s apples are grown, consider some of these recent examples:  Lego’s Digital Box: An in-store kiosk by Lego lets a child hold up the box set he or she is considering in front of a camera on the kiosk, which then overlays the fully constructed set right on top of the box. The child can move it around, turn it over, and get a feel for whether this is the set they really want to put on their Christmas list.  Zugara: Zugara uses its Magic Mirror, which lets an online shopper stand in front of a webcam and try on different clothing styles, without the aid of a mouse or keyboard. In addition to overlaying the clothes from the online catalog, Zugara overlays controls in the camera’s view so that the user can use gestures to interact with menu options or share their new outfit over their social network.  FoodTracer: This project by Giuseppe Costana uses image recognition in AR to give grocery shoppers more information about the food they are buying. Simply wave a smartphone’s camera in front of the grocer’s shelf and information becomes available. There are obvious advantages and appeals to the interactive experience. However, also consider some of the supplemental values of AR. The back end of most of these applications lives on the cloud. Image-recognition algorithms and the camera’s interpretation itself are primarly running on the device, but advertising data, contextual information, location directories, and other dynamic content linked to the AR view can CHAPTER 1: Introduction 4 be loaded from the cloud and in a centralized location where updates are seamless and the applications can always remain current. Gaming and Location-Based AR Retail and in-store kiosks are not the only places that AR is becoming a trend. Social networks, location-based services, and gaming are leveraging AR as well. Imagine using your camera to interact with the real world in a gaming scenario. I recently saw a demo at a conference in which 3D models of zombies were rendered in the AR view of an iPhone and the user could shoot them by just tapping on the screen. It has spawned a secondary market for accessories like the iPhone gun, covered on www.augmentedplanet.com. This rifle-sized accessory mounts your iPhone to the scope, so you can have a realistic experience of shooting 3D zombies in an AR fashion. In this book, we’ll cover the basics for creating your own AR game. We’ll look at various approaches to this project, including some available SDKs to speed your time to market. Getting Your House in Order There are a few steps you’ll need to take to make sure everything on your machine is ready to go for iOS programming. In this book, we’ll be using Xcode 4.2 only, and we’ll be storing all our projects on GitHub. Xcode shipped with native Git integration for source-code management, so we’ll be taking advantage of that to make things easier and save setup time. Signing Up for GitHub If you already have a GitHub account, you can skip this section. If not, you’re going to need one to download the assets and starting points for each chapter. Open a browser to www.github.com and click the big Signup button in the middle of the page, as shown in Figure 1–3. Figure 1–3. The Signup button is easy to find on GitHub. For this book, we’re going to be accessing the Git repositories that I’ve already set up for each chapter; and, if you’re into sharing, we’ll be posting any variations back for fellow readers. With that in mind, we really only need the “Free for open source” account type. Click the Create a free account button and fill out your information. CHAPTER 1: Introduction 5 Accessing GitHub from Your Machine If you’ve used GitHub before, you may skip this section, which is for users who have not yet created an SSH key for use with GitHub. There are a few ways to access GitHub’s remote repositories from your machine. We’ll be using SSH access, which means we’ll need to generate a token and post it to GitHub. Open Terminal (Applications ➤ Utilities ➤ Terminal) from your Mac. Take a look at Listing 1–1. Follow this same pattern in your Terminal window. I’ll explain the steps next. Listing 1–1. Create Your SSH Key on Your Mac Kyle-Roches-MacBook-Pro-2:~ kyleroche$ cd ~/.ssh Kyle-Roches-MacBook-Pro-2:.ssh kyleroche$ ls known_hosts Kyle-Roches-MacBook-Pro-2:.ssh kyleroche$ ssh-keygen -t rsa -C "kyle@isidorey.com" Generating public/private rsa key pair. Enter file in which to save the key (/Users/kyleroche/.ssh/id_rsa): Enter passphrase (empty for no passphrase): [enter a passphrase here] Enter same passphrase again: [enter your passphrase again] Your identification has been saved in /Users/kyleroche/.ssh/id_rsa. Your public key has been saved in /Users/kyleroche/.ssh/id_rsa.pub. The key fingerprint is: 26:9d:3a:82:fe:r9:gf:ba:39:30:6b:98:16:fe:3b:2c kyle@isidorey.com The key's randomart image is: + [ RSA 2048] + | | | | | | | . . . 4 | |. . . N | |. o +r | | |=. | | + E.o | | +.==oooo | + + Kyle-Roches-MacBook-Pro-2:.ssh kyleroche$ ls id_rsa id_rsa.pub known_hosts The directory listing commands might have different results if you have existing keys already. In this case, you probably want to back up your key directory, just to be safe. First, we’re going to use the ssh-keygen utility to create a public/private rsa key pair. The utility will ask us for a passphrase. This is optional, but passphrases do increase security. Passwords, as most of us realize, aren’t all that secure on their own. Generating a key pair without a passphrase is equivalent to saving your passwords in a plain-text file on your machine. Anyone who gains access can now use your key. If you’re lazy and concerned about typing it in every time, don’t fret. Keychain (since we’re all on a Mac) will allow you to store it after the first time you use this key pair. So, we have a key pair. It’s stored in the newly created id_rsa.pub file you see in your directory listing. Open this file in your favorite plain-text editor and copy all of its contents. It’s important that you copy everything, even the headers. CHAPTER 1: Introduction 6 Return to Github, which should be open to your account in your browser. Open your Account Settings page from the top-left navigation menu. Then open the subtab on the left-hand side called SSH Public Keys. You should see something similar to Figure 1–4. Figure 1–4. Open the SSH Public Keys dialog on GitHub. Find the Add another public key link in the middle of the page. That will open a dialog where you will paste the contents of the id_rsa.pub file we just created. That’s it! You’re now set up in GitHub and your machine can access your repositories using SSH. Because we’ll be using SSH access in this book, let’s quickly set up our default preferences before we move on. We need to configure our local Git client to use the credentials that we received when signing up for GitHub. First, run the following commands from Listing 1–2 in your Terminal window to set some global flags for Git. This, in combination with your SSH keys, will authenticate your Git client to the remote repository. Listing 1–2. Create Your SSH Key on Your Mac Kyle-Roches-MacBook-Pro-2: kyleroche$ git config global user.name "Kyle Roche" Kyle-Roches-MacBook-Pro-2: kyleroche$ git config global user.email "kyle@isidorey.com" CHAPTER 1: Introduction 7 Setting Up Xcode 4.2 and Your Developer Account If you have Xcode 4.2 already set up, you may skip this section. To publish an app to the App Store, you need Xcode and an Apple Developer account. We can take care of both of these steps at the same time. Open your browser to http://developer.apple.com/programs/register/ and click the Get Started button in the header. There are a few paths to follow here. If you want to use an existing Apple ID, you can fill that in and continue along. See Figure 1–5. Alternatively, you can create a new ID for iOS development. That might not seem reasonable, but there are a few pitfalls with using one account. Figure 1–5. Use existing Apple ID or create a new one? NOTE: Choosing whether to consolidate your Apple IDs or create a second one depends on your intent in regards to publishing your apps in the future. Apple has a restriction on which publishing type you can link to an account. There are two ways to publish applications: through the App Store and through Apple’s Enterprise Distribution program. An Apple ID cannot be tied to both publishing methods. Make sure you decide which ID will be responsible for which method of publishing, if you are going to cover both scenarios. If you just want to use your account to develop and debug, then use an existing account. It’s probably the simpliest path. After you are registered, log in to the iOS Dev Center. Find the link for Downloads. At the time of this writing, there are only two choices: Download Xcode 4.2 and a series of links around iAd Producer 1.1. Download Xcode 4.2 to your machine. The download is fairly large. This is one of the drawbacks of Xcode. Each upgraded version, which have started coming more frequently since iOS, requires a new full download of the IDE. [...]... navigated to the new project’s directory and ran the commands shown in Listing 3–1 The finished project for this chapter is available at github.com/kyleroche Listing 3–1 Connect the Local Repository to GitHub Kyle-Roches-MacBook -Pro- 2 :iOS_ AR_Ch3_LocationServices kyleroche$ git remote add origin git@github.com:kyleroche /iOS_ AR_Ch3_LocationServices.git Kyle-Roches-MacBook -Pro- 2 :iOS_ AR_Ch3_LocationServices... (via Wi-Fi or cell radios) can provide magnetometer Checks for the magnetometer (digital compass) hardware Use this to receive heading-related events through the Core Location Framework gamekit Forces/prohibits use of Game Center in iOS 4.1 and later applications microphone Checks for the built-in microphone or accessories that provide a microphone opengles1/opengles-2 Forces/prohibits use of OpenGL... hardware This will all build up to us coding a fully functional augmented- reality application of our own In the next chapter, we’ll start with the views and layouts available to us within iOS, and the ones that we’ll be using for our sample applications 29 Chapter 3 Using Location Services Although this book is essentially about augmented reality, programming maps and location services make up a lot of the... two ways to build augmented- reality applications over the video capabilities of the phone First, you can actively inspect the video capture for elements, recognizable objects, and so forth Or, you can use the video capture as the background for your application, while completely ignoring the contents We see this approach quite a bit in augmented- reality browsers because of the heavy processing involved... out some details appropriate to augmented reality and memory management as they come up Augmented- reality applications use a lot of delegates in classes, and memory management is an important aspect of structuring AR applications correctly Now that we’ve defined a UITextView that we can use for textual updates, let’s create the component in our XIB file Click FirstView.xib in the Project Navigator in... your application’s Info.plist file does this for you The Info.plist file is a standard component of all the iOS templates Xcode 4.2 templates generate this file under the Supporting Files directory in your project It will be named ProjectName-Info.plist, where ProjectName is the name of your Xcode project 27 28 CHAPTER 2: Hardware Comparison To add hardware requirements to your application, you first... file as the checks for location services and the magnetometer, you can simply append the code from Listing 2 5 Listing 2 5 Import the AVFoundation Framework AVAudioSession *audioSession = [AVAudioSession sharedInstance]; BOOL audioAvailable = audioSession.inputIsAvailable; if (audioAvailable) { UIAlertView *alert = [[UIAlertView alloc] initWithTitle:@"Audio" message:@"Audio Available" delegate:self cancelButtonTitle:@"OK"... Checking for Video Capabilities We’ve verified that most of the components we’ll need to build an augmented- reality application are present on the device However, having a camera available doesn’t necessarily mean it will function for video capture And, after all, the video capture is what makes the augmented- reality application so appealing Checking for video capability of the camera is a bit more complicated... repository for this project is not selected, as shown in Figure 1–10 Figure 1–10 Do not create a local Git repository Great! We just created our first project Only a few more steps and we’ll be able to start updating code on GitHub The following steps can probably be completed in various orders I’m following this path because I find it easier As Xcode 4.2 matures, I’m sure we’ll see some improvements on... Lights, Camera, Action In Chapter 5, I cover sound and user feedback Sound isn’t the most prominent feature in AR apps, but it does lead to a better user experience After that, in Chapter 6, we’ll dive into camera and video programming Because AR apps are all overlaid on our camera view, this is an essential chapter to understand before we start constructing the larger AR projects at the end of the book . your Android coding career. Pro iOS 5 Augmented Reality Kyle Roche Enhance your virtual world with the power of iOS augmented reality Roche Pro iOS 5 Augmented Reality Companion eBook Available SOURCE. into your app • Build an augmented reality feature-rich enterprise game and Facebook apps After reading Pro iOS 5 Augmented Reality, you’ll be able to build augmented reality rich media apps. apps that unleash the full potential of iOS with Pro iOS 5 Augmented Reality. This book shows you how to use the on-board sensors and camera of your iOS device to enhance the environment around