Intelligent Image Processing.SteveMann Copyright  2002 John Wiley & Sons, Inc. ISBNs: 0-471-40637-6 (Hardback); 0-471-22163-5 (Electronic) 2 WHERE ON THE BODY IS THE BEST PLACE FOR A PERSONAL IMAGING SYSTEM? This chapter considers the question of where to place the sensory and display appa- ratus on the body. Although the final conclusion is that both should be placed, effectively, right within the eye itself, various other possibilities are considered and explained first. In terms of the various desirable properties the apparatus should be: • Covert: It must not have an unusual appearance that may cause objections or ostracization owing to the unusual physical appearance. It is known, for instance, that blind or visually challenged persons are very concerned about their physical appearance notwithstanding their own inability to see their own appearance. • Incidentalist: Others cannot determine whether or not the apparatus is in use, even when it is not entirely covert. For example, its operation should not convey an outward intentionality. • Natural: The apparatus must provide a natural user interface, such as may be given by a first-person perspective. • Cybernetic: It must not require conscious thought or effort to operate. These attributes are desired in range, if not in adjustment to that point of the range of operational modes. Thus, for example, it may be desired that the apparatus be highly visible at times as when using it for a personal safety device to deter crime. Then one may wish it to be very obvious that video is being recorded and transmitted. So ideally in these situations the desired attributes are affordances rather than constraints. For example, the apparatus may be ideally covert but with an additional means of making it obvious when desired. Such an additional means may include a display viewable by others, or a blinking red light indicating transmission of video data. Thus the system would ideally be operable over a wide range of obviousness levels, over a wide range of incidentalism levels, and the like. 15 16 WHERE ON THE BODY IS THE BEST PLACE FOR A PERSONAL IMAGING SYSTEM? Introduction: Evolution toward Personal Imaging Computing first evolved from large mainframe business machines, to smaller so- called personal computers that fit on our desks. We are now at a pivotal era in which computers are becoming not only pervasive but also even more personal, in the form of miniature devices we carry or wear. An equally radical change has taken place in the way we acquire, store, and process visual photographic information. Cameras have evolved from heavy equipment in mule-drawn carriages, to devices one can conceal in a shirt button, or build into a contact lens. Storage media have evolved from large glass plates to Internet Web servers in which pictures are wirelessly transmitted to the Web. In addition to downsizing, there is a growing trend to a more personal element of imaging, which parallels the growth of the personal computer. The trend can be seen below: • Wet plate process: Large glass plates that must be prepared in a darkroom tent. Apparatus requires mule-drawn carriages or the like for transport. • Dry plates: Premade, individual sheets typically 8 by 10 or 4 by 5 inches are available so it was possible for one person to haul apparatus in a back- pack. • Film: a flexible image recording medium that is also available in rolls so that it can be moved through the camera with a motor. Apparatus may be carried easily by one person. • Electronic imaging: For example, Vidicon tube recording on analog video- tape. • Advanced electronic imaging: For example, solid state sensor arrays, image capture on computer hard drives. • Laser EyeTap: The eye itself is made to function as the camera, to effort- lessly capture whatever one looks at. The size and weight of the apparatus is negligible. It may be controlled by brainwave activity, using biofeedback, so that pictures are taken automatically during exciting moments in life. Originally, only pictures of very important people or events were ever recorded. However, imaging became more personal as cameras became affordable and more pervasive, leading to the concept of family albums. It is known that when there is a fire or flood, the first thing that people will try to save is the family photo album. It is considered priceless and irreplaceable to the family, yet family albums often turn up in flea markets and church sales for pennies. Clearly, the value of one’s collection of pictures is a very personal matter; that is, family albums are often of little value outside the personal context. Accordingly an important aspect of personal imaging is the individuality, and the individual personal value of the picture as a prosthesis of memory. Past generations had only a handful of pictures, perhaps just one or two glass plates that depicted important points in their life, such as a wedding. As cameras became cheaper, people captured images in much greater numbers, but still a WHERE ON THE BODY IS THE BEST PLACE FOR A PERSONAL IMAGING SYSTEM? 17 small enough number to easily sort through and paste into a small number of picture albums. However, today’s generation of personal imaging devices include handheld digital cameras that double as still and movie cameras, and often capture thousands of pictures before any need for any to be deleted. The family of the future will be faced with a huge database of images, and thus there are obvious problems with storage, compression, retrieval, and sorting, for example. Tomorrow’s generation of personal imaging devices will include mass- produced versions of the special laser EyeTap eyeglasses that allow the eye itself to function as a camera, as well as contact lens computers that might capture a person’s entire life on digital video. These pictures will be transmitted wirelessly to friends and relatives, and the notion of a family album will be far more complete, compelling, and collaborative, in the sense that it will be a shared real-time videographic experience space. Personal imaging is not just about family albums, though. It will also radically change the way large-scale productions such as feature-length movies are made. Traditionally movie cameras were large and cumbersome, and were fixed to heavy tripods. With the advent of the portable camera, it was possible to capture real-world events. Presently, as cameras, even the professional cameras get smaller and lighter, a new “point-of-eye” genre has emerged. Sports and other events can now be covered from the eye perspective of the participant so that the viewer feels as if he or she is actually experiencing the event. This adds a personal element to imaging. Thus personal imaging also suggests new photographic and movie genres In the future it will be possible to include an EyeTap camera of sufficiently high resolution into a contact lens so that a high-quality cinematographic experience can be recorded. This chapter addresses the fundamental question as to where on the body a personal imaging system is best located. The chapter follows an organization given by the following evolution from portable imaging systems to EyeTap mediated reality imaging systems: 1. Portable imaging systems. 2. Personal handheld imaging systems. 3. Personal handheld systems with concomitant cover activity (e.g., the VideoClips system). 4. Wearable camera systems and concomitant cover activity (e.g., the wrist- watch videoconferencing computer). 5. Wearable “always ready” systems such as the telepointer reality augmenter. 6. Wearable imaging systems with eyeworn display (e.g., the wearable radar vision system). 7. Headworn camera systems and reality mediators. 8. EyeTap (eye itself as camera) systems. 18 WHERE ON THE BODY IS THE BEST PLACE FOR A PERSONAL IMAGING SYSTEM? 2.1 PORTABLE IMAGING SYSTEMS Imaging systems have evolved from once cumbersome cameras with large glass plates to portable film-based systems. 2.2 PERSONAL HANDHELD SYSTEMS Next these portable cameras evolved into small handheld devices that could be operated by one person. The quality and functionality of modern cameras allows a personal imaging system to replace an entire film crew. This gave rise to new genres of cinematography and news reporting. 2.3 CONCOMITANT COVER ACTIVITIES AND THE VIDEOCLIPS CAMERA SYSTEM Concomitant cover activity pertains generally to a new photographic or video system typically consisting of a portable personal imaging computer system. It includes new apparatus for personal documentary photography and videography, as well as personal machine vision and visual intelligence. In this section a personal computer vision system with viewfinder and personal video annotation capability is introduced. The system integrates the process of making a personal handwritten diary, or the like, with the capture of video, from an optimal point of vantage and camera angle. This enables us to keep a new form of personal diary, as well as to create documentary video. Video of a subject such as an official behind a counter may be captured by a customer or patron of an establishment, in such a manner that the official cannot readily determine whether or not video is being captured together with the handwritten notes or annotations. 2.3.1 Rationale for Incidentalist Imaging Systems with Concomitant Cover Activity In photography (and in movie and video production), as well as in a day-to- day visual intelligence computational framework, it is often desirable to capture events or visual information in a natural manner with minimal intervention or disturbance. A possible scenario to be considered is that of face-to-face conver- sation between two individuals, where one of the individuals wishes to make an annotated video diary of the conversation without disrupting the natural flow of the conversation. In this context, it is desirable to create a personal video diary or personal documentary, or to have some kind of personal photographic or video-graphic memory aid that forms the visual equivalent of what the elec- tronic organizers and personal digital assistants do to help us remember textual or syntactic information. Current state-of-the-art photographic or video apparatus creates a visual disturbance to others and attracts considerable attention on account of the gesture CONCOMITANT COVER ACTIVITIES AND THE VIDEOCLIPS CAMERA SYSTEM 19 of bringing the camera up to the eye. Even if the size of the camera could be reduced to the point of being negligible (e.g., suppose that the whole apparatus is made no bigger than the eyecup of a typical camera viewfinder), the very gesture of bringing a device up to the eye would still be unnatural and would attract considerable attention, especially in large public establishments like department stores, or establishments owned by criminal or questionable organizations (some gambling casinos come to mind) where photography is often prohibited. However, it is in these very establishments in which a visitor or customer may wish to have a video record of the clerk’s statement of the refund policy or the terms of a sale. Just as department stores often keep a video recording of all transactions (and often even a video recording of all activity within the establishment, sometimes including a video recording of customers in the fitting rooms), the goal of the present invention is to assist a customer who may wish to keep a video record of a transaction, interaction with a clerk, manager, refund explanation, or the like. Already there exist a variety of covert cameras such a camera concealed beneath the jewel of a necktie clip, cameras concealed in baseball caps, and cameras concealed in eyeglasses. However, such cameras tend to produce inferior images, not just because of the technical limitations imposed by their small size but, more important, because they lack a viewfinder system (a means of viewing the image to adjust camera angle, orientation, exposure, etc., for the best composition). Because of the lack of viewfinder system, the subject matter of traditional covert cameras is not necessarily centered well in the viewfinder, or even captured by the camera at all, and thus these covert cameras are not well suited to personal documentary or for use in a personal photographic/videographic memory assistant or a personal machine vision system. 2.3.2 Incidentalist Imaging Systems with Concomitant Cover Activity Rather than necessarily being covert, what is proposed is a camera and viewfinder system with “concomitant cover activity” for unobtrusively capturing video of exceptionally high compositional quality and possibly even artistic merit. In particular, the personal imaging device does not need to be necessarily covert. It may be designed so that the subject of the picture or video cannot readily determine whether or not the apparatus is recording. Just as some department stores have dark domes on their ceilings so that customers do not know whether or not there are cameras in the domes (or which domes have cameras and even which way the cameras are pointed where there are cameras in the domes), the “concomitant cover activity” creates a situation in which a department store clerk and others will not know whether or not a customer’s personal memory assistant is recording video. This uncertainty is created by having the camera positioned so that it will typically be pointed at a person at all times, whether or not it is actually being used. What is described in this section is an incidentalist video capture system based on a Personal Digital Assistants (PDA), clipboard, or other handheld devices that 20 WHERE ON THE BODY IS THE BEST PLACE FOR A PERSONAL IMAGING SYSTEM? contain a forward-pointing camera, so that a person using it will naturally aim the camera without conscious or apparent intent. The clipboard version of this invention is a kind of visual equivalent to Stifelman’s audio notebook (Lisa J. Stifelman, Augmenting Real-World Objects: A Paper-Based Audio Notebook, CHI’96 Conference Companion, pp. 199–200, April 1996), and the general ideas of pen-based computing. A typical embodiment of the invention consists of a handheld pen-based computer (see Fig. 2.1) or a combination clipboard and pen-based computer input device (see Fig. 2.2). A camera is built into the clipboard, with the optical axis of the lens facing the direction from bottom to top of the clipboard. During normal face-to-face conversation the person holding the clipboard will tend to point the camera at the other person while taking written notes of the conversation. In this manner the intentionality (whether or not the person taking written notes is intending to point the camera at the other person) is masked by the fact that the camera will always be pointed at the other person by virtue of its placement in the clipboard. Thus the camera lens opening need not necessarily be covert, and could be deliberately accentuated (e.g., made more visible) if desired. To understand why it might be desirable to make it more visible, one can look to the cameras in department stores, which are often placed in large dark smoked plexiglass domes. In this Computer Battery pack Comm. system Camera (main camera) Aux. camera Pen Main screen Aux. screen Ant. Body worn system Figure 2.1 Diagram of a simple embodiment of the invention having a camera borne by a personal digital assistant (PDA). The PDA has a separate display attached to it to function as a viewfinder for the camera. CONCOMITANT COVER ACTIVITIES AND THE VIDEOCLIPS CAMERA SYSTEM 21 Computer Battery pack Comm. system Camera Viewfinder screen Writing surface Paper sheet to conceal screen Pen Ant. Body worn system Figure 2.2 Diagram of an alternate embodiment of the system in which a graphics tablet is concealed under a pad of paper and an electronic pen is concealed inside an ordinary ink pen so that all of the writing on the paper is captured and recorded electronically together with video from the subject in front of the user of the clipboard while the notes are being taken. way they are neither hidden nor visible, but rather they serve as an uncertain deterrent to criminal conduct. While they could easily be hidden inside smoke detectors, ventilation slots, or small openings, the idea of the dome is to make the camera conceptually visible yet completely hidden. In a similar manner a large lens opening on the clipboard may, at times, be desirable, so that the subject will be reminded that there could be a recording but will be uncertain as to whether or not such a recording is actually taking place. Alternatively, a large dark shiny plexiglass strip, made from darkly smoked plexiglass (typically 1 cm high and 22 cm across), is installed across the top of the clipboard as a subtle yet visible deterrent to criminal behavior. One or more miniature cameras are then installed behind the dark plexiglass, looking forward through it. In other embodiments, a camera is installed in a PDA, and then the top of the PDA is covered with dark smoky plexiglass. The video camera (see Fig. 2.1) captures a view of a person standing in front of the user of the PDA and displays the image on an auxiliary screen, which may be easily concealed by the user’s hand while the user is writing or pretending to 22 WHERE ON THE BODY IS THE BEST PLACE FOR A PERSONAL IMAGING SYSTEM? write on the PDA screen. In commercial manufacture of this device the auxiliary screen may not be necessary; it may be implemented as a window displaying the camera’s view on a portion of the main screen, or overlaid on the main screen. Annotations made on the main screen are captured and stored together with videoclips from the camera so that there is a unified database in which the notes and annotations are linked with the video. An optional second camera may be present if the user wishes to make a video recording of himself/herself while recording another person with the main camera. In this way, both sides of the conversation may be simultaneously recorded by the two cameras. The resulting recordings could be edited later, and there could be a cut back and forth between the two cameras to follow the natural flow of the conversation. Such a recording might, for example, be used for an investigative journalism story on corrupt organizations. In the early research prototypes, an additional wire was run up the sleeve of the user into a separate body-worn pack powered by its own battery pack. The body-worn pack typically contained a computer system which houses video capture hardware and is connected to a communications system with packet radio terminal node controller (high-level data link controller with modem) and radio; this typically establishes a wireless Internet connection. In the final commercial embodiment of this invention, the body-worn pack will likely disappear, since this functionality would be incorporated into the handheld device itself. The clipboard version of this invention (Fig. 2.2) is fitted with an electronic display system that includes the capability of displaying the image from the camera. The display serves then as a viewfinder for aiming the camera at the subject. Moreover the display is constructed so that it is visible only to the user of the clipboard or, at least, so that the subject of the picture cannot readily see the display. Concealment of the display may be accomplished through the use of a honeycomb filter placed over the display. Such honeycomb filters are common in photography, where they are placed over lights to make the light sources behave more directionally. They are also sometimes placed over traffic lights where there is a wye intersection, for the lights to be seen from one direction in order that the traffic lights not confuse drivers on another branch of a wye intersection that faces almost the same way. Alternatively, the display may be designed to provide an inherently narrow field of view, or other barriers may be constructed to prevent the subject from seeing the screen. The video camera (see Fig. 2.2) displays on a miniature screen mounted to the clipboard. A folded-back piece of paper conceals the screen. The rest of the sheets of paper are placed slightly below the top sheet so that the user can write on them in a natural fashion. From the perspective of someone facing the user (the subject), the clipboard will have the appearance of a normal clipboard in which the top sheet appears to be part of the stack. The pen is a combined electronic pen and real pen so that the user can simultaneously write on the paper with real ink, as well as make an electronic annotation by virtue of a graphics tablet below the stack of paper, provided that the stack is not excessively thick. In this way there is a computer database linking the real physical paper with CONCOMITANT COVER ACTIVITIES AND THE VIDEOCLIPS CAMERA SYSTEM 23 its pen strokes and the video recorded of the subject. From a legal point of view, real physical pen strokes may have some forensic value that the electronic material may not (e.g., if the department store owner asks the customer to sign something, or even just to sign for a credit card transaction, the customer may place it over the pad and use the special pen to capture the signature in the customer’s own computer and index it to the video record). In this research prototype there is a wire going from the clipboard, up the sleeve of the user. This wire would be eliminated in the commercially produced version of the apparatus, by construction of a self-contained video clipboard with miniature built-in computer, or by use of a wireless communications link to a very small body-worn intelligent image-processing computer. The function of the camera is integrated with the clipboard. This way textual information, as well as drawings, may be stored in a computer system, together with pictures or videoclips. (Hereafter still pictures and segments of video will both be referred to as videoclips, with the understanding that a still picture is just a video sequence that is one frame in length.) Since videoclips are stored in the computer together with other information, these videoclips may be recalled by an associative memory working together with that other information. Thus tools like the UNIX “grep” command may be applied to videoclips by virtue of the associated textual information which typically resides as a videographic header. For example, one can grep for the word “meijer,” and may find various videoclips taken during conversations with clerks in the Meijer department store. Thus such a videographic memory system may give rise to a memory recall of previous videoclips taken during previous visits to this department store, provided that one has been diligent enough to write down (e.g., enter textually) the name of the department store upon each visit. Videoclips are typically time-stamped (e.g., there exist file creation dates) and GPS-stamped (e.g., there exists global positioning system headers from last valid readout) so that one can search on setting (time + place). Thus the video clipboard may be programmed so that the act of simply taking notes causes previous related videoclips to play back automatically in a separate window (in addition to the viewfinder window, which should always remain active for continued proper aiming of the camera). Such a video clipboard may, for example, assist in a refund explanation by providing the customer with an index into previous visual information to accompany previous notes taken during a purchase. This system is especially beneficial when encountering department store representatives who do not wear name tags and who refuse to identify themselves by name (as is often the case when they know they have done something wrong, or illegal). This apparatus allows the user to take notes with pen and paper (or pen and screen) and continuously record video together with the written notes. Even if there is insufficient memory to capture a continuous video recording, the invention can be designed so that the user will always end up with the ability to produce a picture from something that was seen a couple of minutes ago. This 24 WHERE ON THE BODY IS THE BEST PLACE FOR A PERSONAL IMAGING SYSTEM? may be useful to everyone in the sense that we may not want to miss a great photo opportunity, and often great photo opportunities only become known to us after we have had time to think about something we previously saw. At the very least, if, for example, a department store owner or manager becomes angry and insulting to the customer, the customer may retroactively record the event by opening a circular buffer. 2.3.3 Applications of Concomitant Cover Activity and Incidentalist Imaging An imaging apparatus might also be of use in personal safety. Although there are a growing number of video surveillance cameras installed in the environment allegedly for public safety, there have been recent questions as to the true benefit of such centralized surveillance infrastructures. Notably there have been several instances where such centralized infrastructure has been abused by its owners (as in roundups and detainment of peaceful demonstrators). Moreover public safety systems may fail to protect individuals against crimes committed by members of the organizations that installed the systems. Therefore embodiments of the invention often implement the storage and retrieval of images by transmitting and recording images at one or more remote locations. In one embodiment of the invention, images were transmitted and recorded in different countries so that they would be difficult to destroy in the event that the perpetrator of a crime or other misconduct might wish to do so. Moreover, as an artistic tool of personal expression, the apparatus allows the user to record, from a new perspective, experiences that have been difficult to so record in the past. For example, a customer might be able to record an argument with a fraudulent business owner from a very close camera angle. This is possible because a clipboard may be extended outward toward the person without violating personal space in the same way as might be necessary to do the same with a camera hidden in a tie clip, baseball cap, or sunglasses. Since a clipboard may extend outward from the body, it may be placed closer to the subject than the normal eye viewpoint in normal face-to-face conversation. As a result the camera can capture a close-up view of the subject. Furthermore the invention is useful as a new communications medium, in the context of collaborative photography, collaborative videography, and telepresence. One way in which the invention can be useful for telepresence is in the creation of video orbits (collections of pictures that exist in approximately the same orbit of the projective group of coordinate transformations as will be described in Chapter 6). A video orbit can be constructed using the clipboard embodiment in which a small rubber bump is made on the bottom of the clipboard right under the camera’s center of projection. In this way, when the clipboard is rested upon a surface such as a countertop, it can be panned around this fixed point so that video recorded from the camera can be used to assemble a panorama or orbit of greater spatial extent than a single picture. Similarly with the wristwatch embodiment, a small rubber bump on the bottom of the wristband