31 SoundScapes/Artabilitation – Evolution of a Hybrid Human Performance 695 resonance. This aesthetic is an inner resonance recognized from achieved user- experience. Having experienced a vocation as a stage performer I relate to emotions achieved through a satisfying entertainment of an audience. Thus, in the therapy the user entertains him or her-self through the mediating interactive digital media system. This can be from controlling a game or creating art, i.e. music making or painting via digital tools, and is evident as FUN resulting from the created situ- ations. Facilitator knowledge of the potentials of the digital media is essential to user-experience, successful guidance (targeted as via mediating technology), and user self-determined entertainment. The same system that was created specific for the rehabilitation training was used by me to perform abstract expressionistic stage art and interactive installations at various large exhibitions. These include ongoing tours in Museums of Modern Art as well as at International-National Cultural events (e. g., Danish NeWave, New York 1999; Cultural Olympiad 1996, 2000; and European Cultural Capital events 1996, 2000). Underground events were as informative to the concept as the larger venues (see next section). Insight was gained from these situations. By combining self-reflection with collaborative inter-/multi-disciplinary reflective analysis of the human performance situations a significant input to designing the system, facilitat- ing sessions, and evaluating outcomes evolved. Underground Non-formal Learning An example of how learning was apparent from the entertainment and arts is where I directed, produced and performed in a one-man-show realized as the inaugural Aarhus Festival Fringe (‘Festuge’). This 1999 event took place in an emptied storage rooms that were adjacent to my sizable interactive room installation titled Circle of Interactive Light (COIL), hosted by the Radisson SAS Scandinavia Hotel, Aarhus, Denmark. The Festival committee had declined my proposal to bring together East and West German artists to feature alongside North and South Korean artists as too politically sensitive. The festival theme was ‘THE WALL, Ten Years After’ – and my proposal involved artists of distinction from both sides of the Berlin Wall and the DMZ to discus and showcase how political issues affect art and artists lives. Understandable that the committee declined! The eventual “FESTUGE FRINGE” protest performance featured my digital in- teractive media in the form of the motion sensitive system utilized in various ways with projections inside and outside the human body. The final section of the perfor- mance was where I used fourteen infrared sensors to control one sound and a library of image manipulations. The purpose was to explore non-control and subliminal performance by enabling the feedback to control me and this was done by mapping the data to maximum parameter control of a specific aspect of the sound envelope. In four of the twelve performances I achieved what I refer to as non-control with subliminal interaction. This was where I, as performer, experienced being as one with the feedback stimuli. If conscious intent or query was involved, the experience 696 A.L. Brooks and higher-state of interaction (or inbetweenness – see Kidd) was lost. I relate this targeted non-control and interaction to the therapy work with profoundly disabled where often the user’s abilities are of a non-controlled state so my learning experi- ences as performer/user assist by informing my role as facilitator. Following the performances the wall would open to exhibit the technology used, and then audience walked through the wall into the COIL exhibition space where they could experience the virtual interactive spaces and debate over a drink in the caf ´ e I had built for the occasion. The piece titled “Behind the Wall” targeted beyond Berlin and DMZ issues as it reflected on human barriers across ages and cultures – and especially committees. The next section introduces the ArtAbilitation 2 workshops from 2007 and 2008. The ArtAbilitation movement has evolved from the SoundScapes work. It is where digital interactive media is used to create an entertaining user-experience. ArtAbilitation Workshops, Casa da Musica, Porto, Portugal The “Ao Alcance de Todos - Within Everyone’s Reach” festival was hosted at Casa da M ´ usica (Figure 13), Porto, Portugal in April 2008. Eight one hour workshops were held with 144 disabled children and adults attending with caregivers. The workshops were created in a room 238 square meters floor area and approximately 20 meters high. Additionally, a symposium for professionals was hosted immedi- ately following the workshops for 35 professionals (international, national, regional, and local // social workers, psychologists, researchers, teachers, and students :::) - many of whom had attended the workshops. This event was a continuation of the previous year’s inaugural event where the theme was ‘Music, Technology and Disability’, and where six workshops were attended by 91 attendees including 61 from special care institutes, of which 39 had profound disability. 30 student music teachers also attended. A local crew assists in the realization of the workshops (see Petersson and Brooks 2007; Brooks 2008). This section does not detail the workshops but rather it exemplifies the design issues where digital interactive media, entertainment and the arts are combined to result in various user-experiences that inform the ongoing research and refinement of the acknowledged international ArtAbilitation movement (Wiederhold 2007). I begin by giving an overview of commonalities in designing both workshops. The 2007 workshop is then detailed in more depth followed by the 2008 workshop. Both workshop designs targeted participant learning from interactive experiences where active participation formed a context for meaning. Offered were opportuni- ties for augmenting learning and awareness. Observations were of self-determined active participation that established goals that in turn engaged interest, curiosity and play. In this way, motivation to achieve goals is considered intrinsic rather than extrinsic, which can result from an activity, task or goal introduced by someone else, 2 http://www.ArtAbilitation.net 31 SoundScapes/Artabilitation – Evolution of a Hybrid Human Performance 697 Fig. 13 Casa da Musica, Porto, Portugal (photo with permission Casa da Musica) and or with possibly over zealous guidance/facilitation. Contextual design thus be- came a way to configure learning resources and interaction (Kress & van Leeuwen, 2001). Exploration, play and transformation were also targeted in both events such that each participant had an ‘action-stillness’ profile that evolved through inter- actions. Each cycle of activity was considered a new creation that contributed to form patterning of actions resulting from the activity. Each activity was designed to increment challenges of confrontation. The resultant ‘play’ and created ‘creativ- ity’ scenarios involved manipulation of provided tools. The manipulation required a degree of competence that was learnt through exploration of the tool’s traits (Bruner 1972). Absence of negative consequences seemingly encouraged participant free ex- pressive explorations, which in turn, can over time result in development of un- employed skill (Beach, 1942). Such development is significant even if it is at a micro scale, i.e. micro-development (Yan and Fischer 2002). Responsive environ- ment composition and learning process/outcome evolve as the assessment focus instrument of observers. Realized was new learning spaces and approaches to learn- ing and rehabilitation through emphasizing user’s creation of meaning via serious play. The approach is that no aspect of the learning process and outcome is taken for granted. Rather it is formulated into play and creative activities that are inherent to e.g., games and art making. The activities of play and creativity conceal the embed- ded learning and training involved for the user. An emphasis is on a supplementing tool for traditional practices rather than a replacement. In this sense, learning is at a ‘subliminal’ level for the user as he or she engages in the responsive environment. Thus, motivation is optimized through action and stillness cycles where the user 698 A.L. Brooks Fig. 14 Sala de Ensaio 1, Casa da M´usica, Oporto, Portugal (C Dcamera) – (Brooks 2008) iteratively explores and transforms the feedback media. This process contains choices and decisions that indicate learning, e.g. in the form of increased repertoire of expressions, changes of skills, and new patterns of social interaction (Brooks 2008). In the 2007 workshop a private (2) and a public (1) area were designed – see Sala de Ensaio diagram (Figure 14). This was inspired from the evaluation consul- tations mentioned earlier in this chapter where a caregiver experienced that a private space, without any intervention, was desiredbefore a user felt comfortable enough to express through the SoundScapes system. My previous research (Brooks et al. 2002) presented how certain individuals prefer to explore, play and create without any oth- ers being present (Figures 15–16). Exemplifying the SoundScapes open system and concept a digital video camera was used to create a feedback loop play-space where participant gesture distorted image silhouette and color change to a RGB lighting system that reacted to voice input. Interactions in the private space were video taped and analyzed as achieving a positive flow state (Csikszentmih ´ alyi 1996). It was observed in the private space that the chain of exploration-play-creation began with a curiosity that evolved out of the isolation and initial stillness that was first encountered within the created environment. Thereby, stillness became part of the action and vice versa. Indications pointed to how interactive play and creativity that offered choices between interaction and rest in a silent space enhanced the sense of control. A created public space (Figures 17–18) questioned participant perception and awareness where peer-support and scaffolding of exploration was evident. Over- head infrared camera tracking was mapped to auditory music making (Figure 17) 31 SoundScapes/Artabilitation – Evolution of a Hybrid Human Performance 699 Figs. 15-16 Private space images of total engagement from ArtAbilitation 2007, (photo credit Augusto Brazio, Casa da Musica) and image effects (Figure 18). Many results occurred from this space. One memo- rable instance was when a woman who was profoundly physically disabled insisted in exploring the interactive space out of her wheelchair – evident was that crawling around required immense effort (Figure 18). Her motions were tracked to open up a digital mask that concealed an image of a famous Portuguese footballer. Her disabled colleagues were supporting by shouting instructions. Motivation was stag- gering to achieve the whole erosion of the mask. Another instance was where an autistic group was in the workshop and one of them began to move and open up the same mask. The first exposed facial element of the footballer was the chin and im- mediately another autistic male shouted out the name of the recognized footballer. In both instances (and others) the public virtual interactive space empowered imme- diate learning to be exhibited (Petersson and Brooks 2007). Five physical ‘VIS rooms’ were created in the 2008 workshop. Each was de- signed to offer differing experiences of empowerment through motion creating images and auditory feedback. Large sub-woofer speakers positioned on the wooden floor gave tactile stimulation which was especially important for the attending class of young deaf children. Findings were that the interactive spaces offered a place where comprehension of tasks could be shared from the strong to the weak in each group. Again wheelchairs were discarded to explore by crawling around the spaces. 700 A.L. Brooks Figs. 17-18 Public interactive space: two designs utilizing overhead camera tracking Targeted tangible outcome was that each participant received a painting that they created through gesture. Reports of how many showed these paintings to family, friends and non-attending staff stated to the powerful effect. Both workshops exhibited action-stillness cycles, which consisted of iterative loops of exploring and transforming, constitute one part of a theoretical map for the purposes of analyzing critical incidents in a non-formal learning process. These cycles are related to the user’s learning experience. The other part of the theoretical map concerns design issues in the form of use qualities relative to the user’s interac- tive experience; transparency, social-action space, user control/autonomy, pliability, playability and seductivity (L ¨ owgren and Stolterman 2005). During a whole session 31 SoundScapes/Artabilitation – Evolution of a Hybrid Human Performance 701 cycle of action and stillness, facilitators have the possibility to reflect upon the in- dications of learning that occur during the process. The user profile influences the facilitator’s decisions on how to set up the attributes of the responsive environments relative to the desired learning process and the expected outcome of that process. Once these prerequisites are set, the user is expected to ‘experience from the inter- action and learn from the experience’. Hence, the aim of the workshop events was to investigate the user’s performance in using responsive environments designed to engage participants in experience- learning through action and stillness cycles. A sub-question concerns the ability of perception and the associated learning curve of the attendees with cognitive disabil- ities to be able to easily correlate across dimensions of scale and plane – a matter which influences the participant’s interactive as well as learning experience. The final reported event is one that was built upon previous work where third party performance gesture created visuals to complement classical music situations. Visualizing classical music The previous sections present the motion-sensitive environment and its use as a re- habilitation training supplement within the community of people with a disability, both profound from birth, and acquired. The use of the created interactive system is presented in intimate institute settings for individuals and a later section shares two examples of larger workshop situations where groups attended. This section exem- plifies how a similar concept and technique of using digital interactive media was implemented to elicit dynamic performance data from a situation that featured the ‘Orquestra Nacional do Porto’ - regarded by many as The Portuguese National Or- chestra. This was again in Casa da Musica, Porto, Portugal but this time in the main 1200 seat state-of-the-art auditorium. See publication Interpretation 3 (Brooks 2008). A goal of the exploratory study was to dynamically complement the music by offering an experience of inter/multi-sensory stimulation for both audience and per- formers. This built upon my earlier work in Auckland, New Zealand, and Aarhus, Denmark where the different situations, one with orchestra and one with choir, both had a similar stimulation goal. An aim of this work is to offer inclusive access so that even people who are deaf may have an opportunity to appreciate classical mu- sic. Important to mention is how in the choir study three sensors mapped to RGB enabled the conductor to paint the choir backdrop through his gestured interactions with the singers. The session was in the Danish Radio TV studio and six ‘takes’ were recorded. It was evident that his gestures were expanding as he became used to the interaction and – in his own words - “how it felt that the air around my body was activated”. This expansion of gesture relates to the work in the disability sector where targeted in subliminal motivated augmented motion. The following Figs.19–26 illustrate the result from the visualization experiment. Overhead cameras and stage sensors sourced the performance data and this was 3 http://www.icdvrat.reading.ac.uk/2008/interpretations.htm 702 A.L. Brooks Fig. 19 Data from stage performance – conductor gesture, section/musician expressiveness, lights and music - sourced to dynamically affect the synchronized visuals for environment change in real- time (photo credit Jo˜ao Messias, Casa da Musica) Fig. 20 Sourced stage data mixed and matched to digitally mirror performance in abstract – here camera and music stand lights - creating and effecting projected visuals (photo credit Jo˜ao Messias, Casa da Musica) 31 SoundScapes/Artabilitation – Evolution of a Hybrid Human Performance 703 Fig. 21 Projected moving image dynamically matched to digitally mirror performance (photo credit Jo ˜ ao Messias, Casa da Musica) mapped by created algorithms to effect the environment variables in real-time to complement the music. A technical deficit proved the biggest hurdle and results were unsatisfactory as image mixing was via switching instead of faders on a mixer. Thus, nuance of performance was unable to be matched as I desired. However, as an experiment to explore the concept it did build my knowledge of the concept. The work is planned to evolve further locally for the next phase of experiments due to the problems of accessing the specialized equipment required for technical setup. The work has been accepted for the Cultural Olympiad 2012 London. 704 A.L. Brooks Fig. 22 Projected image (center of screen) shows data capture from camera overhead (photo credit Jo ˜ ao Messias, Casa da Musica) Conclusions and Future Directions This article informs of digital interactive media sculpted into a responsive ges- ture driven environment that is used within rehabilitation training. Entertainment achieved through a user creating and playing acts as a stimulant towards augmenting life quality and motivating participation in training that otherwise is boring, tedious and mundane – often for both patient and therapist/facilitator. Artistic composition in the form of music-making and digital painting is empowered through an interface and multi-sensory virtual environment content that is adaptive and flexible enough to address idiosyncratic needs. However, whilst I create for another’s creativity, my [...]... how much information to give, with what detail and composition, and F Sparacino ( ) Sensing Places and MIT, Santa Monica, CA, USA e-mail: flavia@sensingplaces.com B Furht (ed.), Handbook of Multimedia for Digital Entertainment and Arts, DOI 10.1007/978-0-387-89024-1 32, c Springer Science+Business Media, LLC 2009 713 714 F Sparacino how to sequence and articulate various fragments A lack of interpretive... recognition using HMMs Brand, Oliver, and Pentland [5] have demonstrated the higher performance of coupled HMMs for tasks which require gesturing with both hands at the same time Campbell and others [8] have studied the effects of the appropriate feature choice for a gesture recognition task, using stereo vision The author has developed a variety of natural interfaces for art and entertainment Installations... (Figs 2 and 3) The 3D web landscape is a city Known cities’ layout, architecture, and landmarks are input to the program and are used as orientation cues and organizing geometry This virtual internet world grows dynamically as new information is 722 Fig 2 Aerial view of City of News Fig 3 City of News after exploration F Sparacino 32 Designing for Architecture and Entertainment 723 Fig 4 User standing... quantitatively, the right hand acts as a 3D pointer and traces the 3D shape of known objects The error of reconstruction of a hand position is on the order of 2 to 3cm when the user is up to 4 meters away from the screen This error is due both to estimation error and the crudeness of using the hand position to represent a point in space Fig 6 3D tracking of user’s hands and head, view from the left and right camera... Pfinder (person finder), and the real time 32 Designing for Architecture and Entertainment 717 blob tracking approach associated with it and described in section 32 [46] Stereo tracking of pointing and command gestures, and HMM based gesture recognition is discussed in [40] Jojic, Brumitt, and Meyers [20] also use stereo cameras to detect people pointing and estimate the direction of their pointing As... for architecture and entertainment The main contribution of this paper is to show that Bayesian networks are an ideal modeling tool for all three forms of space intelligence and to provide a unifying mathematical description for robust sensing, user and context modeling, and articulated information delivery (storytelling) Related Work The work here presented is highly interdisciplinary and it draws from... complexity, still an open field of research: scene interpretation and object recognition are today active and open territories for scientific investigation Augmenting a space with all three forms of intelligence can be seen as endowing the space with a mind of its own, which transforms it from a simple container of people and things to an active participant and cooperating agent of our lives Perceptual intelligence... urban-like distribution of information facilitates navigation of large information databases, like the Internet, by providing the user with a cognitive spatial map of data distribution To navigate this 3D environment, people sit or stand in front of a large screen (Figs 4 and 5) and use hand gestures to explore or load new data Side-pointing gestures allow users to navigate along an information path Holding... time On the basis of my work and research I will argue that intelligent spaces need to be supported by three forms of intelligence: perceptual intelligence, which captures people’s presence and movement in the space in a natural and non-encumbering way; interpretive intelligence, which “understands” people’s actions and is capable of making informed guesses about their behavior; and narrative intelligence,... Therapy’ body of research This was extemporized in a 2002 article (Brooks et al 2002) following findings that explored interactive visuals in the form of animations, virtual reality, and robotic lighting devices with colored and patterned gobos This aesthetic is a representation of a system where affordances were perceived and the targeted action achieved Resultant sense of self-agency and inner empowerment . Furht (ed.), Handbook of Multimedia for Digital Entertainment and Arts, DOI 10.1007/978-0-387-89024-1 32, c Springer Science+Business Media, LLC 2009 713 714 F. Sparacino how to sequence and articulate. ideal modeling tool for all three forms of space intelligence and to provide a unifying mathematical description for robust sensing, user and context modeling, and articulated information delivery (storytelling). Related. inter/multi-sensory stimulation for both audience and per- formers. This built upon my earlier work in Auckland, New Zealand, and Aarhus, Denmark where the different situations, one with orchestra and one with