text:
VIEWS: 10 PAGES: 141 POSTED ON: 10/15/2011 Public Domain
A Brush with the Real World: the future of inertial motion capture in live performance John Haag Drama (Performance Studies) Creative Industries Faculty Queensland University of Technology September 2009 This thesis has been submitted in fulfilment of the requirements for the award of Masters of Arts (Research) in the Creative Industries Faculty at Queensland University of Technology, 2009. Acknowledgements This study has been made possible through the collaboration of cutting edge equipment supplier VR Solutions who have been very generous in allowing me significant time using their motion capture equipment. Michael Bosworth, as director, was open to experimentation and was most helpful in facilitating sessions at their studio. Jon Newell, lead developer at visualisation company VR Space, provided many hours of his own time in ironing out problems with the suit, guiding me through the complexities of the software and working with me on numerous motion capture sessions particularly in the earlier stages. During this time I also received timely assistance from QANTM graduates, Kim Allen and Vana Prayitno, who helped with the initial modeling. Rosetta Cook provided insight into the world of dance and, with Joe Lau and Bridget Fiske, contributed many hours in the studio patiently improvising dance moves as I tested the real-time environments. A Brush with the Real World was performed as an improvisational installation at Judith Wright Centre of Contemporary Art in Brisbane. It was a difficult undertaking and we were provided with generous assistance from the administrative and technical staff, along with the small dedicated team comprising Rosetta Cook, Miriam Hanmer and Adrian Green. Academic research did not come naturally but my supervisor, Paul Makeham, has been tremendously helpful in provided clarity in designing the written component of my research as well as assistance in structuring the research as a whole. ii Statement of Original Authorship The work contained in this thesis has not been previously submitted to meet requirements for an award at this or any other higher education institution. To the best of my knowledge and belief, the thesis contains no material previously published or written by another person except where due reference is made. Signature Date / / iii Table of Contents Acknowledgements ii Statement of Original Authorship iii Abstract viii Keywords ix Note on the Accompanying Media x 1 Preface 1 2 Contextual Review 7 2.1 Introduction 7 2.2 Motion Capture 9 2.2.1 Issues of terminology 9 2.2.2 From performance to screen 12 2.2.3 The actor 14 2.3 Performativity 17 2.3.1 Introduction 17 2.3.2 Spectacle 28 2.3.3 Reality 22 2.3.4 Presence 26 2.3.5 Liveness 30 2.3.6 Spontaneity 33 iv 2.4 Motion capture in the arts 35 2.4.1 Motion capture in visual media 35 2.4.2 Digital Performance 37 2.5 Conclusion 42 3 Methodology 45 3.1 Research question 45 3.2 The nature of research 46 3.3 Postmodernism 47 3.4 Performative research 52 3.5 Practice-led research 54 3.6 Investigative framework 56 3.7 Methods 59 3.7.1 Case study 59 3.7.2 Creative practice 64 3.7.3 Industry input 67 3.7.4 Applied methods 70 3.8 The working environment 72 4 Description of practice 74 4.1 Overview 74 4.2 Case 1: A Brush with the Real World 76 4.3 Case 2: Chasing Shadows 80 4.4 Case 3: Private Eyes 82 4.5 Case 4: Motionics 84 4.6 Log of practice 87 v 5 Evaluation 91 5.1 Inertial motion capture 91 5.2 The Case study 95 5.2.1 Case 1: A Brush with the Real World 96 5.2.2 Case 2: Private Eyes 100 5.2.3 Case 3:Motionics 101 5.2.4 Case 4: Chasing Shadows 104 5.3 Observations of the performer 105 5.4 General findings 107 5.5 Implications for live performance 109 5.6 Summary 113 5.7 The future of motion capture 113 6 Conclusion 120 7 References 124 Diagrams Figure 1: VRS Studio (2005) 1 Figure 2: The dancer's movements are captured and applied 13 Figure 3: Using rear projected animation in public windows 68 Figure 4: Large-scale shadow puppetry using animation 68 Figure 5: Using animated effects triggered by movement 68 Figure 6: Projected background moves with actor's movements 69 Figure 7: Projection within sets onstage 69 Figure 8: Interactive animated shadow 69 Figure 9: Shadow projection on the tent wall in circus 70 Figure 10: Improvisational screen character 70 Figure 11: A Brush with the Real World 76 Figure 12: The layout for A Brush with the Real World 77 Figure 13: Chasing Shadows 80 Figure 14: Private Eyes 82 vi Figure 15: Motionics 1 84 Figure 16: Motionics 2 84 Figure 17: Motionics 3 84 Figure 18: Motionics 4 85 Figure 19: Trialing the paintbrush 96 Figure 20: Early relations constraint for activating the colour orange 99 Figure 21: Reactive background 101 Figure 22: Eyes follow the performer 102 Figure 23: Moving through a forest of eyes 102 Figure 24: Triggering effects 102 Figure 25: Using video 102 Figure 26: The paint colour changes universally 103 Charts Chart 1: Attributes of inertial suit 93 Chart 2: Limitations and problems 93 vii Abstract The history of art is a series of successful transgressions. (Sontag cited in Kramer, 1999: 9) 3D Motion capture is a medium that plots motion, typically human motion, converting it into a form that can be represented digitally. It is a fast evolving field and recent inertial technology may provide new artistic possibilities for its use in live performance. Although not often used in this context, motion capture has a combination of attributes that can provide unique forms of collaboration with performance arts. The inertial motion capture suit used for this study has orientation sensors placed at strategic points on the body to map body motion. Its portability, real-time performance, ease of use, and its immunity from line-of-sight problems inherent in optical systems suggest it would work well as a live performance technology. Many animation techniques can be used in real-time. This research examines a broad cross-section of these techniques using four practice-led cases to assess the suitability of inertial motion capture to live performance. Although each case explores different visual possibilities, all make use of the performativity of the medium, using either an improvisational format or interactivity among stage, audience and screen that would be difficult to emulate any other way. A real-time environment is not capable of reproducing the depth and sophistication of animation people have come to expect through media. These environments take many hours to render. In time the combination of what can be produced in real-time and the tools available in a 3D environment will no doubt create their own tree of aesthetic directions in live performance. The case study looks at the potential of interactivity that this technology offers. The written thesis comprises 100% of this Masters by Research (35,000 words). viii Keywords Motion capture, performance animation, performance capture, real-time, inertial, live, liveness, spectacle, virtual reality, paralanguage, improvisation ix A note on the accompanying media The documentation presented here includes a DVD and a CD. This is intended to support and further explicate the written data, and may be viewed at any time prior to, during, or after reading the written thesis, at the reader's discretion. It is recommended, however, that the written thesis be completed prior to viewing the DVD. A Brush with the Real World installation (DVD): The DVD contains edited video of the technical proof of concept performance installation, A Brush with the Real World. The installation was held each evening from 16-18th July 2009 at the Judith Wright Centre of Contemporary Art, Fortitude Valley, Queensland. The performances were improvisational. All video was taken on the 18th July. (There were some software problems on the night and several interactive features were disabled which prevented full use of the painting environment and participation by the dancer.) Research studio trials (CD): Over the period of the research many animation features were trialed in real-time to gauge their suitability as tools for creating visuals in performance. The CD contains short clips showing some of these trials. It should be noted that most of these clips are not intended for performance but for testing real-time performance of software features. x 1 Preface The theoretical activity of humans, like the practical, is not the independent knowledge of a fixed object, but a product of ever-changing reality (Horkheimer, 1982: 28) When I began this research my intention was to explore the use of projected animations for public events and performances. I had accumulated a series of projection concepts and was keen to develop the best of them. Fortunately, during my early enquiries Jon Newell, software engineer at VRS in South Brisbane, told me about Figure 1: VRS Studio (2005) their newly acquired motion capture suit. This was an opportunity to explore how real-time animation could work in live performance situations, and it fitted in well with several of the concepts I planned to explore. The motion capture suit was an inertial based system, which appeared to have features compatible with working in a live environment. As a performer and deviser, I had a long interest in mediatized and convergent artforms, producing stage acts dating back to the early eighties that mixed visual media, such as video and animation, with sound and lighting effects to enhance live performance. It was the "Bjelke-Petersen" era. Politics in Queensland was strongly polarised and there was a strong political undercurrent in much of the arts scene at the time. The motivation for using a hybrid visual approach was less to do with art than working out ways to maximise the appeal of a message. 1 My early career had diverse artistic influences, originally studying fine arts and later drama focusing on group-devised and effects-based theatre. I also learned fine art screen-printing for a while from Charles Bannon and became involved in politically based street theatre with Pitjatjantjarra Land Rights Movement under Ruby Hammond. I taught for several years, followed by itinerant work in northern South Australia and Western Australia, before heading up the Birdsville track on my motorbike and meandering towards Brisbane. I worked as illustrator for Queensland National Parks for a while but grew increasingly interested in theatre. Much of the early involvement in performance was with political and community theatre, working on campuses, at rallies, in cabaret venues, ZZZ venues, the Community Arts Centre and other fringe venues, as well as facilitating unemployed and community group projects. The work at cabaret venues took an increasingly mediatised focus, melding into a form that came to be known at the time as mixed media, in which visual techniques, sound and lighting effects were incorporated into live performance to create a performance montage. So, as much a reflection of social change, the arts, innovation, even science appeared more as a vast wilderness open to exploration than a cluster of genres separated by fences. Mixing genres and exploring new venues along with experimentation in theme and form at the time was unencumbered by the weight of tradition. The visual theatre, mostly devised in collaboration with musician Janelle Skinner, was an attempt at placing impact ahead of tradition in performance. It was influenced by video clips, and freely mixed movement, synthesizers, animation, black light, video and a bizarre array of themes, from performing cockroaches to a baby Jesus ventriloquist doll; all aimed at visualizing a message rather than tugging the bellropes of an artform. This visual form seemed to have more impact than some of the didacticism in political theatre and could be used in rock venues as comfortably as theatre cabarets. During this time I experimented with the use of animation in live performance, collaborating with Terry Kavanagh on several simple stop-frame animations using cutouts on layers of glass. This, along with Terry's individual work, was 2 used in several music-based performances. This sowed the seeds of what now has become an important part of my work. After a ten month community theatre tour through the Queensland Arts Council, I worked in an array of positions including artist in residence for the Multicultural Festival for 1984 and 1985 as well as designer, director, and tour manager with productions through the late eighties. Commedia dell'Arte and improvisation were strong influences around this time and touring became a big part of the lifestyle. General audience tours, schools tours, interstate tours, overseas tours, residencies, festivals and workshops filled the nineties. My illustration work also continued with regular contracts for illustration and graphic design work including a regular graphic work for the Queensland Arts Council. More recently, I took on two stints as an events artist through the Premier's Department, three productions for Out of the Box Children's Festival and design for two children's productions. From the mid nineties more of the design and illustration work gravitated to computer and so in 2002 I studied multimedia and later received a scholarship to study animation at QANTM. In the last couple of years, my work has diversified to incorporate 2D and 3D animation. This has included several projects that used motion capture. The performance work has also continued with regular stage and roving at festivals and conventions. This research is a culmination of a long and ongoing learning process through practice as well as an opportunity to explore an emerging field that makes it possible to use live performance, animation and illustration in combination. The research uses 3D motion capture as a tool in live performance. For a variety of reasons, both technical and economic, the use of real-time 3D motion capture in live performance and at live events has only recently been feasible and, while not yet cost effective, it will soon reach a stage where its use at public events is not limited by cost or technical difficulty. The four cases underpinning this research collectively shed light on ways of creating real-time narratives or "instant movies" as a projected visual form: 3 movies that are presented as they are performed, in which actors and screen depictions can both be part of the performative dynamic. The task of creating complete movie performance would be a large collaborative project and beyond the scope of this research. By segmenting the practice into the four cases, each with its own techniques and outcomes, many of the technical boundaries and creative possibilities can be tested in a more resource efficient way that, when viewed collectively, still provide insight into the logistics of creating big picture projects. Animation based on motion capture has been used with increasing frequency in the past few years in live dance, creating a visual landscape or synaesthetic enhancement to the performance. One case, Motionics, investigates the range of dance visuals that can be generated in real-time with an acceptable level of quality. Chasing Shadows explores hybrid theatrical performance, using a live actor onstage together with his projected shadow. The shadow slowly takes on a life of its own and, eventually, a mind of its own. Private Eyes focuses on the cinematography of a real-time movie, created as it is acted: a misty street at night, a lonely figure and his story. It is screen theatre, with the reality of the "live" blended with a world of cinematic imagination. The primary case, A Brush with the Real World, is improvisational animation that uses painting as a way of communicating to an audience. Projected on a large screen is a cluttered 3D art studio inhabited by a virtual artist. Armed with his paintbrushes the animated artist, Rupert, can paint in the air around him. His movements and his painting are controlled by a live performer who is able to set up visual, gestural and artistic communication between Rupert and audience. This allows the audience to influence Rupert's world onscreen, especially his artwork. The research I am undertaking centres on mediatized performance. It uses a live performer to actuate movement in virtual 3D space in real-time. The investigations are underpinned by technology with a very recent history. The Gypsy Gyro Inertial suit used in the research was the first one commercially produced, and arrived in Brisbane about the time I commenced the research. Learning how to use the suit effectively required an understanding of how it 4 worked: studying software, exploring the physical parameters, learning how the hardware interfaces, and understanding the organizational procedures for motion capture. This was a necessary prelude to developing performances. The research is practice-led, using a case study approach comprising four cases to reach general conclusions on the suitability of inertial motion capture to live performance. Inertial motion capture has properties that suggest it could indeed be used broadly with live performance. It transmits wirelessly, operates in real-time, fits into a small suitcase, can run on batteries and record to a laptop. Practice-led research is still evolving as an investigative approach, recognising and encompassing the valuable contribution creative practitioners make to the pool of new knowledge. The juxtaposing of disparate objects and ideas has, after all, often been viewed as an intrinsic aspect of creativity. The interplay of ideas from disparate areas of knowledge in creative arts research creates conditions for the emergence of new analogies, metaphors and models for understanding objects of enquiry. (Barrett, 2006: 8) The outcome of this research is not primarily to create a performance but, through practice, to assess and evaluate techniques that can be used in performance. There is little information on the use of motion capture in live performance and, for directors and choreographers, there is no clear picture of the tools available and how they can be used. There is little commonality of language between performative technologies and performance per se. There is also a difference in the reflexive approaches that lead to creative outcomes. This can result in unrealistic expectations, reinventing wheels, or putting hybrid forms into the too hard basket. Along with some well documented successes, the nineties are littered with performance/technology experiments that fell short of their aims. There is a need to know more about the mechanisms of real-time motion capture that can be used in real world performance and included in real world events. It is important that the 5 outcomes can be practically applied to the conditions of live performance. Motion capture is usually a highly collaborative process with technicians, motion capture director and performance animator using costly equipment in dedicated space in combination with teams of animators and a studio production infrastructure. Live performance is not a mass saturation medium and so the economies of scale necessitate a more modest operational structure. The cases use a simplified workflow and limited collaboration to approximate the lower budgets of live performance. Often in the context of practice-led research into performance there has become a de facto expectation of a completed performance from which research conclusions are drawn. However, in emergent research generally, the emphasis is not so much on the artifact created through practice, the usual activity of artisans, but on matters of process or perception. In the Methodology chapter I will show that this approach is strongly reinforced by both practice-led theorists and supported by the Australian Department of Education, Employment and Workplace Relations' definition of research. As anyone who works in the field is aware, 3D animation is time consuming and complex. Creating a complete live performance using real-time motion capture is more than twice the sum of its parts. It is the creation of an animation environment, the creation of a live performance and the solving of all problems that melding these distinct and different forms entails. With the many techniques and methods in animation that motion capture opens up to live performance, the series of strategically targeted cases, while collectively providing a more sweeping evaluation than a single performance could, are just a small peep into a field with vast potential. 6 2 Contextual Review 2.1 Introduction We fear things in proportion to our ignorance of them. (Titus Livy cited in Bock, 1970: 379) This chapter surveys the historical background of motion capture, and performance that incorporates motion capture. It explores the theoretical and semantic foundations of the research. While animation has, at its roots, a theatrical basis and uses many terms derived from theatre, it has developed separately and some of the language mutual to both has meanings specific to each. Animation has its own evolved aesthetic, governed more by its ability to represent fantasy, to shape moments and create worlds visually without the need for descriptions embedded in storytelling. While some see motion capture as offering a way for live performance and animation to interact, opening up new artistic possibilities, others see it as undermining the art of animation, taking the "art" of animation out of the hands of the animator and automating it. As motion capture becomes more popular, many animators have begun to think of it as the anti-Christ. They talk about its instabilities, lack of flare and expense, but in reality, they see it as a threat to their livelihood. (Lindsay cited in Liverman, 2000: 45) However, recent history of its use has indicated that in most areas motion capture is not a threat to conventional forms of animation. This is partly because motion capture has only limited application as an animation tool. People have developed a perception of animation based on the visual and behavioural conventions and aesthetics that have evolved with animation. There is an evolved system of visual symbols and cues that make up a visual language we have come to recognise and accept. When a character is 7 animated using captured motion, audiences ironically often perceive these movements as incompatible with animation. Moreover, contrary to the notion that actors are taking the place of animators, the overall number of animators is growing quickly. In the U.S. the number of animators and multimedia artists rose from 23,000 in 2005 to 26,000 in 2006 despite a growth in the number of motion capture studios. During this same period the number of actors fell from 59,590 to 51,880. (U.S. Bureau of Labor Statistics Office, 2007: n.p.) The performing arts have a long history of experimenting with and integrating new ways of working based on opportunities that arise with the introduction of a new technology. For instance, the machinery used in some Baroque Theatre, based on ship technology, used elaborate systems of winches and pulleys to facilitate scene changes and enhance the visual spectacle of theatre (Kirshenblatt-Gimblett, 2004: 49). Likewise electric lighting has reshaped the way in which theatre is presented, embellishing the mood onstage, changing the relationship between the actor and the audience, even bringing about changes in the physical layout of the theatre itself. Technology has opened new paths and led theatre practice into exciting new places as well as dead ends. It has transmuted into forms that are no longer recognisable as theatre. As Steve Dixon (2007) points out, the actor can interact with his/her double in its many iterations, or can capture one moment from a number of perspectives. Time can pass at different rates through the same moment; narrative can accommodate concurrent multiple perspectives. As the relationship between performing arts and a new media technology matures, the technology can disappear behind the effect it creates, so that audiences no longer see just the "gee whiz", but are led once again on a journey or engrossed in an experience. This happens when the building blocks of the constructed reality are familiar. The dimensions of space and time which constitute our conventional understanding of reality have already been destabilised, expanded and complicated so that this virtual world seems less wholly 'other' and more 8 like just one more manifestation of the real-imaginary, equally flexible and contingent, no more or less constructed than our interpretations of the previous live action. (Smart, 2005: 49) 2.2 Motion capture 2.2.1 Issues of terminology Motion capture still lives at the bleeding edge and most who have experimented with it have experienced more bleeding than edge (Lindsay, 2000: 45) Motion capture is distinct from other visual media in that it translates physical movement to digital motion data. Motion capture or motion tracking can refer to a broad range of data input techniques; the computer mouse is a simple motion tracking device. In the context of this research motion capture refers to the plotting of human motion in a form that can be represented as performance in 3D virtual space. The three dimensional movement data collected can be assigned to drive equivalent movement by a virtual character. This data can also be used to control visual effects, sound, virtual scenery, in fact anything that can be manipulated by computer data, including things in the physical environment, from lighting rigs to robots. Motion Capture is the process of recording a live- motion event and translating it into useable mathematical terms by tracking a number of key points in space over time and combining them to obtain a 3D representation of performance. (Menache, 1999: 1) Unlike linear or interactive games media, in motion capture the effect created can be driven by movement as it occurs. As the motion data is generated by a live actor in real-time the effect on screen can be sympathetic to the 9 organic timing of live performance. The metronomic inflexibility of linear media has been a drawback when projection is combined with live performance, which may help to explain the limited use of synchronised projection in theatre. This inflexibility has been overcome to some extent with the triggering capabilities of interactive games technology. Visual media is more commonly used in dance, an artform that usually accompanies rather than leads the timing. This is also because dance is as much a visual artform as a performance genre, while theatre generally relies less on spectacle and more on the flow of human interaction. Inertial motion capture uses strategically placed sensors at points around the body to measure changes in orientation. These orientation values are sent wirelessly to a receiver that is attached to a computer. This data is used in combination with an actor file, a file that contains the performer's body measurements, to portray performance in digital form. The sensors only calculate orientation values and not positional data, and so they provide no clue as to their location in space. However, using the orientation of other sensors and appropriate body measurements each sensor can approximate where it is in relation to other sensors. The animation and games industries are driving the rapid development of tools that can be used to apply motion data to sophisticated animation environments, and many of these tools are making their way to real-time animation environments. A fast 3D environment such as MotionBuilder can apply a broad range of animation and scripting options to motion data, providing a growing number of creative possibilities for its use in live performance. However, the future probabilities are exponentially greater as many of the extensive features available in mainstream 3D software make their way into the real-time environment. In creative terms motion capture is a pipeline between live performance and animation. The motion is derived from the movements of a live performer and converted into language that can be used by an animator. The animator is able to create an environment that can be affected by the performer's movements. This environment can then be a responsive projection in a live performance scenario. With the exponential growth of 3D animation, it has a 10 growing presence in cartoons and other animated sequences. Motion capture had a marginal presence in movies until the very successful Lord of the Rings trilogy when it moved to mainstream status, as much the result of technical advances as the celebrated performance animation skills of Andy Serkis who provided most the movement and facial expressions for the character, Gollum. This type of animation, where some or all of the motion is derived from the movement of a live performer, is called "performance animation". So "… motion capture is the collection of data that represents motion, whereas performance animation is the final product of a character driven by a performer" (Menache, 1999: 1). Real-time motion capture occurs when there is no discernible delay between the recording of motion and its conversion to motion data. In real-time performance animation a 3D character or effect onscreen occurs simultaneously with the movements of the performance animator. The distinction between motion capture and performance animation raises questions as to how motion capture relates to art. Is motion capture a tool that can be employed in the creation of art or is it a medium for reproduction of art? Is the art in the motions of the actor or the effect of these motions on the screen? The actor embodies the ritualistic nature of the "live", the authenticity of the original. Through capture these movements become reproducible and this reproducibility "emancipates the work of art from its parasitical dependence on ritual" (Benjamin, 216) However, works based on "real-time" motion capture disembody the art, but arguably preserve the ritual, for the work is an altered vision of what's actually happening at the time and the possibilities for, at least, non tactile interplay with an audience are preserved. The work itself is not just a reproduction, as in a movie; it is a pipeline between the performer and a visual environment that creates unique, highly responsive and ever-changing artwork. The art is both in the actor's movements, as live performance, and in the visual environment, as computer art. Stripped of this uniqueness, it becomes a technology demonstration. The terms "real-time" and "live" both have multiple definitions that overlap and, when used with motion capture can be misleading. A physical 11 performance that happens at the same time as its digital depiction onscreen is considered "real-time". However, the term is also used to describe instantaneous game-play action as a player navigates a virtual world, a situation in which only the triggering of the animation is "real-time". With movies the animation derived from real-time motion capture is usually further modified before it is presented to the public; the physical performance and the public showing are at different times. So "real-time" has been used refer to the entire process from live performance to screen performance as well as only part of this process. The word "live" also has levels of meaning that are ambivalent. When we refer to dance as "live" it indicates that the dance is physically being presented at the same time and place it is viewed. However, when referring to a mediated event such as live broadcasting, the event is happening at the same time but at a different location to where it is viewed. The term "live" was originally used in this context in the 1934 BBC Yearbook to differentiate broadcasts that were created as they were delivered from recorded broadcasts (Auslander, 2002: 18). The problem in the case of this study is that it looks performatively at the intersection of performance and broadcast media. So to avoid confusion, the word "live" is used to describe performance that involves the physical presence of performers, and the expression "real-time" refers to the process of representing a performance in an animated form onscreen as it is physically performed. 2.2.2 From performance to screen ...the issue is not whether or not to uphold a particular definition of performance over and against media and technology, but to work with relationships between them. (Kirshenblatt-Gimblett, 2004: 49) 12 Motion capture is suitable for certain styles of animation and certain circumstances in which animation is used. It is an expensive technology that is maturing at a rapid rate, driven by two of the world's most lucrative industries, movies and interactive games. It may one day change the face of mediatized performance, because it is a medium that can respond instantly to a person's input, using language and gesture that is familiar, and a sense of timing that flows organically with the whims of human interaction. Human motion is captured with the subtleties of expression and nuances of secondary motion that are unrepeating. This is Figure 2: The dancer's movements are structurally different from the triggered captured digitally and then applied to a 3D character behaviour used in interactive games environments (a limited set of pre-animated responses and simple behaviours as opposed to an unlimited set of appropriate spontaneous responses). Appropriateness of response is intrinsic to the way we communicate and triggered responses, no matter how well devised, are only a limited simulation of communication. In contrast, responses in real-time motion capture are potentially as diverse as human interaction, albeit on a screen with a projected 2D simulation of a living being. A screen performance using motion capture can convey paralanguage, the behavioural and gestural language of human movement, as it is happens. With motion capture it is possible to use an accurate screen depiction of one person and combine it with the motions of another. Human movement can be assigned to a bird, or bovine movement can be assigned to a human. In fact, human motion can be isolated and used to affect a projected scene or 13 character, generate sounds, change scenery, direct robots and conduct spectacles. Furthermore, the motion of a person can potentially control any conceivable device or digital image that can be digitally networked anywhere on the planet without any discernible delay. 2.2.3 The actor When an individual plays a part he implicitly requests his observers to take seriously the impression that is fostered before them. They are asked to believe that the character they see actually possesses the attributes he appears to possess, that the tasks he performs will have the consequences that are implicitly claimed for it, and that, in general matters are what they appear to be (Goffman, 2004: 61) As mentioned earlier, both actors and animators have felt some trepidation about the use of motion capture in film. Actors see it as supplanting their identity, and animators as automating their skills. Bill Nighy played the villain, Davy Jones, in the Pirates of the Caribbean trilogy (2003, 2006, 2007). Motion capture was used to give his face a realistic octopus-like appearance, with tentacles curling and twisting in concert with his expressions and moods. Initially, he was skeptical but came to appreciate that the facial animation, rather than obscure the nuances of facial movement, actually enhanced them. They can enlarge an actor's experience. There are so many stories we're going to want to tell that require those kinds of skills and those kinds of images, and I see no reason why actors and CGI can't work hand in hand. There's no reason one should threaten the other. (Bill Nighy, 2006: 1) Motion capture does not replace traditional animation or acting. Both these 14 skills have traits that motion capture cannot or, at least, does not need to emulate. It can offer an extension of what exists, new opportunities for artisans and new possibilities for moviemakers. It helps to expand and diversify the realm of visual representation. …all performance involves a consciousness of doubleness, through which the actual execution of an action is placed in mental comparison with a potential, an ideal, or a remembered original model of that action (Bauman cited by Carlson, 2007: 73) Steve Dixon, author of probably the most comprehensive compendium of digital performance to date, expands on Artaud's notion of "the double" to explore the interplay between the live actor and his/her projected portrayal, drawing on a broad range of productions including 4D Art's Anima and the one man show, Afasia by Marcel-Lí Antunez Roca. The actor's digital double can be a distinct and separate entity, as in a traditional stage role, or can play another side to the person onstage, a role that is more common in mediatised performance. This enables us to view multiple facets of the same person, allows a person to interact with themselves, or creates an interplay in which the screen double can extrapolate, intensify, magnify, diminish or make absurd the actions of the physical self onstage. With performance animation the performer can disappear leaving the double as the storyteller, the puppet on the screen. A production in which the actor can be represented on screen in real-time straddles the divide between what is mediatised and what is media. The actor can totally control his/her screen image while not being visible to the audience. When live actors join real-time avatars onstage, as in Saltz's interpretation of The Tempest or Pinhanez's one-man show, A Computer Theatre Play at the MIT Media Laboratory, a dynamic is established that is unique as a mediatised experience, freeing the actor to control the pace of the stage action. Dixon refers to this type of digital representation, as an "avatar stage double". A stage double can be invisible while, in a liminal way, the subtleties of body language can still be seen. Jane 15 Harris dispenses with a visible body altogether in Portrayal, using only a flowing dress to convey the captured dance movements of Ruth Gibson, "emphasising the beguiling qualities of fluid material and form." (Harris & Walsh, 2004: 139) The dimension of time is a playground for mediatized performance, allowing asynchronous and multiple views of events, juxtaposing the same person in the same space at different times simultaneously. In the play, Memorandum by the Japanese performance group, Dumb Type, one event is shown at different rates on multiple screens at the same time in "a montage of time perspective" (Peterson, 2004, 32). Only a few bits of furniture litter the stage, behind which are four projections each showing the scene onstage but at varying speeds from very slow to very fast. All have been synchronised to end at the same point in time on screen. This "telematic" performance shows how visual media can introduce time, not just as a linear progression of moments, but as another perspective. This same juxtaposition of passing time can also be used in animated representations of a scene onstage, opening up new ways of portraying a double. In 3D space the same scene or action can be shown from any perspective at the one time; so dimensionality of the space becomes a dynamic element. Having a pipeline between the performer and a 3D animation environment opens up the huge legacy of visual and interactive software to live performance, making possible all sorts of anthropomorphic and highly interpretive visual derivatives of live action. Not only can multiple perspectives be used but distinctly different visions of the same event, each with its own place, context and aesthetic. These visions can happen concurrently with real life performance and, with the dimensionality of 3D space, each can focus on a different aspect of the same set of actions. 2.3 Performativity One is not simply a body, but, in some very key sense, 16 one does one's body and, indeed, one does one's body differently from one's contemporaries and from one's embodied predecessors and successors as well (Butler cited by Felluga, 2003: n.p.) 2.3.1 Introduction Performativity as a concept has expanded significantly from J. L. Austin's originally intended meaning, when he coined the term to describe self- actuating statements or "performative utterances". Judith Butler (in Jagger, 2008) used the term in a societal context to highlight how speech acts shape who we are. We absorb the language of the social constructs that surround us and this becomes a script for the way we act out our lives. For instance, Judith Butler sees gender as an act rather than a category. Performativity these days can refer to a broad range of social self-actualising processes and has been associated with body language (Schechner, 1993: 45) and with emergent research through creative practice (Haseman, 2006). Motion capture can be described as a performative medium in that it facilitates a creative transition that encapsulates the spontaneity and immediacy of human movement. This section looks not only at the performativity of live performance but the performativity of the medium and how this combination can create performance that preserves a sense of immediacy while allowing for new visual possibilities. It also looks at the social context of visual media from a postmodern perspective, and theoretical factors that come into play when working with mediatized performance. Finally, it outlines the history of performance that uses motion capture including an overview of its use in the animation industry. Motion capture is a fast evolving technology at the beginning of its history. This contextual review takes a broad overview of this type of performance, rather than dealing at length with specifics, because the heritage of this type of performance is far outweighed by the scope for what can happen in the future. The words "convergence" and "diversification" are commonly used when referring to innovation in visual technology, emphasising dynamic changes in 17 applications and structures in a rapidly evolving field, rich with niches for exploration. The cases of practice used in this research are not only a convergence of two fields, motion capture and live performance, but also part of the diversification of each field, and highlight some antithetical and juxtaposing concepts. Immersion/distraction, theatre/spectacle, live/mediatized, simulation/original, presence/disappearance, reality/virtual reality, visual/aural are some binaries at play when visual technology is used in the performing arts. They provide an insight into the social context, the effect on the audience, the way in which live performance and projected images interact and the nature of the virtual world. 2.3.2 Spectacle The visual pictorial aesthetic triumphed over the aural, and an important reason for this victory must be the greater degree of innovation in the technology of the visual, above all in the production and control of artificial light. (Christopher Kent, 1996: n.p.) In broad social terms media is a pervasive influence that exposes us to enormous volumes of information while at the same time subjecting us to unprecedented levels of influence, shaping our sense of reality, removing our lives further from the physical and, some may argue, exalting the trivial above the consequential, hiding the "actual" under layers of subjective interpretation. In mass visual media, it seems, we prefer to be distracted rather than informed. Debord's "spectacle", Baudrillard's "simulation" and Lyotard's "metanarrative" allude to the mechanisms that underlie this pervasion. In the information age, knowledge and meaning can be inhumed by the sheer enormity of the information available and this information typically reflects the dominant values of the technologically advanced societies. It is integral to the mechanisms of post-colonialism and consumerism. While an understanding of the forces at play in visual media provides the context in which one applies the tools of art practice, this 18 research is investigating the tools as much as applying them. Along with the growth of visual media, there has been a parallel shift towards visual performance. The evolution of modern circus with companies like Circus Oz, Footsbarn, Circus Archaos and Cirque du Soleil, along with the phenomenon of performance spectacle with groups such as La Fura dels Baus are an indication of a trend in live performance towards increasingly awesome spectacle. This is reinforced by the greater use of visual embellishment in stage productions such as modern renditions of Phantom of the Opera or Les Miserables (Kellner, 2004: 8). It emphasises people's greater focus on the ocular that, through film, television and more recently video games, has become so entrenched in our lives. While it can be argued that spectacle has always been compelling, there has never been a previous time in history where spectacle has had the means, the methods or the mass audiences it has today. Developing countries and the globalized world are emerging into a culture of media spectacle that constitutes a novel configuration of economy, society, politics, and everyday life. It involves new cultural forms, social relations, and modes of experience. It is producing an ever-expanding spectacle culture with its proliferating multimedia and interactive spectacles that are intensifying and expanding as wireless technologies develop. Critical social theory thus faces compelling challenges in theoretically mapping and analyzing these emergent forms of culture as well as democratization and social justice. (Kellner, 2004: 17) Spectacle, as described by Guy Debord, is both a pervasive visual inundation and a means of social control, distracting thought and subjugating reality with consuming images that are a reflection of power, where society is relegated to the role of observer, displacing the real with a prevailing appearance of reality. The receptiveness of an observer and acceptance of the unreal as a 19 part of reality are an indication both of the power of spectacle and the message it imparts. In this world there are no longer points of reference and so "the spectacle is the real" (Debord, 1968: 9). In a similar vein, Baudrillard postulates the mechanisms that underlie this media inundation. Through the media, larger-than-life models of events and people are created which are representations of "real" events and people but the events and people do not exist and cannot exist in the form they are shown. These "simulations" exist only in a world which Baudrillard calls "hyperreality". In this world signs become more important than the commodities they signify (Baudrillard, 1988: 166). Live performance is both a victim and an accommodator of spectacle and is reflexive in its reaction to media's simulation. Not only are theatre audiences seeing live performances that resemble mediatized ones as closely as possible, they are also apparently modeling their responses to the live event on those expected of them by television (Auslander, 1999: 29) The notions of spectacle and simulation highlight how media influences our perceptions of reality. In convergent performance that incorporates visual technology there is a need to understand these mechanisms, recognizing the distinction between creating a different reality and creating a different perception of reality. The defining feature that separates spectacle, a visual event, from spectacle, the societal organising principle, is marketing. The economy that drives the creation of ever more eyecatching visuals is based on selling a product or idea. This consumerist barrage has reached a point where, according to Rushkoff, people: …want a moment of free time and of genuine free time. And there's no more public space left to do that in. There is no more free space left. We've had a real contraction of public space both mentally and in the media and in the real world, where there's nowhere to 20 go where you're not being marketed to. (Rushkoff, 2008: n.p) Kellner concurs with Debord in recognizing spectacle as "one of the organizing principles of our economy" (Kellner, 2003: n.p.) but, more than just a means of control, he believes it also reflects the nature of the underlying culture. Along with the engineered entertainment "megaspectacle", the Iraq War, came the mega-exposé, Abu Graib, indicating that there is room for opposing interests (maybe only enough to validate the impression of impartiality). Baz Kershaw points out that digital technology is minaturising spectacle to the small screen. This process diminishes the individual while at the same time spreads him/her everywhere (2003: 604). However, he recognizes that spectacles can also be a powerful force for change as they "open up new domains for radical revisions of the way things are" (2003: 599). To install such humanity at the heart of spectacle in order to give it a radical new lease on life means continually recognising and reproducing this mutual vulnerability, recognising in the other something akin to the qualities of spectacle that provoke contempt and curiosity. (Kershaw, 2003: 611) In comparison to live performance, new media is often regarded as subtractive, a ghostly imitation of reality. Nevertheless, the ability to reach and influence mass audiences using media technologies makes it inevitable that, to compete for attention and to provide alternative perspectives, new communication tools are explored and used. New media that are at the basis of spectacle are not the sole preserve of large corporations. An effective alternative voice has evolved in more established forms of mass media, such as film and television, and there is social benefit in harnessing new tools of media for use outside the constraints of purely commercial projects. 21 While recognising the broad social impact of visual mass media, it is important to understand the mechanisms that make visual media so compelling. As visual mass media has matured so have our expectations and our choices. We can be absorbed, influenced, entertained, distracted and deluded on so many levels. For instance, movies can relate on an emotional and empathetic level. A movie can be thought provoking. They can draw us in to another existence, making us feel like we are somewhere else in some other time. Visual media are no longer just an escape from day to day reality they are now an integral part of our reality. 2.3.3 Reality The opinion which is fated to be ultimately agreed to by all who investigate, is what we mean by the truth, and the object represented in this opinion is the real. That is the way I would explain reality (Peirce, 1992: 139), Media has greatly facilitated and democratised the availability of information but it has also expanded the scale on which information can be manipulated. From Adam and Eve to WMDs there have always been mass "truths" that are detached from real evidence, but now they can be disseminated quickly on an unprecedented scale. Reality appears more durable than truth, constantly affirmed by the consistency of the physical world. However, even the physicality of reality may not be so affirmable, as immersive technologies become better at simulating visual, aural and haptic experiences. This is still a way off (Krueger, 2002: n.p.) but there are other ways in which new media are challenging our sense of reality. Research into "presence" is helping to differentiate the role of mental perception, in contrast to that of physical sensation, in shaping our reality. Studies using interaction with avatars confirm the role of social responses in our construction of reality. Under certain conditions people will respond to an avatar in the same way as they would respond to another person. Experiments by Slater and Garau et al. 22 (2005: 104-116) established that people in convincing circumstances would relate and respond to screen characters as if they were real, even if they have a more ontological awareness that the environment they are exposed to is virtual. This indicates that a character onscreen, using the same devices and strategies that actors use, can engage an audience in two-way communication. Improvisational techniques in which the actor allows the story to unfold as it is created, set up a reality of the moment that is easily transposed to real-time screen interaction. Performance operates by constructing realities. Performance can be considered to consist of special ritual situations in which the actions are part of a meta- fictional linguistic universe and in which, therefore, the performers should not be understood to be executing purposeful actions. (Peterson, 2004: 35) When investigating a mixed reality performance, it is relevant ontologically to distinguish physical reality from projected reality. Reality is not necessarily being substituted by reproduction of reality but is extended to include reproduction as part of reality. However, reality itself changes with advances in media technology so that the infiltrating simulation and spectacle that accompany ever more pervasive media add to what we accept as reality (Dixon, 2007: 143). Despite some of the ominous projections in post- structuralist and situationist theory as to the effect on society of pervasive visual media, people in general are still aware of the distinction between physical reality and media reality. Arguably, any blurring is as much based on our formulation of truth as our perception of reality. The media image may have exerted an increasing influence or power on human consciousness, but that consciousness has not undergone total metamorphosis and collapsed into helpless zombiedom…. Media's afterimages need not insidiously corrupt, nor be indelibly inscribed on the 23 subject's mind and body (Dixon, 2007: 143) This statement does not negate the influence of media rather it hints at the compound way we relate to media. Generally, we can easily distinguish between media and reality; we are aware that it is diversionary and we may know the mechanisms and strategies that underlie media. However, recognising the nature and influence of media does not preclude its ability to influence us. To deny this influence is to negate the entire advertising industry. Media both reflects and creates our reality. However, while people may respond to a screen character in a screen world, or even be lured into dialogue, this does not suggest that a sense of "being there" with a person translates into believing the person is actually present. The interaction is not authenticated sensually. However, it does raise questions as to where reality ends and virtual reality begins. Gabriella Giannachi points to the blurring boundaries between the real world and virtual space and its impact on arts practice. This inevitably reflects on our perception of nature, what it is to be human and the dawn of post- humanism, and how this impacts on and is pioneered by arts practice. We, it is argued, have already become melded with and inseparable from the technology we have created (Giannachi, 2004). The virtual has become a subset of our reality. This reinforces Baudrillard's stance that much of what we see in the media is simulation: it never happened in the terms it is presented. The First Gulf War, a war without bodies, a war seemingly conducted as a video game, is a compelling example. Virtual reality is in a paradoxical relationship with the real. On the one hand, it is part of the real; yet, on the other, it has been constructed to be different from the real in order to be perceived as separate from it. (Giannachi, 2004: 123) Dixon sums the matter up by identifying it as a question, not just of real and virtual, but of the march of technology itself. 24 … the real has changed, as it has always done. The real, and our consciousness of what is real, is subject to time, and for several hundred years it has had a conjoined twin: technological progress. (Dixon, 2007: 144) In movies, new animation techniques have made it sometimes difficult to discern what is physically recorded and what is animated. However, any difficulty in recognising the difference between reality and virtual reality is a dilemma for future generations rather than the present. Nevertheless, the possibility that a reality constructed virtually could substitute for physical reality provokes a serious re-examination of what reality is. Through simulation technologies, the day we can enter an environment where we cannot tell the difference gets incrementally closer although it is still a long way off. In the meantime, a better point of reference artistically may be realism rather than reality. The flight from reality is not exactly a new issue. My parents were once worried that I was being seduced-- by the world of books. Certainly, movies, videos, video games, and the internet have already successfully seduced us from the real world. However, there are no instances of anyone being seduced by current immersion technology. It is not good enough yet. (Krueger, 2002: 1) 25 2.3.4 Presence … the impression of reality we get from a film does not depend at all on the strong presence of an actor, rather on the low degree of existence possessed by those ghostly creatures moving on the screen and they are, therefore, unable to resist our constant impulse to invest them with the "reality" of fiction, a reality that comes only from within us, from the projections and identifications which are mixed in with our perception to the film. (Metz cited in Callesen, 2001: 4) This impression has an interesting and related parallel within animation. Highly stylised characters existing in a stylised world of the kind we see regularly in mass media, such as the Simpsons, Futurama or SouthPark, are readily invested with life. As animation comes closer to visually simulating the real world, the mind switches to focus on anomalies that separate it from the real, as pointed out by Masahiro Mori (1970: 33) in his "Uncanny Valley" Theory; the more realistic a screen world the more we tend to scrutinise it for faults. The fault may be a subtle lack of texture, odd reflectivity of surfaces, shortcomings in secondary motion, moments of inappropriate timing, a slight stiffness of mouth movement, little disparities that form fissures in the real- worldness of the representation. When a virtual world comes close to representing reality it is entering a realm that we are experts in: the appearance of reality. Tiny shortcomings stand out like headlights in the night and jolt us out of the fantasy. Other factors affect the way we relate to projected scenarios. Using the example of a mediatised performance where performers temporarily leave the stage and the audience is left with media alone, Dixon in Digital Performance points out that there is a discernible change in the posture of the viewing audience, which, he indicates, is not a relaxation or a diminishing of attention, but a change in "the type of attention." This suggests broadly 26 that live performance is an inclusive experience, a form of dialogue; an audience likes to respond in a way that communicates to the performer(s). Viewing media may be engaging but not usually inclusive; there is no compunction for an audience to communicate with characters on a screen. For instance, it is uncommon for people to clap at the end of a movie but is almost inevitable at the end of a live performance. The extent to which we connect with the screen environment and the characters on screen is the subject of study into "presence". Presence is a word used both in live performance and virtual performance but is usually applied in different ways. Presence in theatre can refer to the aura that is endowed by the spectator onto an actor. It is a quality rather than a quantity. It is a magnetism that imbues a heightened sense of belief. Since Stanislavski, when actors are observed carefully and perform well, they are said to have "presence". The actor achieves presence through performing as if another. (Phelan, 1993: 117) Presence in virtual reality is somewhat aligned but, unlike in live performance, it has no bearing on the feelings and emotions of the spectator. It is more akin to the level of being that an environment or a screen character might endow. In research, as a factor in games and other interactive media, "presence" is treated as a quantifiable entity. Presence is the response to a given level of immersion (and it only really makes sense when there are two competing systems - one typically the real world, and the other the technology delivering a given immersive system). (Slater, 2003: n.p.) In a virtual environment "presence" is a personal response, and can be 27 described in what Maslow (1968: 133) calls "being values". Presence is a sense of "being there", while the term "co-presence" is a sense of "being with someone". The notion of presence in performance, therefore, has more in common with co-presence in that it relies on the presence of both a performer and observer(s). In fact Erica Fischer-Lichte uses it definitionally when she states, "the bodily co-presence of actors and spectators enables and constitutes performance." (Fischer-Lichte, 2008: 32). In both performance and virtual reality "presence" is also used as an umbrella term covering a number of subtypes such as "co-presence", "behavioural presence" and "social presence". In this broader sense Lombard and Ditton define presence as "the perceptual illusion of non-mediation" (1997: n.p.). This more inclusive meaning provides common ground, as both a virtual environment and a live performance are mediated. If both types of presence are viewed not as an attribute of the performer but as a response of the spectator to the performer, the two meanings are no longer so mutually exclusive. For instance, an observer might regard an actor as having presence because (s)he draws them into the moment, makes them feel like they are with the character. Co-presence does not rely on the visible presence of two people; a person talking on the phone can feel a sense of co-presence (Heeter, 2003: 340), indicating that the actual presence of another person is not essential. A sense of "being with someone" therefore relates to communication as much as to appearance. This suggests the spontaneity of communication that is achievable using motion capture may be very effective in establishing a sense of "co-presence" between a person and a virtual character. In such a production the virtual character is performative, conveying the "liveness" of the performance animator. Presence is crucial to the successful implementation of virtual environments within the performing arts. Actors are able to imagine, to role-play, to 28 communicate and create a shared environment, but they require a high level of responsiveness from the system and a transparent interface. Perhaps more than other users, actors can meet the technology halfway, forgiving it for some constraints. (Reeve, 2000: 209) Thus how a virtual environment is integrated into a performance and how a virtual character appears and relates are important considerations when creating a digital performance. As pointed out earlier, a virtual environment does not need to be realistic, it does need to be integral and consistent. For an onscreen character to be successful in relating to an audience, there needs to be some common social ground. Performance animation is capable of providing the instantaneousness of verbal and bodily communication. Nevertheless, quirks and faults in the real-time animation are common so when a scenario is devised for this medium the choreography, direction and visual design need to minimise the likelihood of these anomalies occurring. This is similar to the attention to detail required in live performance when transposing the audience to another place and time. Little things such as the flickering of a light, the incessant hum of a generator or an inappropriate hat can undermine the world that is created, irrespective of how good the production is otherwise. Function and memory are also factors. Orliaguet (2001: n.p.) points out that in representing a real object, a virtual object needs , at least, to incorporate its function. 29 2.3.5 Liveness Performance's only life is in the present. Performance cannot be saved, recorded, documented, or otherwise participate in the circulation of representation: once it does so, it becomes something other than performance. To the degree that performance attempts to enter the economy of production, it betrays and lessens the promise of its own ontology. (Phelan, 1993: 41) It is ironic that we could define theatre based on the irreproducibility of performance when, in times predating film and television, through stringent adherence to script and choreography, it has strived to be repeatable. In- house theatre played to a different audience every night, and so there was no advantage in incorporating variation. Film was much better at repeatability than live performance, and in many respects, forced live performance to look for a new rationale and to explore new approaches and venues. Live performance has adapted, sprouted new forms that otherwise may never have arisen, intertwining with many human endeavours and merging with other genres to be the diverse, pervasive and often transparent fabric that it is today. Peggy Phelan argues that live performance is transgressive, making each performance unique and irreproducible, separating it ontologically from the mediatised (Phelan 1993). This is quite famously rebuked by Philip Auslander who paints this as a nostalgic notion that bears only a residual resemblance to the reality of modern representation. He demonstrates that modern theatre is almost inextricably mediatised, so much so that theatrical productions often draw their content from film and television. At the time Auslander's book, Liveness: Performance in a Mediatized Culture (1999) was published the two stances, that of Phelan and Auslander, were generally portrayed as irreconcilable. In recent years commentators such as Fischer- Lichte and Dixon have argued that both positions can mutually exist. Dixon 30 elaborates on the liveness debate claiming that "the work of art" has been used to affirm both arguments. On the one hand it stands as evidence for the unique aura and presence of live performance, which can only be damaged and robbed by technology, and on the other as proof that technological incursion does not significantly alter reception of performance, since our minds (and performance itself) are already mediatised. (Dixon, 2007: 117) Intriguingly, some aspects of live performance that Phelan uses to define its uniqueness are also seen by those working in digital performance, such as Callesen (2001) and Saltz (2001), as intrinsic to the level of "presence" in a mediatised production. In live performance it is the variability, the spontaneity, contextual appropriateness, the responsiveness, the timeliness of inference and gesture, the signs in secondary motion, the proxemics, the consequent unrepeatable interaction between the actor and the audience that are claimed to provide an intimacy of experience and an authenticity of interaction. In a performance in which real-time motion capture is used the animation can move beyond the linear confines of other visual media and interact with the audience in the moment, incorporating many of the traits that are seen to be intrinsic to "liveness". These traits are all manifestations of paralanguage, the language of the body. Through performance animation many aspects of paralanguage are no longer only the preserve of human-to-human interaction. How effective this is as a means of communication between a person and a virtual person is relatively unexplored, although the way in which a person relates to the streamed video of a friend during a "live" chat session suggests they would feel a strong sense of co-presence. Using performance animation, the body language of the virtual character is not generated in software but is a digital reconstruction of a live actor's movements. This is significant as a means of incorporating human facets once considered exclusive to live theatre into a 31 digital environment. The underlying awareness of physical reality is always there but, as in movies and games, people can be absorbed by the actuality of the interaction. Unique to real-time performance animation, this media interaction may lever the spontaneity and complexity of paralanguage. This research is therefore not just exploring the use of real-time performance animation in live performance but may shed some light on the nature of the paralanguage between a person and virtual character. The potential for motion capture in live performance is that it offers a choice of possibilities in a way that is transgressive. There is much to be learned from the burgeoning research on human/avatar interaction in the games industry that can help create a broader understanding of how onscreen characters can be used sympathetically in live performance situations. In referring to network interaction, Garau and others highlight the importance of appropriate responses in creating meaningful exchanges between real and virtual worlds. The challenge with human-human communication is to drive avatar behaviors that enrich, rather than hinder, communication between remote participants. With human-agent communication, the challenge lies in simulating intelligent social responses by coupling the agent's dialogue with appropriate conversational behaviors. (Garau et al., 2000: 105) 32 2.3.6 Spontaneity There is an important difference between the way someone would relate to a person in a recorded video sequence or movie and how they relate to a "live" feed or video chat. This difference lies in the spontaneity of interaction that can occur with real-time visual media. In an experiment conducted by Garau, Slater, Pertaub and Razzaque in which participants were exposed to varying levels of interaction with avatars it was found that all participants were able to "react to agents both rationally as computer interfaces, and non rationally as people" (Garau et al, 2005: 114). Generally, presence increased with the level of interaction. By using performance animation, actions and responses can be physically diverse, complex and appropriate because all movements are a real-time copy of the motions of the performer, providing a more complex, spontaneous and, assumedly, credible environment for interaction. The Garau, Slater, Pertaub and Razzaque study supports the notion that responsiveness is an important factor in communication. Our current findings support the hypothesis that increasing responsiveness even on a simple level can have a significant impact on certain aspects of people's social responses, both in terms of their perceptions of virtual humans and of their actions in response to them. (Garau, Slater, Pertaub and Razzaque, 2005: 105) Saltz views live performance as "inherently interactive" (2001: 109). Irrespective how rigid and demanding the rehearsal process is, a live performance is always subject to variation. Spontaneity is a crucial element of effective interaction. Saltz defines and describes many of the ways in which media can be used in live performance, including virtual scenery, interactive costumes, alternative perspective, subjective perspective, 33 illustration, commentary, diegetic media, affective media, synaesthesia, instrumental media, virtual puppetry and dramatic media (Saltz, 2001: 124- 126). He argues that inherent in the perception of liveness is interactivity and if media can comfortably sit within this interactivity then it becomes integral and not detached. Spontaneity, as it relates to paralanguage, is performative. The spontaneous give-and-take between performers and spectators, and among a group of sensitive performers, is integral to theatre's appeal as an art form, both in the most highly stylised genres of theatre and in "realistic" theatre. (Saltz 2000: 109) Time is an ingredient in communication. In conversation, for instance, the gap between a statement and a reply can radically affect the meaning. A slightly extended pause in a conversation can, with subtle changes in gesture and in different contexts, convey incredulity, thoughtfulness, anger, vagueness or lack of understanding. This interplay of timing and body language is the essence of performativity. Schechner regards performativity as underlying theatricality. Gestures, movements and sounds, "if not universally understood come close to conveying the same feelings everywhere". (Schechner, 1994: 43) Body language is a universal language. In Schechner's interpretation, performativity in physical communication is transcultural, unlike verbal or written communication, which is more local. This indicates that body language is more fundamental to communication than speech. 34 2.4 Motion capture in the arts 2.4.1 Motion capture in visual media From the late 1960s Myron Krueger started a series of groundbreaking experiments in human/computer interaction that have become iconic in technology and the arts. In 1970 he devised and directed a computer based installation called Metaplay in which a participant could see their digitised silhouette moving in real time on a rear projection screen. Movement of the silhouette generated changes to the projected image. In one iteration the person could draw lines on screen by moving his/ her finger; an open hand would erase areas of the screen. Krueger used an artist hidden at the back to draw on screen giving the person at the front the impression that the computer was tracking their finger movement. (Krueger, 1989: n.p.) The technique, while deceptive, was also a precursor of what lay in store for computers in the future. In today's terms his set-up was a simple operation, but Myron Krueger's experiments were conducted at the dawn of computer graphics and provided a valuable insight into real-time interaction between people and virtual space. The experience during METAPLAY had demonstrated that two people who saw their images juxtaposed would interact as though they were actually together. It also showed that people have a proprietary feeling about their image. What happens to it happens to them. If two people are together, they can see, hear and touch each other. The ultimate consequence is an artificial reality experienced through the participation of one's video image in the portrayed world. (Krueger, 1991: 37) The first recorded motion capture system was developed by Lee Harrison III in the early 1960s. Using potentiometers attached to a body suit he was able 35 to animate stick figures in real-time on a cathode ray tube attached to an analogue computer (Carlson, 2007: n.p.). Much of the early history of 3D motion capture was outlined by David Sturman in his paper, A Brief History of Motion Capture, presented at Siggraph 1994. In 1980, Tom Calvert used potentiometers attached to an exoskeleton that, in turn, was attached to a person's leg to gather analogue data of human movement. Around the same time, Polhemus developed a computer navigation system that used motion trackers. Through triangulation among trackers, their positions relative to each other could be established. By the late eighties several projects had the animation industry paying close attention. Mike the Talking Head produced at Silicon Graphics Studios was a digital puppet, a virtual head whose facial expressions and movement were controlled in real-time by a puppeteer. This concept was taken further by Jim Henson Studios in collaboration with Pacific Data Images, and Waldo C. Graphic was created. Waldo was a computer- generated puppet manipulated in real-time using a multidirectional hand-held controller that manipulated body and mouth movement. The real-time footage was composited with video so that Waldo was able to interact in real-time with the physical puppets. Influenced by this development, French company Videosystem developed a character called Max the Ghost who appeared daily on television for three and a half years. Other characters followed. Mario used the face tracking system, Face Waldo, created by Simgraphics (1992). Brad de Graf developed a face tracking system that was used for a real-time talking spaceship. He also developed cartoon show host, Moxy the Dog, which used electromagnetic tracking. The first full-length movie that incorporated motion capture was Total Recall in 1990. Motion capture has refined and diversified significantly since those early days and is now used regularly in popular movies including Shrek and Happy Feet. The television series, Jane and the Dragon, made extensive use of motion capture. Being a medium that translates live performance into a digital form, it was inevitable that motion capture would also be used as a way of digitally mediating live performance. Projected images had been used for many years 36 in live performance dating back to at least 1911 when Loie Fuller used film projection onto her costume in a Berlin revue. The first anthropomorphic interaction between screen and stage was in 1914 in Gertie the Dinosaur. Winsor McCay dressed in a costume reminiscent of a lion tamer, complete with whip, "controlled" an animated dinosaur called Gertie. As McCay bellowed orders Gertie the dinosaur would respond in the same way a trained lion would. He culminates his performance by walking behind the projection screen instantly appearing on the screen the same size but as a stick figure, moving fluidly from real life to projected image. This synchronisation between stage and screen has parallels with more contemporary works using games technology to create interactive mediatised stage scenarios. 2.4.2 Digital Performance Theatre is not concerned with reality; it is only concerned with truth (Sartre cited by Styan, 1981: 182) This section chronicles live productions that use motion capture and does not include performances that are primarily demonstrations of technology. Forms of motion capture have been used in live performance since the eighties, most commonly with dance. Coniglio and Stoppiello, who went on to form Troika Ranch, used flexible rods to provide digital information on a dancer's movements from 1989, developing software that is still the basis for what is commonly used now for creating real-time visual effects. Dance companies such as Riverbed, AlienNation Co. and Troika Ranch experimented with 3D motion capture at a time when real-time use of motion capture for stage was an expensive and technically difficult undertaking. Performances using motion capture were uncommon and were often more a display of visual technology than an evocative performance. Scott Delahunta stressed the need to engage the audience more actively and to emphasise performativity over visual display. Since the mid to late 80's (with precedents established 37 earlier), some dancers and choreographers have been exploring various interactive computer systems, but their works tend to integrate these systems into presentations in essentially proscenium-like settings and not engage in open and participatory models allowing the audience/user/viewer to cross the border between performance space and spectating space. (Delahunta, 2002: 114) A very early performance using 3D motion capture took place when Riverbed members, Paul Kaiser and Shelley Eshkar combined with choreographer Merc Cunningham to create Biped (1999). This was a landmark performance in which motion captured sequences were used to drive abstracted 'hand- drawn' models of dancers projected onto scrim at the front of stage. Ghostcatching, a digital dance installation, was a collaboration between Riverbed members Paul Kaiser, Shellay Eshkar and Michael Girard, and dancer/choreographer Bill T. Jones. A blackened room and ghostly hand- drawn figures move about in projected three dimensional space tracing paths as they move. These paths gradually fill the space and obscure the figures creating them. Robert Wilson's digital opera, Monsters of Grace also used prerendered 3D motion capture sequences as the basis for visuals. This blend of technology and dance was more art than performance, leading Birringer to remark: All of the Wilson productions I have seen, including his collaborations with East German playwright Heiner Müller, were marked by a depersonalized formalist aesthetic of the painterly which, comparable to Cunningham's choreography, emphasized technical execution and precision, drawing lines in space and filtering out all psychological and emotional connotations (Birringer, 1999: 365) Dance can more easily deploy visual technology than theatre because it 38 involves a unified sense of time in which the tempo of sound, movement and visual elements can blend harmoniously. Programs such as Jitter, Max MSN, Isadora, EyeCon, vvvv and EyesWeb are used commonly to create mediatized dance performance. Western Australian troupe, Jambird, used this type of technology to create stunning visuals in Metadance in Resonant Light. Chunky Move incorporated visually captivating real-time imagery developed by Frieder Weiss in their productions, Glow (1999) and Mortal Engine (2008). These types of technology differ from 3D motion capture in that the motion is captured in video as two-dimensional data. 3D motion capture potentially offers more flexibility and more possibilities but it will be years before it builds a similar artistic legacy in live performance. ACCAD at the University of Ohio has been a hub for interdisciplinary productions using 3D motion capture. Landing Place (2004) choreographed by Bebe Miller and animated by Vita Berezina-Blackburn was a collaboration among artists, performers and technicians, located in different parts of the US. In the production, prerecorded motion capture sequences were used to drive scenes, which were projected as visuals in dance. These sequences were projected onto a semi-transparent scrim melding the animations and the dancers into a visual unity. In 2005 Paul Kaiser and Shelley Eshkar, and Michael Girard along with composer Curtis Bahn worked with Trisha Brown and Bill T. Jones to create the dance production, Motion-e. This was carried out as a broad multidisciplinary collaboration through the Herbinger College of Fine Arts and the Fulton School of Engineering. Unlike the earlier productions in the late 1990s, the stage was set up as an optical motion capture area so that the dance onstage could drive real-time effects and sound. The real-time element had a gentle atmospheric quality sometimes forming columns of translucent light, sometimes radiant waves that moved in sympathy with the dancers' movements, a connection enhanced by the immediacy of the visuals. Other productions that used motion capture include Point A to B (2007) by UK based Urban Freeflow, performance visuals based on the urban sport of parkour, a way of creatively and acrobatically getting from one 39 point to another in an urban environment, and Loops (2001-), a continuous open-source digital artwork based on the movement of Merc Cunningham's hands. David Saltz's theatrical production of The Tempest (2000) was arguably the first production to use full axis 3D motion capture in real-time onstage. Ariel was played both as a live actor and a virtual actor. The actor was fitted with electromagnetic trackers, essentially controlling the movement of the virtual Ariel. This allowed for a consistency of motion and manner between the two forms. The virtual Ariel was not in any way live, a creation of a 3D artist, but Ariel's movements, mannerisms and responses were inherently humanlike, accommodating unanticipated variations onstage. Saltz concludes, "an age of interactive, live media is upon us." (Saltz, 2001: 21) Real-time puppetry was used In Jorgen Callesen's production of The Family Factory (2002); a play that used animated alter egos of four family members onstage. These animated doubles were rear-projected onto a row of screens at the rear of stage. Each double was controlled onstage by the corresponding family member. The technique involved the use of wired electromagnetic sensors that could be rotated to control a range of movements in the projected characters in real-time. The productions after 2000 were carried out at a time when the fervour that had surrounded the use of the latest interactive technology in live performance waned. In many respects some of the promise of what digital technology offered in the nineties as a tool in live performance fell victim to overexpectation (Daniels, 1999: 7). The publicity surrounding new technology often inflated the reality, and what was seen as achievable was caught in a web of technological immaturity (Grady, 2003: 167). Also, movies and advertisements were using sophisticated effects creatively with budgets live performance companies could only dream of, so when effects were used on stage they were often underwhelming or seen as passé. 40 Unfortunately, when an innovative new technology such as virtual reality comes along, people often do not distinguish between what it might be able to do and what it truly can do. (Grady, 2003: 167) It is early days in the evolution of real-time digital technologies and the experimentation since the 1980s has provided the performance world with a framework for how real-time visuals could work onstage. While some works were pivotal and others fell short of their grand aims, all helped to shape the parameters for successful mediatized performance. When devising visuals for performance, it is useful to recognise a distinction between performance and spectacle: spectacle appeals visually, whereas performance focuses more on what is communicated. When media is introduced into performance, performativity is easily compromised by the aesthetic and subsumed in the image. However, most productions could not fully explore a performative link between performance and projected image because they did not have access to a real-time environment. Many of the early technical limitations are being solved, allowing technology to become an invisible element where only its effect is seen. Moreover, the digital landscape comes closer to resembling reality as people's reality has shifted more closely to the aesthetic of the digital vision. When concrete space and time is no longer an untouchable basis on which the performance is built, then other more abstract or functional manifestations can become the new basics. This can then be redefined with each new performance – and even altered within a performance as part of the performance narrative (Peterson, 2004: 40) In 2008 there is a world of difference from the computing power, the sophistication of software and the state of the technology that existed only five years ago. Such is the speed of technological change that many earlier 41 works appear primitive in comparison to what now can be readily achieved. Technology moves forward relentlessly in a self-perpetuating drive for greater functionality and many of the early limitations are solved. The digital landscape also comes closer to resembling reality as people's reality has shifted more closely to the aesthetic of the digital vision. The increasing realism of 3D environments, the level of interactivity that can be achieved in real-time are also opening up the options for exploring psychological and emotional connotations through meaningful communication with an audience. Digital performance itself has matured, focusing more on using technology for artistic outcomes rather than stampeding headlong over the cutting edge of its promise. 2.5 Conclusion The real-time capabilities of motion capture open up new possibilities for the way digital images can work with live performance. The contextual review outlines the theoretical, historical and linguistic environment for practice. Virtual reality, liveness, presence and paralinguistics are part of the web of tensions that factor in the success of digital performance. Overarching post- structural perspectives such as Debord's spectacle, Baudrillard's simulation and Lyotard's metanarrative contextualise the role of modern media, shedding light on its nature and power of media, its ability to both persuade and distract. Together they give insight into how virtual reality fits into the physical world, how people relate to a screen environment and how visual images can work in unison with the variability of live performance. The way people communicate with each other involves physical signals more primal than speech. Eye contact, mood, gesture and attitude, the primary and secondary movements of the body, collectively called paralanguage, send signals about our intentions, our status and our state of mind that are not easily conveyed in words. The conveyed meaning is linked to the timeliness of the action or response, making it difficult to re-create in media unless the medium operates in real-time. The spontaneity of these body signals is intrinsic to liveness in performance. With motion capture the spontaneity of 42 liveness can be reproduced in an animated environment, allowing for sophisticated interaction between an audience and a virtual character. The success of this interaction depends on whether the virtual environment engenders a strong sense of presence in the viewer, and whether the viewer feels co-present with the virtual character. Improvisation in performance uses techniques that enable the performer to react in ways that are spontaneous but also to incorporate the viewer in "constructing realities". Improvisation is inclusive, opening up performance to participation by the audience in a way that allows them to influence the direction without wresting the control. The contextual review presents some guidelines for how people are likely relate to real-time performance: Our sense of "presence" in a mediatized environment is enhanced by the consistency of the virtual environment, by its familiarity and by the nature and extent of the interaction. The type of attention we give to live performance is different to the attention we give to media. Spontaneity is an integral element of liveness and is likely to strongly enhance a feeling of presence when used in a virtual interaction. The more realistic a screen world the more we tend to scrutinise it for faults. Experiences with new media in the nineties demonstrated the importance of being pragmatic, not lured by the promise of what technology might do rather than what it actually is capable of. In its totality, performance is the creative outcome and technology is a medium for delivering it; so creative decisions should be based on artistic intentions rather than demonstrations of capabilities. This research is not just exploring real-time animation as an augmentation of live performance but also the interaction between a person and virtual character, what Steve Dixon describes as the "digital double". Research into presence indicates that communication is an important factor in creating an engaging virtual experience. So the way in which a virtual 43 character communicates is just as fundamental to the success of a virtual performance as it is to a live performance. 44 3 Methodology ...we have the habit of combining certain concepts and conceptual relations (propositions) so definitely with certain sense experiences that we do not become conscious of the gulf, logically unbridgeable, which separates the world of sensory experiences from the world of concepts and propositions. (Albert Einstein, 1944: 289) 3.1 Research question The research has been conducted from the perspective of a developer of projection projects for live events. This research attempts to answer the following question: In what ways, and with what outcomes, can inertial motion capture be used in live performance? This question is explored as a practice-led case study comprising four research cases that use real-time motion capture. Each investigates a distinct way in which real-time animation can be used in a live performance. Primary case A Brush with the Real World: a real-time virtual artist. Secondary cases Chasing Shadows: a stage interaction between an actor and his shadow. Private Eyes: an instant movie experiment. Motionics: an exploration of performance animation and dance. 45 3.2 The nature of research The value of artistic research is related not only to the products of creative arts practices, but also to methodological, material and social processes through which they operate. (Barrett, 2006:1) Where practice-led research in the arts usually leads to the creation of an artifact, such as a painting, a play, a game, music, a poem, there are questions as to what aspect of this creation constitutes research. Does the research relate to the artifact, the technique, the process of creation, the tools used in the creation, or how any of the above impact culturally, academically or artistically? Can research simply open up new pathways to further research? Is the new knowledge gained through research an instance of "knowing how" or "knowing that" (the Ryle distinction)? Does an outcome need to be functional and what constitutes function? Practice-led research in creative arts has been subjected to widespread scrutiny as to how it fits into research. In general, research is undertaken to add to what is known. The Australian Department of Employment, Education and Workplace Relations further elucidates on the nature of research: 1. Creative work undertaken on a systematic basis in order to increase the stock of knowledge, including knowledge of humanity, culture and society, and the use of this stock of knowledge to devise new applications. 2. Any activity classified as research and experimental development is characterised by originality; it should have investigation as a primary objective and should have the potential to produce results that are sufficiently general for 46 humanity's stock of knowledge (theoretical and/or practical) to be recognisably increased. (DEEWR, 2007: 8) While methods used for practice-led research in the arts do not easily conform to quantitative and the broad norms of qualitative research, the boundaries that have confined creative practice as a method in research are dissolving as the more rigid disciplines, such as the sciences, are finding it increasingly necessary to diversify their methods to include more creativity and extrapolation. The dynamic reordering in the structure of knowledge as old fields merge and new fields are created has led to a situation where not only have the arts embraced research techniques used in science (in particular, technology) but science has embraced arts methodologies to help solve scientific dilemmas. Hassan defines four areas of contemporary culture characterising this flux: The creative process in science and art The new twilight zone of experimental science The incorporation of technology into the arts, both as theme and form The existential search for a unified sensibility (Hassan cited in Dixon, 2007: 150) While traditional research usually extracts new knowledge from what is known, there is increasing recognition that knowledge is gained in the process of creating. There is a blurring of the edges between art and science and for arts practice this has led to a greater emphasis on being able to express the methodologies of arts practice. 3.3 Postmodernism Duchamp really abandoned the idea of making art to take up the role of judging it and mocking it, he opened up the door for a great deal of what we now call Postmodernism. (Kramer, 1999: 131) 47 Common throughout postmodernist theorists is the recognition that a reductionist approach to enquiry was inadequate; this approach leaves a trail of exceptions that diminishes its authenticity. In culture there are no simple answers; the search for knowledge and understanding is obfuscated by pluralism and the indeterminate nature of truth. Postmodernism focuses on representation and on the gaps in cultural narratives. In research it is recognised that what is seemingly verifiable could not be described without using language that was unverifiable. Everything is relative. Postmodernism highlights the premises on which knowledge claims are made by exposing the complexity of the seemingly simple, drawing attention to the hidden inconsistencies and exceptions that undermine social metanarratives, exposing the plurality of these social models and recognising the subjectivity of knowledge claims. Any objective determination is based on a subjective acceptance of a procedure towards its validity. So Postmodernism does not logically undermine all authority or negate all knowledge. Nietzsche rejected the nihilism of many postmodernists, determining that we should strive for unified subjectivity. When Lyotard drew attention to the metanarratives that preserve the power of the status quo he also advocated a new focus on social justice. So deterministic, empirical, rational, logical conclusions can be drawn provided we recognise that any conclusions are an acceptance that there are no absolutes, that knowledge is ultimately based on subjective interpretation. As human beings we are a part of a world that we may also observe. Therefore we can only perceive it from the inside. (Weibel cited in Druckrey 1996: 344). Art as a field is highly subjective. Rules are meant to be broken. The artifact has no intrinsic value beyond the value placed upon it by the artist or the viewer. It is not based on the materials used or necessarily the skill of the 48 artisan. It has value as the implementation of an idea. In mediatized performance, a juncture where art and technology meet, there is a mix of methods, some stemming from creative practice while others are rooted in the more deterministic logic of software. The controlled chaos of creativity often mixes with hypothetico-deductive reasoning in the pursuit of answers that can be visualised but may or may not be possible. Postmodernism being relative, pluralist and subjective is a sufficiently diverse umbrella to cater for what is, in essence, an art/science fusion with a socially defined value. The postmodern artist or writer is in the position of a philosopher: the text he writes, the work he produces are not in principle governed by pre-established rules, and they cannot be judged according to a determined judgement, by applying familiar categories to the text or the work. Those rules and principles are what the work of art is looking for. The artist and the writer, then, are working without rules in order to formulate the rules of what have been done. (Lyotard, 1979: 81) Steve Best and Douglas Kellner (2000: n.p.) expand from Debord's situationism to contextualise interactive media and how it relates to the "the spectacle", distinguishing between "genuine interactive spectacle" and "pseudo interactive spectacle", the essential difference being that one is self created and open, such as one's website or a chatroom, whereas the other is moderated, such as radio chatshows or email feedback, where opinions and comments are sifted through a policy filter. "Interactivity" has become a buzzword of the digital age but is it an ideal? The internet, for instance, offers new ways for like minded people to organise and effect change, but there are such volumes of opinion being created digitally that much of it ends up spiraling down the cyber plughole or, otherwise, its relevance is diluted by the sheer quantity of other material. Nevertheless, the ability to actively engage rather than compliantly absorb is empowering because the internet user becomes a participant. 49 Performance is structured play. It is a "testing ground where possible events are tested out in a ritualised framework, so that we can weigh their consequences without our actions becoming reality or having real consequences." (Peterson, 2004: 40). Interactive performance allows others to play in the "testing ground". The innate interactivity of real-time motion capture offers a new way to structure play, and research based on practice can help determine the parameters. Postmodernism in performance is, as it is understood today, a dramatic form that is based on a semiotic understanding of traditional theatrical practice. The semiotics (or visual signs) are frequently deconstructed (taken apart) thereby opposing their original interpretation and leaving them open to translation. In this style of theatre the performer is more of an artist, seldom a character like an actor, and the content rarely being traditional in format or presentation. (Hurstfield, 2008: n.p.) When using technology in research there is an underlying assumption that whatever its limitations in a particular context, these will be overcome with the introduction of the next model. With the industrial revolution and the growth of twentieth century capitalism came increasing pressure for science, not just to provide knowledge about the natural world, but also to alter the world. Technology is undergoing a runaway evolution in an economic ecosystem that thrives on increasing efficiency and rapid turnover of commodities. Information has itself become a commodity, and differentiation between information and promotion is gradually disappearing. So technology represents a landmark shift in the direction of science from a purely inquisitive activity to a performative activity. In the late twentieth century, however, science occupies a crucial post in the formation of the postmodern condition. This is less because of the content of scientific ideas than it is due to the 50 increasingly powerful economic situation it enjoys under a capitalism that has become wedded to technological advance. Science can no longer afford to be the disinterested quest for the truth about nature, since its economic survival is bound up with the production of new technologies: science cannot simply desire to know; it must perform. (Hamilton-Grant, 2001: 59) So when assessing a new technology it becomes, not just a review of its function but, incidentally, a guide to its marketability. As scientific enquiry, in a postmodernist sense, is no longer recognised only as an impartial acquisition of knowledge but as a moderated activity whose direction and application are influenced by social, political and economic priorities, so the way in which we use and promote new technology influences the course of innovation. 51 3.4 Performative research The best way to predict the future is to invent it. (Ferryman in Thacker, 2000:158) It is difficult to address the question at the heart of this research through purely qualitative enquiry: there are too few precedents from which to draw any conclusions, and the most important outcomes are derivatives of practice rather than interpretations of practice. The research centres on discovering pathways to performance rather than assessing performance. These pathways are based on well-founded animation techniques that, through recently developed technology, can now be used performatively in live performance situations, with quality and stability approaching a broadly acceptable level. So the research focuses on the performative elements derived from a combination of the two artforms, animation and performance. Quantitative research is also inappropriate because the projected outcomes are intrinsically tied to the emergent processes of creative practice and the outcomes are neither measurable nor conclusive. However, there are methods in both quantitative and qualitative research that can be used in a practice-led approach. This study combines the mathematically derived field of software, with its clearly defined and often inflexible procedures that are sequential and repeatable, with the emergent approaches underlying artistic investigation. This is a multi-arts investigation that uses a visual medium to explore live performance using elite technology. Elements of the research are quantifiable; others are interpretive, and so the research can draw on the methods of both quantitative and qualitative research where appropriate, along with methods derived through practice, under the umbrella of performative research. When research findings are presented as such 52 utterances, they too perform an action and are most appropriately named Performative Research. It is not qualitative research: it is itself. (Haseman, 2006: 100) As pointed out earlier, not only arts practice, but technological innovation itself is performative, distinguishing it from more traditional areas of science. It seems an anomaly that human investigation has taken a seemingly irreversible turn towards performativity and yet performativity is still struggling to be recognised as a valid paradigm in research. This process of research brings a product or a set of conditions into existence that did not exist before. It uses performative methods of inquiry in that the research procedure is not hypothesizing and proving nor investigating and hypothesizing so much as adapting to the outcome of each step in a process that may lead the research in an unforeseen direction. The investigations are expressed in the language of practice and any determinations are intrinsically tied to the artifact(s) produced and drawn from the techniques used. The methods used are based on achieving experiential outcomes rather than reaching conclusions from a set of opinions or subjective perspectives. First hand interpretations of an artifact, such as a play or performance installation, can involve some sensations that are ineffable, and will lead to a "plurality of answers" that may be, in a Wittgensteinian sense, limited as validation. Also, "[a]udiences who will be satisfied with the outcomes of research share a context with the researcher" (Biggs, 2002: 10) limiting the generality of qualitative outcomes as a knowledge claim. The few examples where real-time motion capture has been used in performance do not provide a sufficiently broad understanding of the processes that underlie its use. Few people in live performance have been exposed to motion capture and so it would be hard for them to envisage how motion capture can apply to performance. To be useful and understandable to choreographers and devisers there needs to be more generalised research that gives an overview of what the technology can do and how this can be applied to performance. So it is from the practice rather than product that new knowledge is likely to come. These are the techniques, tools and 53 processes gained from the experience of practice (Mafe and Brown 2005: 12). 3.5 Practice-led research Pierre Bourdieu argues that tacit knowledge and the alternative logic of practice underpins all discovery, yet the operation of this logic is often overlooked because it is subsumed into the rational logic of discursive accounts of artistic production. (Barrett, 2003: 3). Practice is the language of art and through practice new forms and pathways emerge that feed the creative process. It is through practice that ideas are investigated, hypotheses are tested and new questions emerge (Burdick cited by Mafe & Brown, 2003: 82). Practice as research in its various forms, while affirming its place in academic institutions is widespread throughout the creative industries. These include movies, games, animation, print media, advertising, fashion and web design. Along with the use of practice as applied research, an emergent approach is a prevalent strategy driving rapid cultural and technological change. Although both applied research and emergent forms of research tend to use practice as the research framework, the two approaches are different. Applied research generally has an identified aim that helps to develop or extend functionality whereas emergent research is more akin to prodding a stick at the unknown to see what happens, in an informed and skilful way of course. Brown, Gough and Roddis (2004: 5) differentiate the two by claiming that practice-led research is a methodology whereas applied research is a type of knowledge. It could be argued that research into technology-driven arts projects necessarily involve both practice-led and applied research. The methodologies used in artistic approaches are considered necessarily emergent (Barrett, 2006: 7) in that the path and destination adapt to the journey, and so the artistic aims of the research change with the process of discovery. On the other hand computers are commonly used for artistic 54 outcomes and the creative process can still be emergent but the structured logic of software with its adherence to sequence and its repeatability also requires more deductive methods. There are many variables at play in arts practice and the outcomes are often unique and unforeseeable so deductive and inductive methods can only ever be a part of the research process. This is certainly the case in performance where structure is constantly in tension with individual expression, making portrayals of human interaction a complex and interpretive interplay of factors. When this is combined with highly evolved technology into a form that is related only to practice then any meaningful insight can only be gained through practice. MacMurray stresses the importance of practice underpinning knowledge: ...most of our knowledge, and all our primary knowledge, arises as an aspect of activities that have practical, not theoretical objectives: and it is the knowledge, itself an aspect of action, to which all reflective theory must refer. (MacMurray, 1957: 12) Mafe and Brown (2006: 1) conclude that the significant outcomes of practice in research are based on knowledge obtained through practice rather than the product of practice. The chosen investigations place greater emphasis on the development and application of techniques and processes that are useful in the broader context of the arts in priority to the artifacts produced. This is consistent with the emphasis by the Australian Department of Education, Employment and Workplace Relations on generalized outcomes that are transferable; it also reflects the need to extend beyond the mere application of practice to "improvement of practice" (Haseman 2006:98) so that: …the emergent core of practice-led research produces a value in excess of any single artifact or document produced. (Mafe and Brown, 2006: 11) The works, therefore, need to provide outcomes that are useful and useable to other people in the field. With emerging technology that has the potential 55 to interface with creative arts, there are opportunities not just to create new works but to discover new ways of working in a creative context, ultimately offering gateways to more discovery. 3.6 Investigative framework A set of smaller performance applications were chosen in preference to a large production as the practical basis of the research for several reasons. Firstly, the software has many features and, in a single production with the focus on performance, only a fraction of these features was likely to be used. With a case study that includes a representative selection of cases, a much broader cross-section of techniques could be investigated, because each case can be designed with a unique set of techniques in mind. Secondly, a full-scale production would require a collaborative pool of expertise, and people with experience in motion capture are hard to come by; using a case study does not require the same level of collaboration. Thirdly, there are few precedents for using motion capture in performance and so there are likely to be plenty of unforeseen hurdles, some of which may not be overcome. Placing all eggs in the one basket could lead to a situation where a lot of energy was expended for no final outcome. However, if one case cannot be implemented then it does not risk the outcomes for the entire research. The cases used in the research were designed around the attributes of inertial motion capture that might adapt well to performance. It is a real-time medium and so is projected as it is performed It uses full axis 3D data, which theoretically allows it to use the vast realm of 3D techniques including effects, behaviour and physics It can simulate the nuances and spontaneity of paralanguage It is unrepeatably interactive in real-time, as distinct from games interaction which uses a network of preanimated actions arranged in the form of trigger trees, along with programmed behaviours and simulated physics, giving an impression of realness It can visually represent conversation in real time 56 For the case study to be of value as performative research, the design of each case needed to focus on each as a means to a performance rather than as a technological curiosity, and so motion capture is a medium and not an end. The ability to create real-time interaction is an important focus in the case study as it sets up a three-way communication channel among the actor, screen and audience. Spontaneity is its strength. With motion capture, movies can be created that have a level of variability and responsiveness not found in other interactive media. The technology is still evolving but it heralds a plethora of possibilities that may change the way we perceive arts and entertainment. This research was set up from the early stages as an "enabling" industry partnership with visual technology company, VRS, and much of the technical equipment and logistical support through the investigations was based on this arrangement. The software engineer at VRS took an active interest and spent time in the early stages helping me with the motion capture software. However, his time was limited so it was necessary to be self-sufficient, learning the software and solving technical hurdles involved in each investigation. In the latter stages, once the technical aspects were mastered and real-time environments were at a stage where performance-based experimentation could begin, several performers were invited to work in a number of collaborative movement/dance improvisations to determine the types of movement that worked best with motion capture and to uncover new possibilities. The overall innovation performance of an economy depends not so much on how specific institutions perform, but on how they interact with each other as elements of a collective system of knowledge creation and use, and on their interplay with social institutions (Johnston cited by Grigg, L, Johnston, R & Milsom, N 2003: 15) 57 The research is interdisciplinary involving performance, animation and visual technology. Interdisciplinary research is, according to Julie Thompson Klein, " the necessary 'churn' in the system". It taps into "a set of dynamic forces for rejuvination and regeneration, pressure for change and the capacity for responsiveness" (Thompson Klein, 2000: 22). An interdisciplinary approach tends to prioritise the problem ahead of the artificial categories of knowledge organisation. Once the problem is recognised experts from relevant disciplines can then collaborate to solve the problem. However, interdisciplinary research does not need to be collaborative (unlike multidisciplinary) so long as the researcher conducting the research has the skills and the knowledge to solve the problem. The rapid pace of innovation is driving structural changes in the way knowledge is organised and the way research is conducted. Across many areas of research, multidisciplinary, cross-disciplinary and interdisciplinary approaches involving collaboration have become the norm. Convergence and diversification reflect the state of the flux in research driven by the growth of the knowledge industry and the rapid evolution of digital technology. The growth of computing power and a widespread acceptance of some of the major tenets of Postmodernism have opened the way for the exploration of complex problems that transcend traditional disciplines. Over the course of this century, metaphors of knowledge have shifted from the static logic of a foundation and a structure to the dynamic properties of a network, a web, a system, and a field. Perceptions of academic reality, though, are still shaped by older forms and images. (Thompson Klein, 2000: 22) 58 3.7 Methods 3.7.1 Case study If you tell me, I will listen. If you show me, I will see. But if you let me experience, I will learn. (Lao-Tse cited by Clarke, 2000: n.p.) Lao-Tse's philosophy on learning became the basis of a method of enquiry practiced by ancient philosophers, Confucius and Han Fei-Tzu, that is now referred to as "case study". Case study is a method often associated with sociology but has been used extensively as a research method in many areas of research, from business to nursing. Stake (1995) and Yin (1994) identify sources of evidence for case study that include documents, archival records, interviews, direct observation, participation observation and physical artifacts. The case study, I argued, is best defined as an intensive study of a single case (or a small set of cases) with an aim to generalize across a larger set of cases of the same general type. (Gerring, 2007: 65) Yin (1993) identifies three types of case study: exploratory, explanatory and descriptive. The case study used in this research is exploratory in that, while the projected outcomes are clear, their achievability, effect and function are not verified until the case study is complete. As demonstrated by the works of Darwin, Freud and Piaget, case study is an effective methodology for exploratory research. This is in part because a case study is more likely to expose causal mechanisms than, say, cross-case research. Case study is useful when detailed knowledge of a small sample is more insightful than targeted knowledge of a large sample. It is therefore an appropriate research method for a topic of study that is complex and has few representative cases. Much of what has been written on case study as a method in research 59 relates to its use in the social sciences where qualitative determinations can be drawn. However, the method can apply just as well to performative approaches, drawing conclusions, not from social contexts but from processes in practice. Case study is generally an empirical process that, in this research, can be based on observations of the processes, functions, techniques and procedures as well as in observations of the performance. As the case study is also the creative practice, however, these results are performative. That is, the creative process towards a practice-led outcome is the research framework, and the findings are in the language of practice. Grounding the research through practice-led enquiry was necessary because there were insufficient examples of its use in the field from which to draw conclusions by purely empirical means. Also, the creative practice covers a whole range of software attributes in combination with a high level technology to map a creative performance framework. This is a complex blend that is hypothesis-generating rather than hypothesis-solving (Gerring, 2007). The case study is not representative of a population of similar existing performances but provides a useful set of templates for future creative endeavours. Each investigation is designed to reach separate performance objectives but within a framework common to all; it is the extent of all observed and performed outcomes from all of the investigations that are used, in a holistic sense, to map a creative landscape. The case study focuses on the functionality of inertial motion capture as a medium only, and the performances resulting from the use of that medium affirm its suitability. The standard functions of the software are not objects of study per se, as these have been developed long before to carry out these functions. What is relevant is the context in which these functions are used to meet a creative objective. Case study is less of a methodological choice than a 60 choice of what is to be studied. (Stake 2000: 435) The research is structured as a practice-led case study comprising four cases. Each case investigates a different performance scenario and each sheds light on the processes, the attributes and the performance outcomes, including ways in which this medium can be used to create a performative interaction between the world of screen and the real world. The case study comprises a major case that explores interactions between a real-time screen character and audience, and three minor cases, each exploring a different interactive dynamic. The relationships discovered among different elements of a single case have a prima facie causal connection: they are all at the scene of the crime. This is revelatory when one is at an early stage of analysis, for at that point there is no identifiable suspect and the crime itself may be difficult to discern. (Gerring, 2007: 41) These cases help to determine ways in which motion capture can be used at live events, and give an indication as to whether it is ready now for a broader role in live situations and whether it is likely to become widely used at live events in the future. As there were precedents to draw from, the aims of the research are broad, determining in general terms the technology's suitability to live performance as well as exploring pipelines to creative output, the factors influencing this type of mediatised performance and a spread of creative possibilities. Nevertheless, it was important to gain sufficient insight to be able to construct, or at least know how to construct, real-time narratives. As there were few existing examples to draw upon and these together did not cover a broad range of uses, the cases were devised with the aim of representing an overview. The cases, therefore, needed to be diverse and general in order to explore: 61 the dynamics of hybrid performance animated environments that were conducive to live performance ways of engaging and intercommunicating with an audience the strengths and limitations of the software and hardware presentational considerations such as the visibility of the technology strategies that avoid the splitting of focus differences in the way an audience perceives onscreen performance compared to live performance the use of light in a situation where the two forms have conflicting requirements In a performative sense the case study is not just based on observing phenomena to find causal mechanisms, but also envisaging effects and determining how to cause them. It is a similar situation with process-tracing which, in a performative case, becomes process-constructing. So, rather than observing existing cases and examining them for causal links, the cases are constructed, and it is in the process of construction that experiential knowledge of the creative processes of practice is gained and causal mechanisms are understood. In this respect, the case study is the construction of the cases and this process of construction exposes new knowledge. Each behaviour has a side effect and the sum of the side effects gives the desired functionality (Steels, 1991: 454.) The case study follows the "diverse" model in that each case differs from the other cases so that, in combination, the cases cover "maximum variance along relevant dimensions" (Gerring, 2007: 97). It is through the diversity of the cases representing maximum variance that an overview of the processes, 62 as well as the causal and performative mechanisms can be framed. The experience gained can be used to construct more complex and comprehensive projects such as complete "live" movies that can be manipulated at any moment in a story: participatory cartoons in which members of the public can join in as a character: or virtual installations that are a hybrid of animation and reality television built on spontaneous interaction and a heritage of improvisation. Each case in the case study explores a broad range of software features that are likely to be useful in a live performance scenario and these are listed below: A Brush with the Real World: a real-time virtual artist. Interactive installation Character interaction using voice, expression and body language Movement and dance using a Digital double Painting using a proxy Gestural interface Spatial triggers Virtual improvisation Participatory animation Chasing Shadows: a stage interaction between an actor and his shadow Mediatised theatre Shadow puppetry Swapping between recorded and live action Theatrical interplay between physical and virtual character Movie clip triggering Private Eyes: an instant movie Interactive movie Real-time acting Multiple characters in the one scene 63 Complex real-time environment Cinematography Atmosphere Motionics: exploring techniques for creating real-time 3D visuals for dance Multiple virtual bodies The use of video in animation Dynamic camera effects Reactive scenery Targeted projection Motion trails Interpretive and liminal bodies Virtual lighting effects 3.7.2 Creative practice Practice-led research builds on the fact that creative practice has always been a knowledge generating activity. (Mafe and Brown, 2006: 10) Creative practice in research needs to be differentiated from the normal practice of the artisan and have identifiable goals. The outcomes need to be accessible and open to scrutiny and the structure of the research and the written exegesis need to be clear in theorizing, explaining and contextualizing the research and its outcomes. The outcomes should be sufficiently general to be useful beyond the scope of the research and be able to be applied to other research (Mafe and Brown, 2006: 4). This formal approach to scholarship expected in research driven by a rationale of consistency and order is in contrast to the more chaotic and reflexive approach typical of arts practice. 64 The primary research methodology is problem-solving itself, which often, in practice, cannot be fully explicated. There are, of course, methods, techniques and tools that can assist the researcher, but in many cases, the problem finding and solving processes retain an element of the black arts. (Scrivener, 2000: n.p.) The practice uses both animation and performance as well as sophisticated emerging technology to bond them into a coherent performance framework. Creative practice, in this convergent structure, requires a range of methods as each area has its own unique workflows. Firstly, the design of each investigation looks at the attributes of real-time animation and the unique strengths of live performance to formulate a common ground upon which to create. The animation is devised, storyboarded, characters modeled, environments built, and, in parallel, the performance is devised and developed through rehearsal and improvisation. Through motion capture the animation and live performance are brought together regularly to ensure that they work in combination. The methodologies of creative practice are generally considered to be emergent (Barrett, 2005: 6). However, I would argue the approach could be either emergent or conjectural. In an emergent approach to creative practice the creative pathways adapt to unexpected insights that may not be part of the original intention. With a conjectural approach to creative practice the creative pathways adapt to unexpected insights but without altering the original intention. The emergent approach is led by inspiration, or what Gerring calls "lightbulb moments" (2007: 5), while the conjectural approach is visionary, envisaging a creative outcome and working towards it. In this respect the approach to creative practice used in this research is conjectural, using emergent methodologies. Performance and dance were used in the research for testing the virtual environments and characters as well as developing the performance 65 framework. Improvisational techniques were particularly useful in helping to formulate ways of interacting with an audience that would emphasise the interactivity of the virtual environments in performance. We rarely want "a" motion, but rather want "the" performance. The specifics of the performer and performance are important. Just as actors are trained to know that there is no such thing as a "generic" movement, the sense of character, context, and intent should be conveyed in movements. Just as with stage and film actors, motion capture directors work with performers to achieve what they want to see. Good motion capture preserves this. (Gleicher and Ferrier, 2002: 2) Each case had a specific performance style. A Brush with the Real World uses improvisational techniques to lure passers by and involve them in visual and verbal exchanges. These techniques stem from the work of Canadian Keith Johnstone whose book, Improvisation, sparked an interest in ways to deal with spontaneous performance situations, and my years of working with Tony Kishawi using mask, mime and clowning techniques. Chasing Shadows is built on interplay between two onstage characters, one real and one virtual. It is a choreographed sequence, and motion capture is used to synchronise movement between the actor and the virtual character. In traditional clowning (as with Commedia dell'Arte), differences in status present fertile ground for comedy sequences, and this piece explores a change in the normal relationship between a man and his shadow, the actor and the virtual actor. The other theatrical piece, Private Eyes, uses reflective monologue and is more subtle, sober and mood evoking. Onscreen, this piece is in the form of a movie, using a progression of scenes and camera positions, along with rehearsed movement and gesture. Motionics explores a range of visual techniques that could be used with dance, and dance was used to test each technique in a performative way. Each environment has its own flavour, and 66 dance is used improvisationally to help determine a movement style or dramatic feel that suited each of the visuals. 3.7.3 Industry input The role of knowledge in economic development has been transformed from a minor player to a driving force. The industrial economy, based on goods and services, is being matched, and in some cases displaced, by the global knowledge economy, based on the production, distribution and use of knowledge. (Grigg, Johnston & Milsom, 2003: 15) Industry input was sought initially for determining what the research question should be. I had accumulated a number of ideas for using animation at public events and, rather than rely on informed intuition as to which was most appealing as a public project, I presented the ideas as a folio of prospective projects to a selection of people involved in performance and festivals to get feedback on the type of animated production they would envisage as being successful. On the basis that people may not be aware of some technologies available and ways in which these technologies could be used, I created nine Flash animations, each demonstrating a way of using projection at live events, including some that used live performance. Animations representing hybrid shadow puppetry, performance animation, previsualisation and 2D event projection evoked the most interest. One person with a long involvement in circus was keen to discuss the use of virtual simulation as a method of trialing a potentially dangerous risk to the performer. This method is already used for defence, emergency services and pilot training and, if applied to circus, it could help drive innovation. New acts that are potentially dangerous could be simulated first, minimizing the risk factor. In combination, the industry feedback helped identify a practical case(s) that had industry application and could be used as the basis for research. Below are images from the folio of Flash animations presented to people in 67 key arts groups for feedback. Figure 3: Using rear projection animation as a public window installation Figure 4: Large-scale shadow puppetry using animation Figure 5: Using effects triggered by movement 68 Figure 6: Projected background moves with person's stage orientation Figure 7: Projection within sets onstage Figure 8: Animated shadow 69 Figure 9: Shadow projection on the tent wall in circus Figure 10: Improvisational screen character 3.7.4 Applied methods There is an increasing role for applications-driven approaches in many spheres of research, and a growing share of the national research effort is occurring in commercial settings. These trends are particularly pronounced in the fields of science and technology research, but are by no means confined to them; similar trends can also be observed across a range of the social sciences and humanities. (Kemp, 1999: 1.10) The investigations each had fundamental technical challenges and, frequently, new solutions to problems created new problems to solve. Each time a new workaround was developed it was tried with the inertial suit to determine the effect and what unforeseen consequences might be caused. 70 Some problems were suited to a holistic approach whereas others needed to be isolated because it was too hard to determine the mechanics of the problem otherwise. While technical issues were very important to resolve, the theatrical parameters also needed to be determined; these were limited to the scope that the technology allowed, as well as the way audience and actors related to screen environments. Many of the technical hurdles and artistic aims could be trialed to gain a better understanding of how they worked. Some of these were: Developing a gestural interface to operate a colour pallet Tracking anomalies in movement data Exploring ways in which effects could be used Determining improvisational techniques and theatrical strategies that would prove effective in holding people's attention Assessing how people would react to a real-time screen character Working out the dynamics between a live actor and the virtual environment Identifying moments where the continuity of dynamics or aesthetics might be broken The flux caused by rapid innovation as different fields converge frequently pushes researchers into pockets of unfamiliar territory. Through the collaboration with visualization hardware company, VR Solutions, opportunities arose to apply and expand on the skills and knowledge needed for the research in a real-world context. On a number of occasions early in the research I worked as the performer in motion capture sessions and, after becoming more familiar with the motion capture software, demonstrated real- time motion capture at a number of games studios and at the GamesConnect Conference at the Brisbane Convention Centre. Later I worked on motion capture for a UQ/industry collaborative project, creating and editing motion capture data. A new beta level ultrasonic tracking system was sent from the UK to augment the inertial system and I was employed to help set up the hardware and software. Each of these experiences broadened my knowledge 71 of motion capture and improved workflow in practice. In combination they provided an overview of the entire process in the context of industry workflow, providing know-how not readily available in tutorials and books, and were valuable in reaching a level of competence to be able to complete the investigations. Late last year I traveled to Perth to set up motion capture sessions for a major games company. Since then I have regularly been involved in demonstration and content creation with motion capture. In August I attended Siggraph 2008 in Los Angeles where I was able to see first hand new hardware and software developments in motion capture as well as aligned technologies and presentations of the latest research. Each of these industry-based experiences has provided critical learning experiences in time based workflows and an overview of the entire process of motion capture. 3.8 The working environment The inertial motion capture suit used in the research is the Gypsy Gyro 18, designed and produced by UK based Animazoo. It is an inertial system that uses 17 inertia cubes placed at strategic points around the body. The inertia cubes, made by Intersense, use accelerometric, magnetic and gyroscopic data to determine 3 axis orientation (pitch, roll and yaw). These small cubes are attached by leads to a transmitter worn on the chest or back. The transmitter sends data to a receiver, which is plugged into a computer. This Gypsy Gyro was the first commercially available suit produced and was shipped to Brisbane in late 2004, just before the start of this research. The software used for the motion capture was Animazoo's Cobra 7.11 and Autodesk Motion Builder 7.5. The most widespread type of motion capture is optical motion capture. It is commonly used to create animation for movies and games. Optical motion capture typically requires dedicated studio space and this limits its application to live situations. Inertial motion capture, while not as accurate as optical and 72 subject to drift, has no problems with obstructed sitelines (occlusion) and the unedited real-time data is acceptable. The suit, fitted with a transmitter and cables, is marginally restrictive but it allows the performer to operate wirelessly hundreds of metres away from the receiver. A big advantage is its portability. The entire system, including laptop, can fit into a medium size case. It can be up and running within half an hour and can be used in a wide variety of environments. 73 4 Description of practice 4.1 Overview The practical research explores four cases in which motion capture can be applied to live situations. These examples primarily focus on interactive relationships between the actor and the audience or the actor and the visuals rather than the artistic merit of the visuals. (1) Interactive installation (2) Mediatised theatre (3) Instant movie (4) Interpretive visuals Many of the techniques in these cases are well known to those who work with animation. Applying these techniques using motion capture to create real-time animation is less well known. There are a number of well- documented examples of real-time animated characters being used in a public forum; these are typically real-time applications by animation production houses rather than hybrid performances by live performance troupes. This difference in emphasis is important in determining how the subject matter is approached and how the technology is used. In live performance the "liveness" is primary and animation is an augmentation, whereas in animation the emphasis is on the animation, and the liveness is an attribute that helps the workflow but is disconnected from the final product. The implication that programming is the actual art and what artists make of it will always remain secondary matches up with Friedrich Kittler's suggestion that only our ignorance makes us confuse the products of media with art. (Daniels, 1999: n.p.) 74 For motion capture, or for that matter any form of media, to have a connection with live events it needs to be more than a fusion exercise. It was important to explore composite forms where motion capture was a tool rather than the focus of the visual experience. Otherwise its interest may lie only in its novelty. A performance can "wow" an audience with cutting edge effects that are only captivating until the next technological breakthrough consigns it to obsolescence and, with the frequency of innovation in visual software and exponential increases in computer speed, it is inevitably a fate just around the corner. Chasing Shadows and A Brush with the Real World are theatrically based installations that use animation techniques applicable to dance. Both in different ways explore how a projected story can be created live, including how scene changes can occur, how a scene can operate with more than one character and how the characters can interact with objects in the scene. The installations are both highly portable, and require few special conditions outside those that would normally be expected when using projection. The performance is shown as it is created and so the content cannot be edited. There were often glitches but the chances of them occurring or being seen were minimised in the choreography and camera perspective. The technology presented opportunities to develop or enhance ways of interacting and this was the primary consideration when looking at how these cases were constructed. Although there was a need to adapt to the limitations and to frequently use workarounds, there was an assumption that any problem could be solved and any inadequacies of the hardware and software would gradually be eliminated as the technology improved; this often proved to be the case. 75 4.2 A Brush with the Real World Figure 11: A Brush with the Real World A Brush with the Real World is a real-time performance installation. The screen character, Rupert, is an artist and the screen is a window into his world. He works in his small studio painting on his canvas but Rupert can also paint in the air around him. His brush becomes his main form of communication with his passing audience. Rupert is also able to communicate through actions, moods and, most importantly, his paintbrush. He can paint on the inside of the screen and in the virtual air of his studio. A Brush with the Real World allows for spontaneous visual exchanges between screen character and audience. All movement is transposed from the actor onto the screen character allowing him to communicate directly with the audience. The actor can respond to the audience and that response is portrayed by the double on screen in the visual language of the artist. The actor's role is integral but invisible and the audience perceives itself communicating with the artist and the art. It is a virtual theatre improvisation, where the rules of an improvisational game apply, structured around the 76 characteristics, the quirks, foibles and activities of the screen character and his ability to express through the paintbrush. The communication between Rupert and the viewer subtly unfolds. Initially Rupert is engrossed in his own world, painting and pottering about in his studio. His first awareness of the audience is like someone who is suddenly aware that someone is peering into their window; he tries to block them out and when that becomes unsuccessful he attempts to ignore them. This can only go on for so long before he feels the urge to express his indignation. However, he gradually warms to those peeping into his world and starts a Figure 12: Layout for A Brush with the Real World dialogue, firstly through gestures and then through his art. Finally, he uses his voice, talking directly to them. So the sequence allows the interaction to evolve as the screen character, at first detached from any involvement with the audience, progressively adds to his methods of communication until he includes them wholly into a spontaneous and engrossing exchange. A Brush with the Real World uses a gestural interface to control some actions in the projected scene. A gestural interface is an interface in which actions are triggered by the body movements of the operator. With Rupert, positions of the non-painting arm were used to control colour while forearm rotation was used to control flow. The paint was activated when the hand was a certain distance forward of the body and any distance beyond that determined the extent of the paintflow. Late in the development of A Brush with the Real World real-time speech was added to Rupert's communication repertoire. This gave Rupert another layer of communication that could add to the dramatic depth of the character and his ability to interact. In a performance situation Rupert could join in with dialogue onstage, adding another step in an unfolding communication 77 dynamic with an audience/observer. Facial expressions that express feelings and emotions were also added and these could either be triggered by a third party with a keyboard or game controller, or by the performer using a hand controller such as a thumb joystick. The speech used MotionBuilder's voice recognition engine to trigger blendshapes, giving a realistic impression of speech. In live situations this can work well but, because the mouth movement is triggered by sound, any stray sounds such as audience noise can also cause the mouth to move. So when using this voice engine in live performance precautions should be taken to filter out extraneous sounds. Although A Brush with the Real World was designed as a theatrical installation, the outcomes from this type of performance are also applicable to dance. The screen is a visual extension of the stage and can be used to enhance the plot, the characterisation or context of the live action. With Rupert's ability to speak and to express moods, characterisations shown onscreen through dance can be quite complex, extending the nature of dance visuals to being a sophisticated communications medium. The real- time nature of the visuals may help to overcome the limited ability to emotionally connect with the audience, prevalent in dance productions of the past that used motion capture. According to Birringer these productions "were marked by a depersonalized formalist aesthetic of the painterly" (1999: 365). The A Brush with the Real World environment was also used to explore the "digital double" as a way of conveying dance in an animated form. Through a digital double the dancer can be represented visually in an illusory world. In a series of trials, dancers were involved in creatively exploring the virtual character as a real-time visual element in dance. Using Rupert and his virtual paintbrush, the dancer could dance his/her way through a painting as it is created. The trail of paint became a map of the dance. Other relationships between dance and the paintstream were explored, including one iteration where the colour of the paint trail changed according to the height of the brush, and in another the paint was constrained to a set of planar surfaces. Other experiments explored the use of dynamically changing textures. 78 While the aesthetic may be the same whether the visuals are real-time or not, the formalism is not a big factor in real-time because the dance is liberated from the pace of the visuals. The onscreen environment is the puppet of the dancer. So when paint trails from Rupert's hand as he dances through his studio it is the dancer controlling its path and its timing. Through this flow the dancer can manifest precision or symmetry, or create recognisable artifacts that can be woven into the choreography. The paint can also be used as a colour trail of the dancer's movements, a representation of movement over time, as has been used previously with Riverbed productions of the late nineties. 79 4.3 Chasing Shadows Figure 13: Chasing Shadows Chasing shadows is more traditionally theatrical, a combination of drama and shadow puppetry. The audience members are onlookers and the interaction is all on stage between the live actor and his shadow. A detective-like figure stands near a lamppost waiting for a bus. His shadow casts on the wall of the building behind. The shadow gradually takes on a life of its own, becoming another actor onstage, and a scenario ensues among the inadvertent actor, his clumsy shadow and other shadows that pass in the night. With Chasing Shadows the technology is invisible and the focus is the theatrical action onstage. Motion capture is used to generate the shadow of the actor and this is projected in real-time on the wall (projection screen) behind. The actor's shadow is in fact a real-time projection of the actor's movements. It is worth noting here the reason for choosing a story about a shadow. One inescapable problem whenever there is interplay between live performance and projected images is that they exist in different dimensions, the world of live performance and the world of cinema. A shadow, however, is a naturally occurring two-dimensional projection that exists in a three dimensional reality making it a natural player in a hybrid performance that does not strain the bounds of credibility. 80 The shadow becomes aware of a sense of independence, gradually exhibiting more individuality until it becomes a different player. However, as the plot concludes the shadow recedes to its traditional role. The real-time generation of the shadow is intertwined with movie clips to create the subtle shift from obedient mimic to independent entity. Technically, a smooth transition is difficult to achieve with inertial motion capture because the motion data is relative. As an example: if the performance animator walked in a circle back to the exact same position (s)he started, the character on screen may not be in the exact same position it started. While there are probably ways of automating the transitions, there was insufficient time to explore this area and so the changes from live motion to movie clip in Chasing Shadows rely on the actor's intuition. 81 4.4 Private Eyes Figure 14: Private Eyes Private Eyes started as an offshoot from Chasing Shadows. It is a foggy night in the inner city and a lone misty figure stands under lamplight on a street corner waiting for a bus. He slowly becomes aware of and gradually preoccupied with eyes watching him. The headlights of a passing bus or dark figures passing in the foreground occasionally punctuate the scene. Lighting and visual effects are used to create a Singing in the Rain atmosphere. Private Eyes uses particle effects, background video, scene changes, camera switching and mood lighting in real time to test the extent to which an animated movie can be performed "live". Scenes with multiple characters, using a blend of movie clips and real-time motion capture, also added to the range of possibilities. From a dramatic perspective Private Eyes sees into the thoughts of the actor on stage. These thoughts transport the audience to another time and place as the actor reminisces about the past. While the structure relied less on the animation being real-time than the other investigations, it provided a guide as to how far effects could be pushed before the visuals were degraded by dropped frames or stalling. Within the limitations of this study Private Eyes explored a range of movie techniques with the emphasis on determining 82 whether they could adapt to live performance scenarios. 83 4.5 Motionics Figure 15: Motionics 1. By attaching a camera to the head of an avatar the audience can follow the eyes of the live actor through a journey of his/her imagination. Figure 16: Motionics 2. This is a simple experiment in which particle emitters are constrained at varying distances from parts of the body. Figure 17: Motionics 3. This example uses simple emitters that change colour over time to visually represent a dancer's movement. 84 Figure 18: Motionics 4. Software mirrors are used to show different perspectives at the same time. Dance is usually set to metronomic rhythm of music, and visual media is often used as a synaesthetic enhancement to the movement of the dancers onstage. These visuals are sometimes generated using 2D motion capture, a form of motion capture that uses a single camera source to map the shapes of the dancers or a series of identifiable points visible on the dancers so they are able to control the motion and timing of the visuals. 3D motion capture extends this into the three dimensions of virtual space. Motionics is not a single performance structure but a series of trials testing a wide range of visual tools individually. These trials use particle emission and more conventional animation methods to determine ways in which animated visuals could interact with dance. There are many techniques in animation that can be applied to visuals in dance and so this study, rather than focus on a single performance-based outcome, explores a range of visual techniques to reach an overview of real-time performance. It is a study portfolio rather than a single case per se. Visual effects, driven by motion on stage, can be viewed in real-time from a range of virtual camera perspectives that can be toggled or animated. Examples were recorded of scenes created from the view of a character on screen, from multiple camera perspectives that could be switched, from the perspective of a stationary camera locked to the movement onscreen, and from an animated camera. Also, projected video was used, either as a background within the 3D environment, a dynamic projection onto the body of the character, or way to add Flash animation to the scene. Basic video and 85 particle effects were used in scenes without discernible deterioration in the real-time performance. This was subject to such factors as the number of video and particle effects used and complexity of the scenes. Multiple characters onscreen were driven by a single dancer and software mirrors were used to create the illusion of multiple dancers or to extend the scene without adding more geometry. Reactive approaches were also used in several ways. One scene used a sea of blinking eyes that were locked onto the actor; wherever the actor moved the eyes followed. Trigger points were set up at several locations in a scene comprised of a black flashing, reflective figure dancing with staccato movement in front of continuous looping lightning; when the dancer entered certain points in the scene, it caused sparks to fly. There are many attributes of the MotionBuilder environment that could be explored artistically in a real-time projected environment and so a broad determination of its use in live performance is difficult to assess other than by using a broad performative approach. The case in dance explores in broad terms the accepted capabilities of the software and how they work with relatively unexplored hardware. While each of the cases was successful, at least in principle, an acceptable visual performance can only be achieved if there is consideration given to a whole range of factors. Variables such as the complexity of geometry, resolution of textures, the extent to which video and CG effects are used all influence the quality of the output. These are factors that make up the balance between technical capability and artistic intention. 86 4.6 Log of practice Early in 2005 I set up an initial contact with Jon Newell from VRS, hoping to establish a collaboration with the company to gain access to their motion capture technology, and most importantly, their Gypsy Gyro 18 MOCAP suit. It was April before any practical interaction commenced and, despite many setbacks with both software and knowledge of the technology, it opened up possibilities as to what could be achieved in a real-time interactive hybrid performance. As I learned more about the technology I decided to aim for a project that explored the potential of the MOCAP suit and my background in both theatre and illustration. This gelled into the notion of creating a real-time interactive painting environment, A Brush with the Real World. The motion capture equipment was only available when it was not being used for income generation and so the research needed to be flexible enough to adapt to the windows of opportunity that arose. The technology was new, entailing a lot of experimentation and learning. Early in these studies persistent glitches occurred when generating real-time animation with the inertial suit. Some of these were minimized by workarounds. For instance, the inertia cube placed at the base of the spine was pivotal but was subject to shift. I tried using Mastix to directly glue it onto the skin and this helped. Later, I contoured a pad to sit snuggly in position and this worked just as well. I used a similar method for the foot trackers and this marginally improved the results. The angular rates and calibrations were persistently tweaked; this also gradually improved the motion. The latter part of 2005 was spent developing a painting environment. Through early trials it became clear that all aspects were feasible but progress came to a standstill when unexpected software conflicts cropped up. In December 1995 Ali Kord, head of UK based Inertial suit manufacturer Animazoo, visited VRS and demonstrated an upgrade that alleviated the more glaring deficiencies in the software and provided a more stable environment. I was now sufficiently familiar with the software and equipment to handle motion capture sessions unaided. However, in March 2006 the 87 inertial suit was leased to Irrational Studios in Canberra for 4 months and so the project again came to a temporary standstill. This called for a strategic change. While the equipment was still available I arranged to capture all motion for two of the proposed studies, Chasing Shadows and Room Rage, just in case A Brush with the Real World was unable to be finished on schedule. This way, if there were further problems with availability, the data could at least be used for pre-rendered sequences in performance. During the time the inertial suit was away I focused on modeling the characters and environments, assisted by Kim Allen and Vana Prayitno. Shadow models were built and rigged for Chasing Shadows and the first "Rupert the artist" character was built for A Brush with the Real World. When the inertial suit returned it became clear that the characters needed significant revision to work well in a real-time environment. Changes were trialed in the studio and by late September the characters were working sufficiently well to tackle the more tricky technical questions associated with using effects, triggers and behaviours. Chasing Shadows used video clips interspersed with real-time action. The transitions needed to be unnoticeable to an audience and this presented a range of technical challenges. In A Brush with the Real World there needed to be a quick easy way of swapping among colours on the brush, preferably a method that could be handled by the actor. Another major hurdle presented itself around this time; Jon Newell's availability was greatly reduced when his role was changed from motion capture specialist to content developer. As a result I needed to spend large amounts of time through the rest of the year learning more about the software until I was self-sufficient. Each change was trialed in the studio and any results provided feedback for further refinement. During this time a number of short motion capture projects came up and so I cut back on performance work to participate in these. This work ranged from performance animation, working as the performer in the suit and demonstrating motion capture to games studios and human movement related organisations. Further work arose in software demonstration (GamesConnect Conference, Brisbane), running motion 88 capture sessions, edit motion data for animation (Bippo the Hippo) and run training programs. The work give me a more comprehensive understanding of all facets of motion capture, from both the perspective of the actor and that of the animator, and a clearer picture of how it was deployed commercially. By February 2007 A Brush with the Real World, while still needing refinement, was working in principle. By April movie clips for Chasing Shadows had been created and arranged in a trigger tree so they could be activated from the keyboard. By July the environment for Private Eyes was built and several effects-based scenarios had been created as proof of concepts for dance. There was plenty of scope for further refinement, visually and performatively, but all investigations were at a stage where they could be demonstrated to be technically successful. Several upgrades in the software have since reduced these problems further. The real-time performance animation produced on screen was of sufficient quality for at least proof of concept performances and, with Animazoo recently receiving extensive development funding, the software was expected to undergo further improvement. The inertial suit was now more regularly available and this allowed me to spend more time in the studio. Late in 2007 and through to March 2008 a basic gestural interface was created for A Brush with the Real World to control the paint colours and the thickness of the paintflow. Rupert was given a voice using the speech recognition engine in MotionBuilder that recognises phonemes and triggers appropriate blendshapes (shapes of the face representing the corresponding phonemes). This would allow for greater dramatic flexibility with communication between Rupert and audience. Facial expressions and eye movement were also added. With the help of dancers Bridget Fiske and Joe Lau, along with choreographer Rosetta Cook, real-time dance effects were trialed in the studio. These trials were not so much focused on implementing creative ideas as exploring the technical attributes of the software that were likely to be effective as tools for creating visuals. In essence, they looked at what was performatively possible and what was visually and theatrically appealing but the intention was not to produce finished performance pieces. 89 The movie clips for Chasing Shadows were cleaned up, edited and arranged in a trigger tree. The motion sequences for Private Eyes were also finalised and other characters added. Each of these investigations contributed to a broader understanding of how motion capture could be used at live events. Scenery and characters could be changed at an instant; interaction with the screen character could extend beyond the interaction that games have made familiar. In April 2008 more visual possibilities for dance were explored, including the use of video projection and camera animation. As video could be projected on specific objects in the scene or assigned as a texture to those objects in a number of different ways, it was not difficult to create many types of video effects created using real-time 2D. However, the 3D effect was navigable so that the created effect could be viewed dynamically from any perspective. Through 2008 the focus was to incorporate operating methods that gave the performer as much control over the environments as possible. The more control the performer had the more spontaneously they were able to interact with the environment. Control methods developed earlier in the research, while giving the actor control, required too much thought and dexterity, compromising the purpose of using them in the first place. So time was spent developing more intuitive controls: those that more closely mimicked real-life actions. 90 5 Evaluation There is no particular mystery in animation...it's really very simple, and like anything that is simple, it is about the hardest thing in the world to do. (Tytla cited by Graham, 1937: n.p.) 5.1 Inertial motion capture The aim when commencing the research was to investigate a set of public event ideas that incorporated projection. Motion capture, at the time, was seen as a means to create some of these performances. As I began to work with motion capture it became clear that, before it could be an effective and familiar tool for live performance, a lot more needed to be uncovered as to how it could work in live performance. There were questions about its robustness and the quality of the real-time data produced. The performative aims were speculative, as there were few precedents that provided insight into whether the aims were technically achievable or qualitatively acceptable. As a result the research first needed to explore whether inertial motion capture was a useful tool for live events and, if so, in what ways it could be used. So the research focused on examples that explored broad ranging techniques applicable to live arts rather than the creative outcomes. Thus, in the context of the research, the cases are archetypal more than artistic. They answer questions of process more than product. The quality of captured data depends on the real-time performance and this is partly dependent on the performer's awareness of the inertial suit's movement parameters. While the inertial motion capture is able to capture non-vigorous movement with acceptable accuracy, it has limitations. Jumping, or in fact any movement where both feet leave the ground, is predictive and often will not be portrayed accurately onscreen. Highly acrobatic or high impact activities are generally not suitable. This might seem a major drawback in, say, action games, but it is surprising how many of the movements in this context are not in this category. Walking in a crouched 91 position was considered a move to avoid but we had few problems. The screen environment that the actor controlled had definite limitations, some that are intrinsic to the technology and its application, some that are likely to be overcome in the near future; and others that have recently been solved. Inertial motion capture has some inherent traits that differentiate it from the more commonly used optical motion capture. Inertial systems have active sensors that measure orientation. This orientation data is transmitted wirelessly to a computer. Optical motion capture uses an array of cameras to plot the changing locations of points in space. So inertial data is rotational whereas optical data is translational. Optical systems use visible light as a means of detection and so are subject to "line-of sight", contrast and reflectivity problems. With inertial motion capture the data has no global point of reference and this can lead to a range of translational errors. However, it is not the thrust of the research to delve into an in-depth comparison. Instead, inertial motion capture is viewed simply in terms of whether or not it is suitable for live performance. The attributes are set out in the chart below. The second chart shows some of inertial motion capture's limitations as well as some specific recurring problems that are likely to affect real-time performance. 92 Chart 1: Attributes of inertial suit Latency Operated in real-time with few frames dropped There is no significant lag even with more complex 3D environments Range Can operate over distance effectively Signal interference Is not affected by line of site problems Venue Is not subject to jitter or occlusion Ease of use Can be used in a large range of live environments Equipment Entire performance can run in real-time on standard computer equipment fitted with a graphics card capable of running two monitors. Control In the right situation the entire performance animation process, including the acting, can be operated by one person Updates The software is regularly improved and with each new version improves the real-time output Complexity Can use elaborate 3D environments, multiple characters and complex effects Chart 2: limitations and problems Foot slide The foot slides on the floor with each step Foot drop The foot drops a few centimeters to the floor with each step Jumping/displacement Jumping is highly predictive. While vertical jumping is interpolated well, jumping horizontally is not very accurate Actor file It took a number of attempts to get an accurate "actor" file Frame dropping The hardware or software cannot handle the rate of information and drops frames Relative positioning The data only tracks relative positioning. Depending on the accuracy of the calibrations, the location of the virtual character in the scene shifts with movement. Vertical tracking Tracking is based on the assumption that one foot is touching the ground at all times. When both feet leave the ground, say, while climbing stairs, the software will locate the data with one foot at y=0 Physical limitations The suit, while wireless to the receiver, has significant amounts of hardware attached. This limits the range of costuming that can be worn with it for a live performance situation This evaluation of inertial motion capture has been conducted over several years using the first inertial suit available commercially. To gain a better 93 perspective on the problems and limitations listed, it should be noted that since this study began there have been significant advances in the technology that have reduced or eliminated many of these problems and limitations. At Siggraph 2008 all systems presented, both optical and inertial, were able to apply a broad range of vigorous movements to complex 3D scenes in real-time reliably and with few discernible glitches. When trialing the cases it also became clear that, by adapting the visual environments, modifying characters, carefully selecting camera angles, and adapting the choreography, interruptions to the visual continuity of each piece were minimised. While the animation produced is visually impressive and is sufficient to create an engaging interactive real-time production, the technology is still not at a level where it can generate a reliable continuous uninterrupted feeling of co-presence with a character on screen. Small anomalies become obvious. To create a sense of realism or imbue a feeling of co-presence with a screen character in real-time the level of precision in the motion of the displayed animation needs to be high. Animation has direct needs for precision that stem from the sensitivities that viewers have in experiencing motion. For example, a viewer is likely to notice imprecision in a character's interaction with its world. A foot floating slightly above the floor or sliding a small amount, or a hand not quite reaching the doorknob, are tiny imprecisions yet can completely destroy the illusions of realism. (Gleicher and Ferrier, 2002: 1) The primary aim of this practice-led case study, however, is not to create an illusion of realism as this would be an environment too complex to run in real time and would require more resources and know-how than a study of this nature is likely to attract. The cases are not designed to explore aesthetic frontiers or even move beyond the visual conventions of 3D animation. The focus is on communication between observer and screen and, as a consequence, on the many facets of conventional 3D software and how well they adapt to a real-time environment. The worlds that can be created in 3D 94 are years ahead of what can be represented in real-time and so the visual quality of the real-time environment is necessarily less important than the level of interactivity that can be generated. 5.2 The case study Many of the techniques here are well known to those who work with animation. Applying these techniques using motion capture to create real- time animation is less well known. There are a number of well-documented examples of real-time animated characters being used in a public forum; these are typically real-time applications by animation production houses rather than hybrid performances by live performance troupes. This difference in emphasis is important in determining how the subject matter is approached and how the technology is used. In live performance the "liveness" is primary and animation is an augmentation, whereas in animation the emphasis is on the rendered image; the liveness is an attribute that helps the workflow but is usually invisible in the final product. A real-time environment is not capable of reproducing the depth and sophistication of animation people have come to expect through media. These environments take many hours to render. In time the combination of what can be produced in real-time and the tools available in a 3D environment will no doubt create their own tree of aesthetic directions in live performance. The case study looks at the potential of interactivity that this technology offers. 95 5.2.1 A Brush with the Real World A Brush with the Real World was designed to look at ways in which dramatic interaction can take place between a screen character and an audience and so it incorporates several communication techniques. The virtual artist, Rupert, can communicate physically and verbally but, most of all, communicates through his paintbrush. Figure 19: Trialing the paintbrush Using particle effects as "paint" was sufficient for the study but had some limitations. Firstly, the paint was RGB colour. As a consequence, when one colour was blended with another it did not behave the same as real paint. As an additive system blending one RGB colour with another resulted in a lighter, brighter colour. Real paint is CMYK colour and blends subtractively; the resultant blend is darker and duller. As a compromise Rupert used matt colours that did not blend at all. A CMYK colour system could have been simulated but there was another hurdle: any change to the particle emitters' colours affected all its particles onscreen universally. For instance if the paint changes from red to blue all red paint already onscreen will also change to blue. Each colour required its own emitter (figure 20). The other major limitation when trying to paint using an onscreen artist is that the system 96 controlling the hand movement is not highly accurate. This is partly because the data does not represent the subtleties of hand and finger position, and because an inertial system relies on the body measurements of the actor. Being at an extremity of the body, the position of the hands are calculated from all such measurements back to the feet and there is likely to be a significant degree of error. Late in the development of A Brush with the Real World real-time speech was added to Rupert's communication repertoire. This gave Rupert another layer of communication that could add to the dramatic depth of the character and his ability to interact. Facial expressions that express feelings and emotions were also added and these could either be triggered by a third party with a keyboard or by the performer using a hand controller such as a thumb joystick. Altogether, eight expressions were created and these could be used in combination with speech so that a character could talk expressively. The characterisation was important in creating a sense of connection with an audience (co-presence), so the quirks and foibles of Rupert were explored theatrically. A set script or scenario would have detracted from the necessary improvisational quality that made this type of media unique and so loose scenarios were devised and active elements were included in the scene to steer communication between Rupert and the audience. Also, the scene and the viewing position could be switched easily at any time, opening up opportunities to further develop a more narrative and visually diverse approach. No capture device was used for the hands; they were set in a neutral stance and changes were kept to a minimum. The eyes were operated in several different ways; they could be fixed so they always faced forward, automated so they "led the way", or controlled by a joystick or mouse. The eyelids randomly blinked. Objects in the scene could be picked up and put down in real-time. Rupert could also trigger actions by entering a particular location on screen or by putting his hand on a certain virtual object. As an example, 97 active objects in the scene, such as light switches, could be "switched on" by the character. The primary study, A Brush with the Real World, was performed as an improvisational window installation at the Judith Wright Centre for Contemporary Art in Brisbane in July, 2009. The animation was rear- projected onto two large screens, one facing the street for interactions with passing street traffic, and the other set-up in the foyer for more in depth dialogue with a more stationary audience. Cameras and microphones were set up so the actor could see and hear those watching. Sound from a wireless headset worn by the actor was fed through speakers on either side of the foyer screen and also back to the computer to generate mouth articulation in Rupert so that Rupert could talk to the audience. His virtual studio included a range of active elements to engage people in participating. Rupert could turn lights on and off, change appearance, pick up objects and put down objects. He had a range of moods and spoke with articulate mouth movements. He could paint with a range of colours and could dance to create colourful 3D trails. All these elements helped to make it a complex multilevel improvisation where people were generally more engrossed in communicating than passively watching. Four scenes were projected at different times during each two-hour session: the studio, the cellar, an empty black scene as a backdrop for dancing and painting and a 2D poster. The last scene, Brushalex, is presented in the style of an Art Nouveau advertising poster with Rupert as a prominent figure. Council regulations prohibited the use of amplified sound in the street so, for the window projection, the interaction focused more on visual elements. In Brushalex Rupert could move letters in the text of the poster engaging passersby in creating new expressions. In the cellar scene Rupert wandered about in the dark with a flickering candle. He could pick up and put down the candle on several boxes scattered around the space. If he sat on the candle the back of his jacket began to smoke. It could be extinguished by repeatedly hitting it with his hand. The studio scene was the most commonly used because it was more in context with Rupert's nature and occupation, 98 providing easy conversational cues for the audience. While experimental, A Brush with the Real World accentuated that improvisation works well with animation, whether the form of improvisation is physical, visual or verbal. The performance relied on attracting the attention of passersby, holding their attention and drawing them into participating. This provided, at least in quantitative terms, an ongoing indication of the success of the installation. After the first night it was concluded that the audience for the outside screen was comprised mostly of drivers sitting in traffic and diners at the restaurants and cafes across the road; people walking on the footpath directly past the screen frequently failed to notice it. On the second night we lifted the blinds on the windows either side of the projection so people had a clear view of the actor driving the animation. This had an immediate effect as people could clearly see the connection between the actor and the character, realizing the projection was not just another patch of inner city light flickering in their peripheral vision, but something intriguing. figure 20: The two screens Gestural control of the paint worked well. The painting environment was relatively simple but sufficient to demonstrate that it worked as an improvisational tool and, based on its use with an audience, is a device well worth refining and structuring theatrically for future public performances. There was little opportunity to practice the painting before the performances but, based on our work in the studio, the articulation of the brush grew better 99 with practice. In this way the process is similar to drawing with a graphics pad or mouse in that it relies on watching the progress of the image being created onscreen, a situation where the cause and the effect are separate. The project was technically challenging and there were a number of equipment and software conflicts and failures over the three days of set-up. Fortunately, Queensland Academy of Sport and VR Solutions provided additional equipment and timely assistance enabling the performances to go ahead in a modified form. We also had excellent ongoing assistance from the administrative and technical staff of the Judith Wright Centre. Throughout the week we operated with a four person crew consisting of Rosetta Cook, co- performer and public relations, Adrian Green as technician and digital puppeteer, Miriam Hanmer as assistant stage manager, and myself. 5.2.2 Private Eyes Some aspects of 3D animation software environment are metaphorically modeled on film production; these include actors, lighting, cameras and effects. Private Eyes uses particle effects, background video, scene changes, camera switching and mood lighting in real time to test the extent to which an animated movie can be performed "live". The environment used was able to support multiple particle effects, lighting and video at the same time with acceptable real-time playback. Scenes with multiple characters were created using a blend of movie clips and real-time motion capture. Rather than overload the environment with so many dynamic elements that real-time rendering was affected, certain elements in the environment were only used in specific scenes or shots where they were a prominent feature, or when they important to the storyline, mood or effect. In this study there were many variables at play and so it was approached reflexively to get a broad overview of whether all of these factors could work together without degrading the quality of the real-time screen rendering. As would be expected there is a tradeoff between complexity of the scene and the quality of the animation; the juggling of the components of each scene to 100 achieve an optimum result is an artistic as much as a technical decision. What can be drawn from this is that these components can all be used together; the extent to which they can is a question of balance. 5.2.3 Motionics Motionics uses simple constructed environments as well as some generic assets from the application, 'MotionBuilder', to explore a range of software features including virtual cameras, triggering, video, particle effects, scene changes and relationship constraints, using more evident ways in which each can be used to connect the motions of the dancer and the mood of the dance to the projected visuals. The outcomes give an indication of how well each element performs in real-time. Figure 21: Reactive background The way a camera in the scene is used can determine the relationship between the dancer, accompanying imagery and the viewer. Using a first person view, the audience is transposed into the eyes of the dancer as they move through the virtual scene, so it becomes a mutual journey between audience and dancer. This technique was used in a night-time scene, set in a forest of blinking eyes. By animating the camera the movement of the dancer could be viewed in ways that were almost impossible to achieve by other 101 means: the camera becomes part of the choreography as the perspective can pan, zoom and follow complex paths around the motion represented onscreen. This flexibility is further enhanced by toggling between multiple cameras, each of which can have unique visual elements. A moving camera constrained to a path is used in one sequence to show the virtual dancer from continuously changing perspectives. There were several ways of using a dancer's movements to drive multiple virtual dancers. The obvious way was to use several characters, but mirror surfaces were also used to create the appearance of multiple characters. Abstracted liminal bodies were created by constraining particle emitters to the motion. Particle effects were also used to create motion trails and more interpretive visual imagery. Images were applied to particles and these could be dynamically constrained by the dancer's motions. There were several ways in which video can be technically applied in a scene, and these provided some interesting visual outcomes. Video was projected as a background element as well as directly onto a moving character. By applying it to the surface mapping of the character reflective and transparent effects were created. Figure 22: Eyes follow the dancer Figure 23: Moving through a sea of eyes Figure 24: Triggering effects FigFigure 25: Using video 102 Using relations (and expressions) constraints offer many ways of connecting motion data to the attributes of any element in a scene. This is an area that could provide a large range of interesting visual possibilities. However, it was only used in a couple of instances, one for causing a shower of sparks when the character moved through certain points in the scene and to change the colours of particles using hand gestures. These video and particle effects were used in real-time without discernible deterioration in the real-time performance. Simple relationships between the dancer and the environment were also trialled. One scene used eyes that were locked onto the dancer; wherever the dancer moved the eyes followed. Another used a projected backdrop that exaggerated the effect of travel onstage. Motionics provided an opportunity to explore the visual landscape, free from the constraints of a set scenario or a particular choreography. The program, Motion Builder, has an extensive range of tools for working in real-time and the ways these can be used to create interactive visuals can lead to a great range of creative outcomes. Figure 26: The Paint colour changes universally 103 5.2.4 Chasing Shadows Chasing Shadows experiments with interplay between live action, real-time animation and movie clips to create the dramatic illusion that the actor's shadow takes on a life of its own. This was attempted by alternating between capturing live action and triggering movie clips. To make the transitions as smooth as possible, the changeovers from real-time shadow to movie clip shadow took place either out of view or at a set place in the virtual scene. Blending movie clips with real-time motion and live action was a challenge, as the position of the actor's double onscreen had no absolute point of reference and, over time, was subject to drift. To reduce this problem the actor on stage had a set location that he kept returning to. Changeovers between clips and real-time performance animation happened either at this point or when the shadow was out of view. With more time these transitions could have been made smoother by setting the horizontal coordinates of the characters point of reference in each subsequent clip to the final horizontal hip coordinates of the character when driven by the actor. The performance was designed to minimise the audience's awareness of the motion capture equipment. The scene was set in an age of overcoats, broad hats and loose trousers, which made it easy to obscure the motio0n capture equipment worn by the actor. Also, the actor's digital double was a shadow, reducing the likelihood anomalies in the animation would be noticed. A shadow is a naturally occurring two dimensional projection of a 3D object and so long as the application of the shadow is well executed, an audience may accept the scene purely as a physical world performance, rather than be diverted by the technology driving it. Footslide was a recurring problem that was likely to occur during performance, breaking the continuity and the fantasy, so the shadow was projected low enough so that the feet were often out of frame. 104 5.3 Observations of the performer (A Brush with the Real World) You feel very light, a sense of freedom. When you move it feels easier than doing it yourself. Embodying the character is almost instant but there's also a disconnectedness, like in puppetry, where you're inside the scene but watching it at the same time. So you're in two places at once. (Rosetta Cook, 2009: n.p.) While the scene bears little resemblance to the real world, recognising the mechanical relationship to a character onscreen seems to require no adjustment process. Normally the movement of the character onscreen is a diametric opposite of the actor's movements. It is much easier working with a mirror opposite and this easily done by reversing the projected image. The 3D world onscreen is not really a 3D environment and while there is an appearance of depth there are only two dimensions; so the whole sensation is like looking into a psychedelic mirror with one eye open. This is difficult at first but, providing the character is oriented correctly in relation to the screen, it becomes quickly familiar and easy to accommodate. You feel yourself inside and see yourself in the distance, like a mirror except the mirror can move so you can see yourself from any angle. Dancers spend a lot of time in front of a mirror and to be able to see from any angle at will.. dancers don't get to do that. It's a great choreographic tool. (Rosetta Cook, 2009: n.p.) When working in a real-time environment with an audience two screens are needed, one for projecting the animation and the other for monitoring the audience, creating a split focus. All these differences can create anomalies such as walking through a wall or struggling to pick up an object in the scene. In the performances this did not appear to affect the public perception and, after a while, choosing which screen to watch became second nature. Working with a virtual prop meant working with nothing but an image of it in a 105 projected scene and so the temptation was to look at the screen instead of the relative location of the prop. Over the nights of performance it became easier to locate the object first onscreen then rely on memory to interact with it, making the process unnoticeable to the audience. To monitor the audience a camera needed to be placed as close as possible to the character view. The ideal position would be the centre of the projection screen, a problematic undertaking. The camera monitoring the foyer was placed to the side of screen and this worked well making it easy to respond to people's gestures, expressions and movements. Due to council restrictions the camera facing the street needed to be set inside looking from an adjoining window well away from the screen. From this monitor the location of people watching was difficult to determine and this hindered any gestural gameplay. Despite these drawbacks, as improvisation the animation was successful in engaging the attention of passersby often for extended periods. People were generally willing to suspend belief and immerse themselves in the interaction. By the third night the blinds covering the shopfront windows either side of street screen were opened and the actor's performance area was lit. The two screens then each had a different but complementary purpose. The foyer screen was used for improvising with the public whereas the street screen showed the same animation but as an exposé of the process. The mouth animation generated by voice generally worked very well. It fractionally lagged behind the voice but this was barely noticeable. On some occasions if the character talked during action that was processor intensive such as when complex particle effects were used the lag between voice and animation became apparent. Sometimes these conditions would also cause dropped frames. All these anomalies did not have affect the overall interaction. Once set up improvising was as fluid as it would be as a roving character. In reference to performers using motion tracking to affect digital media in real- 106 time Coniglio states: ..we cannot a priori rely on the expectations and understanding the jazz audience brings to a performance. This is because: 1. the audience may not be aware there is some level of improvisation occurring, and 2. because the audience has no prior understanding of the 'instrument' with which the performer controls that manipulation. (Coniglio, 2004: 9) So people are likely to assume the animation is prerecorded unless it is made clear that it is not. Several strategies were used to make it more obvious that the animation was interactive, some that worked very well. The most obvious and effective way is to strike up a conversation. Mimicking, and setting up short scenarios that beg a response were also used effectively to initiate interplay. Once the penny dropped people either watched intriguingly or became enthusiastic players. 5.4 General findings Many animation techniques can be applied to live performance using inertial (and other forms of) motion capture. These techniques are well known to animators but little known in live performance. As a way of adapting this medium to live performance many of the actions onscreen were operated by the performer, so there was a spontaneous relationship between the performer and the visual scenario. Gestural control, voice driven blendshapes and hand held devices were used to control events onscreen. These events range from switching camera perspective to triggering animation. The intention was to minimise the presence of technology, prioritising the "liveness" of the performance over the perceived impact. The ways in which animation techniques could be used at live events are as broad as the imagination but there are some important limitations. Projection 107 is affected by light, so the places and the times for using projected visuals is restricted. Connecting a projector to a computer can substantially increase the workload of the graphics card if used as a second monitor and so it is important to trial the technical set-up well before a performance. A complex blend of models, effects, lighting and camera movement can be achieved in real-time, but there is a threshold at which the quality of the real- time animation becomes too problematical to use. For this reason performance needs to be planned around the capabilities of real-time animation. The integrity of character movement is affected by the calibration of the actor driving the motion; the fluid movement of a character is an art as much as a science. The quality of the real-time animation is affected, among other factors, by the battery voltage, magnetic field and the initialization process. Even with all care taken anomalies can adversely affect the integrity of the animation. These are listed on Chart 2. The animation environment and real-time interface in Motion Builder are very sophisticated and, though there is a steep learning curve and limited learning resources available, its capabilities are impressive. One glaring limitation with using motion capture live onstage is that a performer needs to wear the inertial suit. The range of effective movements is limited and the suit itself is a visual statement that needs to be included as an aspect of the design, covered by an appropriate costume or not visible to the audience. Any choreography that uses this type of equipment needs to be devised around these limitations. From what we found this still leaves a good range of artistic possibilities to explore. Another limitation when using any form of projected image as a part of live performance is the unobstructed space required to project the images. This problem is inherent in the nature of projection but some advances in equipment over the past couple of years may improve the way projection can be used on a stage or other public event space. Rear projection requires ample room behind the projection screen. The focal lengths of more recent projectors have significantly reduced this distance. Projectors have been developed that, as an array, project upwards from the floor in front of the 108 screen at distances of a couple of metres or less, making it feasible to have front projection and live action on stage work hand in hand in places where set up space is limited. Digital projectors have been unable to recreate true black and this affected the quality and intensity of projected images, especially in situations where background light was an issue. Expensive CRT units were used for high quality projection. However, in the past couple of years new lightweight digital projectors have been released with high refresh rates and the ability to project black areas on screen. While a mid-level graphics card is adequate for content creation and viewing animations on computer monitor, it may not be suitable for projecting in real- time. In a perfomance set-up it was important to access the menus on computer screen while projecting the animation from a projector. This required a computer and graphics card that could smoothly handle this output. 5.5 Implications for live performance Animation is a field undergoing exponential development based on massive budgets. It has a rapidly evolved heritage built on a complex ecosystem of software. To address the implications of 3D motion capture for live performance is, in effect, to look at the entire range of creative possibilities for 3D animation. Although there are computational issues that limit what can play in real-time and certain areas of content creation are not supported in a real- time environment, this is changing. The form in which images are presented is integral to the artistic and performative outcomes. In stage performance scrim, cycloramas and transparent screens have long been used to add depth, to circumvent the dimensional limitations of projection. In one visual sequence in the production Glow by Chunky Move, a video template of the dancer's movement was superimposed onto the dancer as she moved about the floor. This was combined with a spyrographic effect that continuously circumnavigated the extremities of her projected frame. Projection used in this way, as a visual 109 extension of the performer, ensures that there is no dilemma in matching the two dimensional projection with the three-dimensional dance and no split focus between the screen and the stage; it is all one. In the production, Motion-e, the movement of dancers shaped shards of soft light and gentle motion patterns on levels of scrim to create a spatiality that melded easily with the performance, removing the projections from their planar origins. The way in which projection can be used onstage is showing artistic maturity born out of years of experimentation. However, the capabilities of projection are also expanding. New research has demonstrated that smart projection can not only follow the shape of a dynamically moving object, it can adjust to the contours and colours of arbitrary surfaces (Bimber, 2008); it can intelligently blend with other projections (Majimber et al., 2008); and it can produce a stationary image from a mobile projector (Rasker, 2008) opening up further ways in which images can be presented in performance. The field of gestural interface, when used in live performance is, in itself, a trove for further development. Through movement an actor/dancer can control any conceivable action in a performance or at a public event that can be connected to a digital network. Live human motion can be used to control the lighting rig, or trigger remote actions, steer robots and puppets. Through performance animation, it is possible for the mood of an actor or dancer, using body language, to initiate changes to the lighting, sound and set, generating a sympathetic mood on stage. Also, based on the established pace of technological development the range of visual possibilities will expand rapidly. The whole mood of a production at any particular time can be in the hands of the actor or triggered by the actor's situation. However, this may only be an advantage when there is an intuitive relationship between the action of the actor and the event triggered. For instance, the brush in A Brush with the Real World worked when the brush was a beyond a certain distance from the body and within a natural range of arm movement. This is similar to the way an artist would normally use a brush and was very easy for the actor to adapt to. On the other hand the colours of the paint were controlled by the position of 110 the non painting arm, an interface that has no parallel in the physical world. This was difficult for the actor to grasp without a significant amount of practice. The prospective use of real-time performance animation at public events, particularly live performance, is related to the sum total of its unique attributes that can be adapted to performance. Conceivably, other art forms such as virtual sculpture (modeling) can also be carried out in the same way, using a digital double in combination with a gestural interface. In combination with 3D printing, this may become a common sculpting technique in the future. It is the ultimate intuitive interface paralleling the way we interact digitally with the way we interact with the world. Gestural recognition is an aligned area of research, investigating ways in which a person's body language can be digitally recognized. In a performance, these algorithms can be used to determine consequences, such as the direction a narrative takes, how an audience is responded to, or the atmosphere created onstage. With current 3D scanning techniques it is simple to recreate the appearance of a person as a 3D object. Hand scanners can scan a body and render it in minutes including the textural appearance; the accompanying software can intelligently re-sample the mesh to optimize it for animation. A character driven by motion capture does not need the kinematics normally applied to a 3D character and so such characters are simpler to create. Consequently, using available technology, it is a straightforward process to create a 3D image of oneself that can move in real-time in digitally constructed worlds. "Mirror, mirror on the wall'" will one day be reality. (The question then is: who will control the mirror?) The semantics of performance are a core part of the language of animation; this has not been through a usurping of the language but is an indication of the roots of animation. Many skills in animation, in its procedures and particularly in its performative form in motion capture, are the same as those used in live performance and so convergence between animation and performance is a rejoining of separately evolved branches of a common 111 evolution. Phillip Auslander (1997: 7) argued that the nature of performance has changed with the advent of new media. As a result performance has become more mediatized. In 2004, based on statistics of the National Endowment for the Arts (NEA), Auslander also pointed out that 78% of people in the United States never attend live theatre. This is, in part, a problem that is more uniquely American because theatre in the U.S. has never held the same cultural status that it does in Europe where the traditions of theatre are woven into the culture. Also, the low participation rate in America is not so much an indication of the demise of theatre as an indication that theatre, and live performance generally, has long left the theatre space; it has both diversified in form, adapting to a more mobile, more convergent, more visually focused world, and has evolved along with and as part of new media. The traditional skills of performance are integral in the processes for creating content in all genres of new media. As digital games become physically performative, the applications for motion capture in live performance may reach the household games console before they ever make it to a live performance. The new level of physical participation that motion capture will open up will once again blur the boundaries of what is live performance and what is new media. Already the Wii has swept the games industry with its ability to engage a person in physical play. In the near future, markerless motion capture using a set of web-cams or "active" clothing may be part of household living. This will make collaborative scenarios in which people physically participate with other people in what is unmistakably performance, an unremarkable activity of lifestyle. If performance genres such as dance and theatre continue to focus on the changing nature of human activity then they will be part of the fabric of these changes rather than left marginalized by what may seem an encroachment. (Motion capture technology may also one day spell the end of such ubiquitous items as light switches and remote controls). 112 5.6 Summary In the grand scheme this research is a tentative foot in the water. It attempts to focus on 3D animation that can run in real-time to enhance live performance but, as a relatively unexplored domain, the outcomes can only be a peep into the future that, combined with many other peeps, starts to give a broader picture of what, how, when and where motion capture will be used. A gestural interface when used in live performance is, in itself, a very promising area for further development. It allows the performer using subtleties of body movement and voice to control actions in a virtual scene, onstage or elsewhere. The whole mood of a production at any particular time can be in the hands of the actor. Whether this is desirable or practical The prospective use of real-time performance animation at public events, particularly live performance, is related to the sum total of its unique attributes that can be adapted to performance. Optical or video-based motion capture can provide more accurate data than inertial motion capture. However, at a public event in which real-time motion capture is needed, the attributes of inertial motion capture, including ease of setup, ease of use, cost and adaptability, and quality as a real-time environment (unedited) make it a better choice. Also, unlike optical motion capture, there are no line-of-site problems; motion data is sourced from the movement of the body directly rather than interpreted from light emitted from the body. Therefore, objects and other people can obscure a person wearing an inertial system without affecting the motion data. 5.7 The future of motion capture In the past two to three years there has been a rapid expansion in the number of universities and colleges with interactive arts labs that have motion capture studios. These labs have been used for analyzing, recording and documenting movement in live performance. For instance, the Advanced Computing Centre for the Arts and Design at Ohio State University recently recorded some of 113 Marcel Marceau's mime routines and these are now stored as 3D motion data that can be used to teach, demonstrate and reconstruct in perpetuity. This growth in motion capture studios on campuses is an indication of the growing importance of motion capture as a tool in research. Several Australian educational institutions have motion capture systems installed, from the small eight camera optical systems at QANTM and QUT to the twenty four camera system at Deakin University. These systems are used in areas such as film, games, biomechanical research, human movement studies and animation. Motion capture is increasingly used in live arts for analysing and documenting performance as well as creating visual effects for use in performance. When exploring a maturing technology in which the possibilities grow regularly many questions arise. Visual technology is already ubiquitous. Film, television and games are some of the world's largest industries. 3D technology has had a major impact in these industries but this has masked its growing contribution in other areas. A quiet revolution is taking place as 3D technology is infiltrating many other fields. Real-time visualisation is already establishing itself as a major tool for large-scale building projects, mining, defence, flight training, population dynamics, human movement, and social modeling. It has recently proved useful to differentiate between Autism and Asperger syndrome. 3D scanning and printing are revolutionising archeology, reconstructive surgery and prosthesis, and is becoming more widely used for rapid prototyping. An important feature unique to 3D environments is not only the ability to represent an object realistically but to simulate the properties of the object. This makes it possible to visually simulate complex interactions among objects enabling 3D environments to be used not only for vastly speeding up experiments that can be done in other ways, but also for conducting experiments visually in virtual macroscopic environments to simulate weather patterns, global warming, solar flares or black holes, and submicroscopic environments to model aspects of quantum mechanics or organic chemistry. It may be used to visually represent the dynamics in non-physical, hypothetical, abstract, 114 conceptual, or non existent environments. The other attribute of the 3D environment is that it can simulate interactions instantly; real-time remote motion data can be used to carry out robotic rescue operations, work in dangerous environments or manipulate tools on space probes. Many of the factors that currently restrict use of inertial real-time 3D performance animation in live performance are diminishing: The price of the technology is declining Devices that use tracking sensors are becoming more widespread The quality of the motion captured is improving The range of uses is expanding The technology is diversifying Systems are becoming easier to use Systems are becoming more robust The software is becoming more stable The software capabilities are expanding The hardware is miniaturizing Small wireless sensors with on-chip intelligence are becoming available The hardware worn by the actor is becoming less obtrusive Computers are becoming more powerful (Most of these factors also apply to optical motion capture) Motion capture technology is shifting from being an elite technology to becoming a pervasive and diverse aspect of everyday life. Already computer games consoles such as the Wii are incorporating basic motion tracking technology into movement-based games. iPods use an accelerometer to change the orientation of the screen when it is rotated. A range of pointing devices or "wands" use motion tracking as a 3D computer interface device. Motion capture in major movies has steadily increased over the past ten years, and in 2006 two of three Best Animated Feature Film nominees for Academy Awards, Happy Feet and Monster House, used motion capture (von Riedemann, 2007). In some movies the effects and atmospheres as well as a 115 character's appearance and movement have become so realistic the animation is almost indistinguishable from live video action. The range of live performance situations in which motion capture can be used is limited by economies of scale. It is also limited by the quality of the animation that can be produced live. However, with new developments, these hurdles are disappearing. Several breakthroughs have been made in the last two years in motion capture. Companies such as Mova ('volumetric capture'), Organic Motion has successfully produced a markerless motion capture system which can capture a person's movements accurately without the use of specialized clothing or markers and this can drive a character with reasonable accuracy in real-time. This has tremendous implications for the way we can interface with the world around us. It could lead to more ubiquitous use of gestural control as a means of interacting with a whole range of electrical and electronic devices: a preemptive living style where things go on and off because of what a person is about to do rather than because a button was pressed. Intelligent software can be used to learn people's habits and act according to their needs. This sounds like technological submission but it has practical application for, say, intelligent energy saving, impact studies, taking care of mundane repetitive tasks or catering for people with disabilities. Large sampling of complex movement data for, say, better town planning and more ergonomic design as well as health and sports research, becomes practical. It also has implications for performance and entertainment: animation has moved from a linear form (television and cinema) to an interactive form (as in computer games) and, through markerless motion capture, it can move to a participatory form. With markerless cameras set up in a public space anyone can join an animation as a performer. Price is another hurdle that is shrinking. The Optitrack Arena is an eight camera full axis 3D motion capture system, selling in its basic form for under ten thousand dollars, perhaps heralding new opportunities for cottage animation and the spread of the technology to smaller learning institutions. 116 More importantly in the context of this research, it makes motion capture accessible to a broad range of performance companies. Dutch company, Moven (recently acquired by Xsens) released an inertial system that has solved many of the problems with inertial capture including off-ground capture and accurate foot movement. A prototype portable system was demonstrated at Siggraph 2007 that combined inertial trackers with ultrasonic emitters enabling it to be used in a large range of outdoor situations without the need to transmit data to a nearby computer (Vlasic et al., 2007). Also demonstrated was a technique that uses very inexpensive photosensing markers for accurate outdoor motion capture (Raskar et al., 2007). Measurand have improved exoskeletal (mechanical) suits by using flexible fibre-optic tape in combination with inertial sensors in place of rigid telescopic frames. Since the release of its first Gyrosuit four years ago, Animazoo's software has undergone improvements and their newest suit has the option of combining with an ultrasonic tracking system for more accurate translation. All major types of motion capture can now minimally produce reasonable quality real time animation. Along with improvements in quality and accessibility there are two distinct trends in motion capture innovation: inertial motion capture is becoming location independent and optical motion capture is becoming costume independent. In the past couple of years the term "performance capture" has become prevalent. Performance capture has come to denote a shift in emphasis from capturing motion to representing performance. By realistically approximating the motion of other elements such as hair and clothing, a more fluid and holistic representation of a performance can be created. Research at the Massachusetts Institute of Technology entitled "Articulated mesh animation from multi-view silhouettes" (Vlasic et al., 2008) presented at Siggraph 2008 showed that by plotting silhouettes and combining this information with motion capture data, not only the body movements of an actor but the movement of clothing worn by the actor could be depicted onscreen. A second paper titled, "Performance Capture from Sparse Multi-view Video" (de Aguiar et al., 2008) showed that by using algorithms a performance could be captured as surface 117 deformation, obviating the need for a skeleton and providing a detailed representation of performance. Not only are we moving towards more realistic, more manipulatable and more instantaneous animation, we are also moving towards "intelligent" animation, with characters that have complex reactions and responses. In games, for instance, behaviours and physics allow characters, objects and other elements to respond to conditions within a scene and this extends the user experience well beyond direct responses from triggered clips. This is an intense area of research that is rapidly evolving and includes aspects such as behaviour planning techniques (Lau and Keffner, 2005), learned character behaviour (Park et al., 2005) and imitation of another character's behaviour (Blumberg et al. 2005). In the near future these and other developments are likely to lead to a rapid evolution in the ways in which independent decision- making by screen characters can be integrated into real-time scenarios in live performance. The locations and the contexts in which real-time motion capture can be used are also expanding. The use of projection at public events has grown and evolved. Large-scale digital screens are becoming more commonly used at major public events and are often installed at major sporting venues and exhibition centres. Sydney based Electronic Canvas use large format projectors and proprietary software to create huge visual spectacles on the sides of buildings. Curved and hemispherical projection, techniques for projecting onto complex surfaces, algorithms for blending multiple projections, 3D projection, and mobile projection, along with systems for integrating multiple image sources are expanding the ways in which images can be displayed. The drive to deliver more, better and bigger spectacles is creating opportunities both in and outside the driving force of mass entertainment/infotainment that is open to exploration by live arts. Much of the infrastructure and demand is already there. Art galleries, performance complexes, libraries, museums, convention centres, theatres, tertiary institutions and festival venues all, to a growing extent, use visual display technologies and are friendly environments for the use of new technologies in 118 live arts. The creative possibilities for the use of motion capture in these environments are extensive because motion capture raises the bar on the level of interactivity that can be achieved. There is a growing interest in the use of real-time virtual hosts such as mascots and animated presenters at corporate events and festivals. In the future there may be caricaure doubles of Ritchie Benaud and Wasim Akram commentating the cricket live onscreen at the Multan Stadium, cartoons that kids can participate in at children's festivals, digital paintings that react to the observer, actors controlling the mood onstage or controlling physical events offstage, a dancer in London controlling a fleet of dancing forklift trucks in Tokyo, a conductor controlling a virtual orchestra or a virtual conductor controlling a real orchestra at the Sydney Opera House. Projections can follow and deform to the shape of a moving changing subject; objects can move, lights can go on and off, and effects can be triggered with the gestures of the hand. The use of hand gestures as an interface is depicted in the movie, Minority Report, in which Chief John Anderton, played by Tom Cruise, manipulates a complex array of screen images using his hands. This, while fictional, was based on the work of John UnderKoffler, technical advisor for the film. 119 6 Conclusion The opportunity to learn and use motion capture equipment as a tool in research has provided an insight, not just in the techniques investigated but into a much broader range of possibilities that came to light as the parameters of the software became more familiar. Synaesthetic connections, gestural control, the use of CG and particle effects as real-time visuals, and real-time voice animation are all areas that can be used easily in a real-time 3D environment. The artistic outcomes that can extend from these elements provide a glimpse into the future; a vision of what will be possible once real- time technology has caught up with current 3D rendering capabilities. Through the exploration of motion capture it became clear that multi-scene, multi-person narratives could be enacted that can combine live action with real-time screen representation. This approach can also apply to narrative dance in which the dancers onstage are enacting the scene onscreen. Multiple characters can be used in a projected scene either by operating several motion capture suits, an expensive option, by using a scenario where no two characters are on stage at the same time, or by the mixing of real-time animation with triggered motion clips. With dance it is a simple operation for a troupe of virtual dancers onscreen to be driven by the movements of one dancer onstage. Dance is clearly a field in which the use of real-time motion capture is likely to become more common. Real-time 2D effects are already used to great effect in dance. This is evident in the work of interactive software creator, Frieder Weiss, whose highly visual motion tracking effects have been used in Mordake, Glow, Onskebronn and Mortal Engine. The huge range of visual tools available in 3D software along with the dimensionality of 3D and the responsiveness of real-time rendering offer an environment that will, in time, greatly extend the way in which visuals are used. 120 Live human motion can be used to control the lighting rig, or trigger local and remote actions or control puppets. Through performance animation it is possible for the mood of an actor or dancer, using body language, to initiate changes to the lighting, sound and set, generating a sympathetic mood on stage. The technology is still relatively new and there are some issues with the quality of the real-time animation produced and limitations on the types of movement that can be captured. However, it is likely, based on the pace of technological development, that these limitations will be overcome in the short term and, with a migration of existing techniques in 3D animation to real-time performance animation environment, the range of visual possibilities will expand rapidly. The primary investigation A Brush with the Real World focused on the use of a real-time animation environment to interactively communicate with an audience. It was also an example of the use of improvisation in animation, using paralanguage as a means of two-way communication between the onscreen character and the audience. Gestural controls can be applied in other situations where it is desirable for a person, either on or offstage, to trigger changes onstage or onscreen, through their movements, their gestures and their interaction with virtual objects. Dance, with its emphasis on the controlled motion of the body, may find a gestural interface especially useful. Chasing Shadows was successful as an experiment but the quality of the animation is still short of being able to create a seamless interaction between live and computer generated performance in this form on stage. The ability to swap between movie clips and live action was established but this transition needs to be explored further to work seamlessly. Private Eyes is an investigation into digital storytelling. The misty streetscape, using multiple particle effects, challenged the real-time capabilities of the software and hardware. Visually, there needed to be a balance between smooth animation and the atmosphere created. However, it demonstrated that the parameters for creating a visual mood that augments a live event are potentially diverse, particularly when taking into account the huge library of CG effects available in 121 a 3D environment. Dramatically, the onscreen scene can depict many aspects of the actor onstage, from scenes of reminiscence, thoughts and schemes, to facets of the alter ego. Other investigations under the title Motionics have been a practice-led search for ways in which 3D motion capture can augment dance that are unique, and distinct from the mature technology of real-time video and 2D effects used commonly as visual augmentation to modern dance. The ability to dynamically change camera perspective was important as it allowed for first person and third person perspectives as well as animated camera tracking. Motion capture was very useful in creating reactive elements such as scenery that responds to movement onstage, actions triggered by location and the eyes that follow the actor in Private Eyes. Other useful elements explored in the case study were the use of video within the projected scene, as well as particle trails that accentuate movement over time Through this limited study a range of tools and techniques were explored, including a visualisation from a dancer's view, movement of virtual objects generated from the dancer's movements, creating motion trails onscreen and using avatars to augment the theme onstage. The findings from the creative practice provide the basis for further exploration and application of performance animation as a part of live performance. While the inertial suit was a breakthrough technology at the commencement of the project, in the past year or so several other suits have been produced that have solved some of its earlier problems. Each investigation provided some affirmation and creative direction that, in total, demonstrate that motion capture has a promising future in live performance and at public events. In the not-too- distant future there are likely to be highly portable lightweight systems that are unrestrictive and accurate. Based on current trends these systems will gradually reduce in cost as the technology matures and the demand moves outside niche markets into more ubiquitous uses, paving the way for its greater use in live performance. The impact of motion capture on performance, however, will not so much open up new ways for performances 122 to be presented, as open up new ways for the public to perform. So what directions will performance take in its relationship with real-time animation technologies? That is for performance to explore and for audiences to judge. 123 8 References 2007 Higher Education Research Data Collection, Innovation and Research Systems Group: Department of Education Science and Training, <http://www.dest.gov.au/Search.htm?query=research%20publications> . Accessed 8/7/08 U.S. Bureau of Labor Statistics Office of Occupational Statistics and Employment: May 2006 National Occupational Employment and Wage Estimates, 2006, 4/2/2008, <http://www.bls.gov/oes/2006/may/oes_nat.htm#b00-0000>. Accessed 4/2/08 Auslander, P 2002, 'Live From Cyberspace or, I was sitting at my computer this guy appeared he thought I was a bot', Performing Arts Journal, vol. 24, no. 1, pp. 16-21. ---- 1999, Liveness: Performance in a Mediatized Culture, Routledge, New York. ---- 'Looking at Records', The Drama Review, vol. 45, no. 1, p. (T169) p77. Barrett, E 2006, Creative Arts Practice, Creative Industries Method and Process as Cultural Capital, Queensland University of Technology Australia. Baudrillard, J 1988, 'Simulacra and Simulations (selection)', in Stanford University Press, pp. 166-84. Benjamin, W 1935, 'The Work of Art in the Age of Mechanical Reproduction', in H Arendt (ed.), Illuminations, New York. Berezina-Blackburn, V & Miller, B 2005, LANDING/PLACE: animation in dance performance, ACCAD, Ohio State University, <http://accad.osu.edu/~vberezin/bm/landing.html>. Accessed 6/3/08 Best, S & Kellner, D 2000, Debord, Cybersituations, and the Interactive Spectacle, <www.jstor.org/pss/3685437>. Accessed 20/5/06 Biggs, M 2004, Learning from Experience: approaches to the experiential component of practice-based research, <http://sitem.herts.ac.uk/artdes_research/tvad/mb/2004a.pdf>. Accessed 7/11/06 Bimber, O 2008, 'Visually augmenting the real world with projectors', paper presented to International Conference on Computer Graphics and Interactive Techniques ACM SIGGRAPH 2008, Los Angeles. 124 Birringer, J 1999, 'Contemporary Performance/Technology', Theatre Journal, vol. 51, no. 4, pp. 361-81. Bock, PK 1970, Culture Shock. A Reader in Modern Cultural Anthropology, Alfred A.Knopf, New York. Brown, B, Gough, P & Roddis, J 2004, Types of Research in the Creative Arts and Design, University of Brighton, Sussex, UK. Calleson, J 2001, 'Virtual Puppets in Performance', paper presented to Proceedings from Marionette: Metaphysics, Mechanics, Modernity. International Symposium, 1/4/01. Carlson, M 2004, 'What is performance', in H Bial (ed.), The Performance Studies Reader, Routledge. Carlson, W 2007, A Critical History of Computer Graphics and Animation, <http://design.osu.edu/carlson/history/lesson12.html >. Accessed 15/7/08 Carver, G & Beardon, C 2004, New visions in performance the impact of digital technologies, Innovations in art and design, Swets & Zeitlinger Publishers, Lisse; Exton, PA. Clark, D 1998, Case Studies, <http://www.nwlink.com/~Donclark/perform/casesty.html>. Accessed 17/9/08 Cohen, D & Nighy, B 2006, 'New techniques make visual effects more actor- friendly: Actors' chops meet techno shops', Variety, 11 December 2006. Coniglio, M 2004, 'The importance of being interactive', in G Carver & C Beardon (eds), New Visions in Performance. The Impact of Digital Technologies, Swetz & Zeitlinger B.V., Lisse, Netherlands, pp. 5-12. Cook, R 2009, interview with the author (spoken, unpublished), 8 August 2009 Daniels, D 2002, 'Strategies of Interactivity', transl. T Morrison <http://www.medienkunstnetz.de/source-text/65/>. Accessed 16/7/05 de Aguiar, E, Stoll, C, Theobalt, C, Ahmed, N, Seidel, H-P & Thrun, S 2008, 'Performance Capture from Sparse Multi-view Video', paper presented to International Conference on Computer Graphics and Interactive Techniques archive ACM SIGGRAPH 2008, Los Angeles. Debord, G 1967, Society of the Spectacle, Black & Red, 125 <http://www.marxists.org/reference/archive/debord/society.htm accessed Accessed 26/3/05 Delahunta, S 2002, 'Virtual Reality and Performance', PAJ: A Journal of Performing Art, vol. 24, no. 1, pp. 105-14. Dixon, S 2007, Digital Performance: a history of new media in theatre, dance, performance art, and installation, MIT Press, Cambridge, Mass. Druckrey, T 1996, Electronic Culture: Technology and Visual Representation, Aperture Press, New York. Einstein, A 1944, Remarks on Bertrand Russell's Theory of Knowledge, <http://evans-experientialism.freewebspace.com/einstein_russell.htm>. Accessed 11/11/06 Felluga, D 2003, Modules on Butler: On Perform, < <http://www.purdue.edu/guidetotheory/genderandsex/modules/butlerpe rformativity.html>. >. Accessed 12/5/08 Fischer-Lichte, E 2008, The Transformative Power of Performance: A new aesthetics, Routledge, London, New York. Garau, M, Slater, M, Pertaub, DP & Razzaque, S 2005, 'The Responses of People to Virtual Humans in an Immersive Virtual Environment', Presence: Teleoperators and Virtual Environments, vol. 14, no. 1, pp. 104-16. Gerring, J 2007, Case Study Research: Principles and Practices, Cambridge University Press, Cambridge, UK. Giannachi, G 2004, Virtual Theatres: an introduction, Routledge, London, New York. Gleicher, M & Ferrier, N 2002, Evaluating Video-Based Motion Capture, <http://www.cs.wisc.edu/graphics/Papers/Gleicher/Mocap/videomocap. pdf>. Accessed 7/2/06 Goffman, E 2004, 'Performances: Belief in the part one is playing', in H Bial (ed.), The Performance Studies Reader, Routledge. Grady, S 2003, Virtual Reality: simulating and Enhancing the World of Computers, New Edition, Facts on File Inc., New York. Graham, D 1937, transcripts of action analysis class with Bill Tytla, Walt Disney Studios, <http://www.soe.ucsc.edu/classes/cmps160/Spring05/p35- lasseter.pdf>. Accessed 17/6/06 126 Grigg, L, Johnston, R & Milsom, N 2003, Emerging Issues for Cross- Disciplinary Research, Higher Education Group, DEST No. 7044.HERC03A. Hamilton-Grant, I 2001, 'Postmodernism and Science and Technology', in S Sim (ed.), Routledge Guide to Postmodernism, Routledge, New York, London. Harris, J & Walsh, B 2004, 'Sorcerers' Apprentice: reactive digital forms of body and cloth in performance', in GaB Carver, C (ed.), New Visions in Performance. The Impact of Digital Technologies, Swets & Zeitlinger B.V., Lisse, The Netherlands. Haseman, B 2006, 'A Manifesto for Performative Research', Media International Australia incorporating Culture and Policy, theme issue "Practice-led Research", February (118), pp. 98-106. Heeter, C 'Reflections on Real Presence by a Virtual person', Presence, vol. 12, no. 4, pp. 335-45. Horkheimer, M & O'Connell, MJ 1982, Critical Theory: Selected Essays, Continuum International Publishing Group. Hurstfield, T 2008, Postmodern Performance Opposing the Aesthetic (the Beautiful) <http://moderntheatre.suite101.com/article.cfm/postmodern_performan ce>. Accessed 27/8/2008 Jagger, G 2008, Judith Butler: Sexual politics, social change and the power of the performative, Routledge, London, New York. Kellner, D 2004, Media Culture and the Triumph of the Spectacle, <http://www.gseis.ucla.edu/faculty/kellner/ >. Accessed 17/5/05 Kemp, D 1999, New Knowledge, New Opportunities: A Discussion Paper on Higher Education Research and Research Training, Commonwealth of Australia, DETYA No. 6376HERC99A, <http://www.dest.gov.au/archive/highered/otherpub/greenpaper/fullpap er.pdf>. Accessed 18/5/08 Kent, C 1996, 'Spectacular History as an Ocular Discipline', Wide Angle, vol. 18, no. 3, pp. 1-12. Kershaw, B 2003, 'Curiosity or Contempt: On Spectacle, the Human and Activism', Theatre Journal, vol. 55, pp. 591-611. 127 Kirshenblatt-Gimblett, B 2004, 'Performance Studies Reader, Second Edition', in H Bial (ed.), Routledge, New York. Kramer, H 1999, Conversations on art and Performance. , John Hopkins University Press, Baltimore and London. Krueger, M 1989, Video Place, <http://au.youtube.com/watch?v=A6ZYsX_dxzs>. Accessed 3/3/07 Krueger, M 1991, Artificial Reality II, Addison-Wesley Publishing Company, Inc., Reading. Krueger, M & Turner, J 2002, Myron Krueger Live, 23/1/2002 edn, a104, Arthur and Marilouise Kroker, 13/2/08, <http://www.ctheory.net/printer.aspx?id=328>. Accessed 11/11/07 Lau, M & Kuffner, JJ 2005, Behavior Planning for Character Animation, <http://graphics.cs.cmu.edu/projects/behavior_planning/behavior_plan ning_sca05.pdf>. Accessed 7/9/08 Lindsay, A 2000, 'Thinking Inside the box: Kaydera's Filmbox2.0', 3D Magazine, no. April 2000, pp. 45-7. Liverman, M 2004, The Animator's Motion Capture Guide: Organizing, Managing and Editing, Charles River Media, Hingham Mass. Lombard, M, Ditton, T., 1998, 'At the Heart of It All: the Concept of presence', Journal of Computer Mediated Communication, vol. 3, no. 2. Lyotard, J-F 1979, The Postmodern Condition: a Report on Knowledge, Manchester University Press, Manchester. MacMurray, J 1957, The Self as Agent, Handbook of Action Research, Faber and Faber, London. Mafe, D & Brown, A 2006, Emergent Matters: Reflections on Collaborative Practice-led Research, QUT Brisbane, Australia. , <http://eprints.qut.edu.au >. Accessed 25/5/07 Majumder, A, Bimber, O, Raskar, R & Lensch, HPA 2008, 'Projectors for graphics', paper presented to International Conference on Computer Graphics and Interactive Techniques archive ACM SIGGRAPH 2008, Los Angeles. Mamoca, <http://www.mamoca.com/>. Accessed 18/7/07 Maslow, A 1968, The farther reaches of human nature, Viking, New York. 128 Menache, A 1999, Understanding Motion Capture for Computer Animation and Video Games, Morgan Kaufmenn,, San Diego, CAL.; London. Mori, M 1970, 'The Uncanny Valley', Energy, vol. 7, no. 4, pp. 33-5. Mova, <http://www.mova.com>. Accessed 20/7/07 Organic Motion, <www.organicmotion.com>. Accessed 18/7/07 Orliaguet, J-M 2001, Virtual Reality and Peircian Phenomenology, Chalmers Medialab, Chalmers University of Technology, <http://www.ckk.chalmers.se/ckk/pdf/hci2001-JM_Orliaguet.pd>. Accessed 5/4/05 Peirce, CS 1978, 'How to Make Our Ideas Clear', Popular Science Monthly, no. 12, pp. 26-302. Peterson, KY 2004, 'The Emergence of Hyperreality in Performance ', in GB Carver, C (ed.), New Visions in performance: the Impact of the Digital Technologies, Swets and Zeitlinger B.V, Lisse, The Netherlands. Phelan, P 1993, Unmarked: The Politics of Performance, Routledge, New York. Rasker, R 2008, 'Mobile projectors and optical communication', paper presented to International Conference on Computer Graphics and Interactive Techniques archive ACM SIGGRAPH 2008, Los Angeles. Reeve, C 2000, 'Presence in Virtual Theatre', Presence: Teleoperators and Virtual Environments, vol. 9, no. 2, pp. 209-13. Rushkoff, D 2008, The Merchants of Cool, <http://www.pbs.org/wgbh/pages/frontline/shows/cool/interviews/rushk off.html>. Accessed 5/10/08 Saltz, DZ 2001, 'Live Media: Interactive Technology and Theatre', Theatre Topics, vol. 11, no. 2, pp. 107-30. Schechner, R 2002, Performance Studies: an introduction, Routledge, London and NY. ---- 1994, Performance Theory, New York and London, Routledge. Scrivener, S 2000, Reflection in and on action and practice in creative- production doctoral projects in art and design, ISSN 1466-4917, <http://sitem.herts.ac.uk/artdes_research/papers/wpades/vol1/scrivene r2.html>. Accessed 19/3/07 129 Slater, M 2003, A Note on Presence Terminology, <http://www.cs.ucl.ac.uk/research/vr/Projects/Presencia/ConsortiumPu blications/ucl_cs_papers/presence-terminology.htm>. Accessed 21/4/06 Smart, J 2004, 'The technology of the Real: Wayne McGregor's Nemisis', in GB Carver, C (ed.), New Visions in Performance. The Impact of Digital Technologies, Swets & Zeitlinger Lisse, The Netherlands. Stake, RE 1995, The Art of Case Study Research, Sage, Thousand Oaks, California. Steels, L 1991, Towards a theory of emergent functionality, MIT Press, Cambridge, MA, USA. Sturman, D 1994, 'A Brief History of Motion Capture for Computer Character Animation', paper presented to Siggraph 1994, Boston MA. Styan, JL Modern Drama in Theory and Practice, Cambridge University Press, Cambridge UK. Tellis, W 1997, 'Application of a Case Study Methodology', The Qualitative Report, vol. 3, no. 3. Thacker, E 2000, 'The Science Fiction of Technoscience: The Politics of Simulation and a Challenge for New Media Art', Leonardo, vol. 34, no. 2, pp. 156-8. Thompson Klein, J 2000, 'A Conceptual Vocabulary of Interdisciplinary Science', in N Stehr & P Weingart (eds), Practising Interdisciplinarity, University of Toronto Press, Toronto, Canada. Turpin, APT, Aylward, MD, Garrett-Jones, DS, Speak, MG, Grigg, DL & Johnston, PR 1999, University and Industry Research Partnerships in Australia, Commonwealth of Australia, <http://www.dest.gov.au/archive/highered/respubs/uni_partner/execsu m.htm>. Accessed12/5/08 Vlasic, D, Adelsberger, R, Vannucci, G, Barnwell, J, Gross, M, Matusik, W & Popović, J 2007, 'Practical motion capture in everyday surroundings', paper presented to International Conference on Computer Graphics and Interactive Techniques archive ACM SIGGRAPH 2007 papers, Los Angeles. Vlasic, D, Baran, I, Matusiky, W & Popović, J 2008, 'Articulated Mesh Animation from Multi-view Silhouettes', paper presented to International Conference on Computer Graphics and Interactive 130 Techniques archive ACM SIGGRAPH 2008, Los Angeles. von Riedemann, D 2007, Is Motion-Capture Animation?, November 13, Suite101.com Media Inc., Vancouver, Canada, <http://hollywood- animated-films.suite101.com/article.cfm/is_beowulf_animation_pt1>. Accessed 1/10/08 Weibel, P & Druckrey, T 2001, Net_Condition: art and global media, MIT Press, Cambridge Mass; London. Weiss, F 2008, <http://www.frieder-weiss.de/>. Accessed 8/10/08 131
Pages to are hidden for
"A Brush with the Real World the future of inertial motion capture in "Please download to view full document