For the first time ever, an opera and a ballet performance using 3D stereo-vision virtual sets were staged at the Hungarian State Opera in September. The interest generated by, and the response to these performances were overwhelmingly positive, with 3D technology seen as having huge potential for a world typically fond of traditions.
Andrew Quinn, one of the first TouchDesigner users, worked with the newly-formed company 3DLive Live and usedTouchDesigner to develop the 3D technology employed in this way for the first time.
Andrew is based in Milan where he teaches a TouchDesigner course as part of a masters program in Digital Environment Design at NABA. He and graduate Nima Gazestani have for some time now produced live audio-reactive visuals to accompany classical music performances. So if at first the notion of opera in 3D was a bit foreign, when we learned that Andrew and Nima were involved the concept gelled. Nima and Andrew were kind enough to speak with us about how these precedent-setting and masterfully designed productions came into being.
“The thing that motivates me in my work with TouchDesigner is the merging of sound and image, creating imagery that is sound reactive.” says Andrew. “I have always been fascinated by the counterpoint between music and image and TouchDesigner provides the toolset to explore a direct relationship between music and imagery generated in real-time.
With a background is 3D animation and compositing Andrew Quinn started using Side Effects’ PRISMS in the late 1980s and graduated to Houdini when it was released in ‘96 while working on Alex Proyas’ “Dark City”. Having worked as a professional musician and composed music using digital tools Andrew found TouchDesigner to be a familiar and natural environment in which to take his 3D expertise and knowledge of music back into performance.
Andrew, how did this project come about? The idea to produce classical opera in virtual 3D?
On this project I was working with 3DLive Live, a new company founded by Francesco Stocchino Weiss, a technology leader in classical music in Europe. The idea for the virtual set was born over a year ago and comes from Ildiko Komlosi, the soprano who sings the role of Judith in this production of Bluebeard’s Castle.
The digital sets for Bluebeard's Castle started off as an architectural project by Gianmarco Campanino, a young architect from Turin who was commissioned to create sets. I came onto the scene about 3 months before the show through a colleague of mine who is a consultant in stereography for cinema, proposing a real-time solution for the virtual sets.
Can you describe the process of producing the visual effects for the production and how you collaborated with the technical crew?
The show was produced entirely with TouchDesigner, with some 3D modelling in various 3D programs. At the start, I worked with the architect for a few weeks importing the sets into TouchDesigner from 3D Studio Max via FBX.
Then started the long process with the director and lighting director of establishing a list of around 70 cues through 8 scene changes, all marked on the score and given cue numbers that were shared with the lighting technicians.
At the performance we had the 'maestro delle luci' at our side to call the cues. The cues consisted of lighting effects, animation of elements of the sets, particle effects, L-systems, and even a flock of bats. The sound reactive was also important. We used audio analysis to displace textures that then became light projections. Audio was also used for modulating particle dynamics, even the geometry in one scene, using a vertex shader. We were getting frame-rates of between 17-60fps rendering stereo at 1024x576.
We used back projection for the performance, 2 Christie projectors at 18K ANSI lumen each, and a PC with an NVIDIA quadro5000 and a backup PC with video mixer, which thankfully we didn't need to use.
Was there a lot of trial and error or finessing involved in creating believable scenarios using these virtual sets on a real stage alongside performers and stage lighting?
At the initial stereo tests in Budapest in early July it became clear that we needed to deal with problems of scale between singers and set, especially in scenes in which the virtual set moves, also the issue of the ground plane seen from various points of view in the theatre, something that we then decided to avoid.
We started experimenting with the integration of physical lighting design with the virtual lighting. Our lighting designer Alessandro Chiodo has an excellent sense of lighting in 3D. I've rarely worked with a lighting designer in my film experience, and I found we spoke almost the same language.
How did opera fans and afficionados react to these pretty radical interpretations of classic operas?
The reaction was excellent. We played to full houses at each of the three performances, and extended applause, even 15 mins: great enthusiasm for the opera management.The dramatic soprano Ildiko Komlosi is considered one of the best in the world for the role of Judith and has performed in major opera houses in the world, including La Scala.
So the consensus was that opera with 3D visual effects works?!
The libretto was written in 1911 by Béla Balázs, a fan of early cinema and so the opera intrinsically lends itself to visual effects: Bluebeard brings his new wife to his dark damp castle and 7 doors are revealed to her that represent psychological states. Tears are seen to flow down the walls, blood is seen to flow in rooms, the music describes all the mood changes and rhythms - so it was not hard to find the appropriate visual treatment.
You can imagine how pleased Balázs would have been with a virtual set!
You must all be very pleased with the results and the reaction to these productions. Are there plans in the works for more performances or to produce new works or elaborate on what currently exists?
We filmed the performance in stereo so this will be a way of selling the production around the world. We will soon start on a ballet program with Lola Greco that will include Ravel's and music by De Falla.
We are hoping to introduce elements of real sets into future performances using mapping techniques and maybe also front projection. The collaboration between TouchDesigner and the lighting designer was one of the more rewarding experiences of the project, and for me the area to explore further.
Nima Gazestani graduated in Product Design at the Faculty of Architecture in Genoa after which he worked as a 3D modeler for various architecture studios. He then did a masters in "Digital Environment Design" at NABA, Milan where he learned to use and then to teach TouchDesigner. Nima is responsible for producing the 3D special effects for Boléro.
Nima, can you please describe to us some of the motivations and concepts driving what you do.
I love to do simple and easily understandable interactive installations and interfacing with other people working in the same field but with different skills. I LOVE sound-reactive visuals! I think it's the most beautiful thing in the world! I love mapping a building or a sculpture with a projection, I love to mix human natural gestures and technology. I love learning and being inspired by others: designers, dancers, singers, lovers, students, kids, books, movies, urbanism etc..
Can you tell us a bit about Boléro’s storyline and how you created design elements for real-time 3D use?
The ballet of the Boléro is based on the story of a man that during his life meet and fall in love with 4 different women. These women represent the different aspect of love in different part of life. Discovery, sensuality, passion, jealousy, fighting and loving are different aspects of the choreography concept.
The ballet was introduced by the opera and is directly linked also in the concept: the 3 women/dancers (that represent the past story of the man) appear in the last door/scene of the Bluebeard, as the ghosts of the past lovers.
The show starts with the night and some stars in stereoscope, a shadow of a "naked" woman waking up and starting to dance with the man, they are divided from the screen, the woman dancer is on the back of the screen and she seems to be on the stage, because of the shadow that the projectors side by side make on the screen. With the sunrise we had little orange particles that float around a red rose.
All the storyboard is designed in a surrealistic way mixed with a few realistic elements, like the rose and a body of a woman that comes out from the ground, but at the beginning it looks like an agglomerate of spiked mountains that gradually become a naked body.
The visuals are designed to be played with colours selected with the lighting designer Alessandro Chiodo. Playing cues together with the lighting is absolutely wonderful, effectively changing the entire environment nd atmosphere at the same time.
You created all the visuals for the performance in a very short amount of time. What was your process and experience?
The most interesting part of the project for me is that I made it in 5 days! I made a lot of changes during the rehearsal according with the dancers, the director Caterina Vianello and the lighting designer Alessandro Chiodo.
TouchDesigner was used for the entire project. Instead of modelling the rose and the woman’s body those were imported as FBX. I worked constantly on the project in real-time, changing light inside the project, modifying the duration of each effect, moving the elements inside the visual to have the best result and to match perfectly with the choreography.
The sound reactive elements worked great, specially on the speed and turbulence of the floating particles!
I loved to work with the dancers, with the director and with the light designer. Adding effects in realtime and showing them on the screen, they were very happy to give me suggestion and to see the result on the stage after 5 or 10 minutes!
If you structure the base of the visual then you can modify all the animation timing and adjust the effects during and between the rehearsal. This is one of the best quality that I love of this software!
Thank-you Nima and thank-you Andrew and congratulations to you both for producing this beautiful work as well as the technology that enables it. To our readers, there's an in-depth article in the cue and coming very soon hilighting TouchDesigner's performance in the Digital Environment Design masters program at NABA. We have interviews with Andrew, Nima and 2 other graduate students discussing their experience with the software illustrated with examples of their work. Very interesting!