Communication with Augmented Reality in the Opera
“For most people the opera is a very obscure form of entertainment media. The language and the story aren’t as easily understandable as a musical or even a non-musical play would be.”
Semperoper Dresden at night
Opera
The general opera goer does so, because they are interested in classical music, plays and sometimes they are just fascinated with the media itself. It combines acting with musical skills in a very dramaturgical way. In some circles the opera is still seen as the media of well-cultured people and by enjoying it you could consider yourself as part of that group. However, if nothing else, the opera is a very educational part of the theater.
Augmented Reality
Augmented Reality is the expansion of our perceived reality through supplemented digital information over any of our senses. True AR adapts to the users perception instantly and intuitively, either through conscious or unconscious user behavior and actions.
Microsoft Hololens
Putting two and two together
This project explores new ways to experience the opera by creating exciting interactive experiences. With the help of augmented reality even newer patrons receive a greater understanding of the content of a play. They can enhance their experience by gaining more insights into the story, the characters and their relationships.
Persona
Our user and market research is based in the german MedienNutzerTypologie model created out of a study by ARD and ZDF. This model sorts types of people into different user types based on demographic data, educational level, hobbies, thematic interests, media usage etc..
From the results of our user and market research we developed a persona of people, who would be most likely interested in exploring an obscure type of media with new technology.
Persona slide of Chris, who represents our user group
Scenario model for a opera visit with possible intersections for our service or product
Scenario
In this scenario the different phases of a opera visit are mapped. For every phase we collected possible pain points. To compare we brainstormed the ideal opera experience with a service including and mobile and AR device.
The main questions we asked ourselves were how we could reduce or even remove pain points and where we could add new features to the experience to enhance it.
User Journey
Based on our scenario mapping we developed our idea of the two different User Journeys for our Persona without our product and with our product.
User journey diagram of a typical opera visit
User journey diagram of a opera visit using the new service
Design Iteration
First we developed a two dimensional paper prototype.
2D paper prototype of the interface
To gain a better understanding on how AR works in the real space we developed a three dimensional paper prototype as well
Test of the paper prototype in a three dimensional space
We digitized the 2D paper prototype and designed a first UI suggestion.
Digital UI suggestions of the paper prototype
Phase Insights
Successes
Imagining the different layers in a multi-dimensional space and figuring out interaction with the AR device based on intuitive gestures worked pretty well.
We could collect many new and interesting ways to integrate AR into the experience. It was exiting and kept us going.
Failures
As humans we already process a large amount of information. When integrating a digital UI into the real world it is easy to overload the UI.
The technical possibilities are not endless. Sometimes a good idea will not work for a very long time, if ever.
Our Solution
We needed to keep in mind that the UI needed to be simple and intuitive.
It's not only a step up from minimalism but also needs to take into consideration the environment in which the AR experience lives in: a dark theater, a stage, bright costumes, a lot of noise. We delved deeper into our research into what an opera entails.
Digital prototype image showing interactive UI elements
Technical Concept and Prototype
The system behind the AR device and service is assembled out of multiple devices and a central server. By centralizing information and timing those devices will stay in sync. One of the more difficult problems to solve would be how to stage and time information in a live, three dimensional play.
Choosing the Form…
We compared the current AR market and the forms out there. By doing that we could categorize the current AR devices into different form factors. It became apparent that the current evolution of Augmented Reality still heavily relies on visual content, which is why we chose AR glasses. However there is precedent for similar form factors used in the opera in the form of opera glasses, so our choice was not detrimental to the media we were working with.
… to fit the Function
A great AR experience depends on many different systems, that all need to work together. The user view and motion needs to be tracked, as well as the space the user is in. All of that information needs to be put into relation to each other and depending on the input a display shows relevant information to the user.
AR device technical sensor concept
Project Demo video
Testing on the Hololens
We tested the actual capabilities of the Hololens through Unity and in extension the sensors it uses. The focus was on figuring out how to place objects (such as set pieces) in a mapped space and interacting with them. Then we connected information to those objects that would be displayed upon user interaction by the object. Due to time limitations we did not explore further than that.
Phase Insights
Limitations
Finding a platform to prototype and test the technical concept and our assumptions with the current limited AR landscape was not easy. The more sophisticated experiences that border on augmented reality revolve around the smartphone. For us those experiences did not fit how we defined augmented reality, nor would they enable the more intuitive interaction we wanted.
Successes
Due to the available technology and expertise on the college campus we had access to mixed reality devices and courses to learn how to code for them. I took a specialized AR course to learn more about the platform we wanted to test our concept on and got into contact with more experts through that course.
Our Solution
Based on the information and skills we could gather through our college courses choosing a platform to prototype and test the technical concept and our assumptions became easier. Thankfully we had access to one of the most advanced head-mounted AR platforms, Microsofts Hololens. This AR device has most of the sensory possibilities we wanted to test as well as a stable coding platform to work with. On the downside this device is still very unwieldy, which is why the Hololens is only the solution to create a proof of concept prototype.
Photo of the inside of a theater before an opera showing
Where we would go from here
Usability
From a usability stand-point the concept is still very unpolished. Most of the concept is based on assumptions and ideas.
Additionally the concept for the service and the system surrounding the device needs to be expended to support testing.
Testing
Testing the assumptions posed in this concept is the next step. To do that there needs to be a reasonable test concept and scenario. For example different ways of interacting with the content and ways to display content could be tested in a/ b tests during rehearsals or even full performances. The chosen form factor of the device would also need to be tested and optimized extensively.
Technology
However to make this testing possible the technology needs to be improved greatly. The current possibilities to track the room is limited to very small spaces, tracking people or even human faces is not possible. Perhaps the tracking technology could be integrated into the space itself similar to what some VR platforms do.
Other Media
While this concept was created for a specific user group and for a specific type of media it could be adapted for other user groups, such as concert goers or school groups. Depending on the user group it could even display different information.
Furthermore this concept could be applied to different forms of (musical) theater and other interactive commercial entertainment experiences.