New Realities in Audio: A Q&A with Stephan Schütze

Stephan Schütze has worked in game audio for nearly 20 years, and, based on his extensive research and experience, he's just published a book on sound for VR, AR, MR and 360 video. In this interview, he looks at the current state of spatial audio, lessons learned writing the book - and why immersive sound matters:

---

Interview by Anne-Sophie Mongeau, photos courtesy of Stephan Schütze. Images not to be reproduced without appropriate acknowledgments. Copyright: Sound Librarian, Photographer: Anna Irwin-Schutze

---

Please share:

• More

Hi Stephan, thanks for doing this interview! First, could you introduce yourself and your work?

Thank you to you and your team for the interest!
My name is Stephan Schütze and I have been a composer and sound designer in the game industry for close to 20 years and I was a professional musician before that.

I have worked in many roles during my time in the industry. As a composer I wrote the first fully orchestral game score produced in Australia, as a location recordist and SFX producer I have created sound effects libraries that are used by EA, Activision, Disney, Warner Brothers and Skywalker Sound. I have created audio content for nearly every game platform over the last 20 years.

In regards to spatial audio I have worked for some of the leading companies in new reality research and production and been fortunate enough to work with some incredible people.
 

Could you please introduce your new book on VR, AR, MR & 360 audio – what is the targeted audience and what motivated you to write it?

The book is called New Realities in Audio: A Practical Guide for VR,AR,MR and 360 Video.
That is quite a mouthful, but it was important to be very clear on which formats the book would cover because this new format of technology is quite complex and already people are wondering how it all works.

I was commissioned by CRC Press over a year ago to write the book and much of 2017 was dedicated to its creation. The book does cover some complex concepts and information, but I also tried to make sure it was something that a broad audience could benefit from. So there are useful tips and information for experienced sound designers, but there is also plenty of information that will benefit students and newcomers to spatial audio.

I also need to acknowledge that this book would not have been possible without the incredible help and support of my wife Anna Irwin-Schutze. Her skills as a professional writer and editor contributed directly to the creation of the book and often helped sort the jumbled thoughts in my mind. Writing a book on a brand new topic is a challenge because you are often forming the very first thoughts on how something is done. Anna was a tremendous help in forming some of those thoughts.
 

How did you personally get to be involved to such an extent with Spatial audio?

Like anything in the entertainment industry, the journey is often an unusual one from point A in your career to point B.

I had a relationship with the company that created FMOD a few years back and at that time the first tools to support spatial audio playback were being developed. As part of that development I was asked to rebuild the audio for the original Oculus Tuscany demo. The very original version only utilized basic 3D positioning of sounds, which is pretty much how game audio had worked for over 20 years. The new tools for spatial playback allowed me to expand on that demo and really create an immersive audio experience.

Later I was approached by George Sanger, the Audio Director at Magic Leap and asked to join their team. I spent 18 months doing research and creating content as part of the development of the Magic Leap AR device. From there I was asked by the Facebook Spatial Audio team to develop a series of video tutorials and a demo project to explore the Spatial Workstation and help users get more out of that toolset.

Along the way I created audio for a variety of VR and AR projects which gave me the opportunity to further develop my spatial audio skills and really understand what works and what doesn’t.
 

What are the main ideas and concepts explained in your book?

Spatial Audio has many elements that can help create a convincing experience to the listener. The technology is a key aspect of understanding how it all works, as such I start with some basic explanations of how humans hear and importantly how we perceive audio in the real world. The perception is absolutely critical to creating effective spatial audio.

It is how we present points of difference that can really enhance the experience

Then the book goes into how to apply this knowledge in a meaningful and practical manner. The technology is critical to allow artists to create spatial content, but it is how we utilize that technology that is key. I found that many of the old stage-craft techniques are as valid and important to creating convincing sound for new reality as they were for their original stage shows. Humans notice change, as such it is how we present points of difference that can really enhance the experience.

A key theme in the book is the weak link that humans present. Humans have very flawed hearing models in regards to locating sounds in space, so as artists we need to understand this weakness, but we can also take advantage of it to “trick” the listener into perceiving things in certain ways.
 

Can you tell us about how you see the challenges brought by spatial audio and some of your strategies to overcome them?

Audio is often too subtle or too “over the top” for our audiences. What I mean is. People will notice a giant planet size explosion and think “ooooh” but will seldom notice the small details that really convey the depth of an experience.

We need to create environments that really make the audience feel like they are wrapped up and surrounded in the sounds of the virtual world they are inside of

One of the key challenges is to convince publishers and producers that the details matter. Interestingly so many of the really successful games and films always do have outstanding audio, so the discussion should be an easy one.

What we need to do is create environments that really make the audience feel like they are wrapped up and surrounded in the sounds of the virtual world they are inside of. Allow the audience to turn their head and “find” items in the world by listening for them. Make the audio part of the experience or part of the gameplay so the user understands its importance.

Exposing our audience to subtlety and allowing them to enjoy it I think is an important part of increasing the awareness of all types of audio. For spatial audio I think it is going to be a critical part of the evolution of the new reality formats.
 

Can you share some of the experiments you have been doing while writing your book and what the results show?

There are some aspects of sound behaviour that most of us understand. Sound reduces in volume over distance and high frequencies drop off sooner than lower frequencies due to being absorbed by the environment more easily. But for spatial audio there is so much more involved that requires complex maths to calculate and in some cases there is behaviour that we don’t even understand yet. So we use the original simple model because the more complex stuff gets too hard.

Wave front curvature (the shape of the sound wave when it hits you) can convey how far away the sound was when it was generated. We often instinctively know this as a human, but how is it that we know this? How much of our perception of sound is from actual brain calculations and how much of it is context? If I play you the sound of a helicopter that is quiet what do we know about it? Is it a very quiet helicopter that is close to us, or is it a more normal helicopter that is distant.

Understanding how humans perceive and interpret sound information is, for me, as important as understanding the hard maths and physics.

With music, I am fascinated in how we arrange ensembles of musicians. Over years we have established a “good” layout for a symphony orchestra on stage, a setup that allows the various instruments to “speak” clearly to the audience. But when VR allows us to insert the audience right into the middle of 60 musicians, then how would we alter the setup to make the best of that scenario? We still have so much to learn.
 

Can you tell us what are, according to you, the greatest strengths of VR, AR, MR & 360 audio and what it can bring to the listener and the immersive experience?

An excellent VR experience with poor audio design will have most of its believability undermined

Immersion is the great strength of all these formats. Being INSIDE a world or an experience is a whole new level of depth for narrative experiences. I have played games in VR that I had previously played on a standard console and the difference is massive. When we insert someone into a world it means that we need to wrap them within the elements of that world and sound envelopes you totally. This is valuable to creating a virtual experience, but it is also critical to get right. An excellent VR experience with poor audio design will have most of its believability undermined. Most humans may not pay much direct attention to the sound of their environment, but they sense it at an instinctual level and when it is wrong, we know…somehow we always know.
 

What other reliable resources and pieces of information on spatial audio would you recommend?

That is a tough question. I think my book may be the first and possibly only full text on spatial audio production. There are lots of research papers on various aspects of spatial audio, but the trick there is to sift through them to find the bits that are relevant to you.

The Audio Engineering Society is an excellent source. Membership will allow you to access all the academic papers in their collection. Attending the various AES conferences, Game Sound Con and GDC are also very good ways to access resources and the people involved with spatial audio.
 

Can you summarise a few things to keep in mind for someone who is only getting started in tackling spatial audio?

“Point of difference” I think this is the KEY factor when creating spatial audio.

A single stereo sound just is not enough anymore

When standing in the rain, if I turn my head I can hear the rain shift around me. The rain doesn’t move, but my head does. So relative to the sound of the rain my position shifts. Also that same rain will sound very different depending on what surface it is impacting. If I am moving through a rainstorm I will hear rain on pavement, then rain on grass then rain on wood, all of these sounds will surround me, but again as I turn my head or move through the rain my position relative to the rain will change. A single stereo sound just is not enough anymore, and it is sad to see VR projects still utilizing fixed stereo to create poor ambiences.

Similar with reverb. The point of change is still more important than a mathematically accurate set of room reflections. We do not yet have the technology to accurately and completely reproduce the exact room reflections we might experience in real life. And, it often does not matter. Our audience will expect to hear a change in the quality of reverb if they walk from outside, into a large hall. In most instances as long as the reverb is generally suitable for a room of that size the audience will accept what we present them. “Point of difference”
 

Finally, can you tell us what is next for you and what to look out for in the specific area of spatial audio?

What to look out for? There are so many spatial audio tools and formats being developed and launched right now that we are kind of spoilt for choice, however not all of them are equal. So I would focus on what tools work best with your workflow on the specific platform you are developing for. Also, there are many platforms popping up. Try them out, work out which you like and WHY you like them. I am really loving VR as an environment to play games in and I think we have only just seen the very tip of the iceberg of what VR and AR can be. I think it is going to be super fun. Trust your ears!

What next for me? Whatever I can do to continue to work on fun and exciting projects in VR, AR, MR or 360 video. It can be a bit scary working in a new format as you are learning while developing, but it is also a time when we can create new content that no one has ever experienced before and that is really appealing. I am honestly open to anything that is pushing the boundaries of what we can achieve with spatial audio.
 

Please share this: