With this revolutionary new approach to entertainment and immersion just around the corner, what does this mean for the way audio is created and used?
I decided to reach out to Varun Nair for some insights. He’s co-founder of Two Big Ears, an Edinburgh-based company developing audio solutions for games and virtual reality. Here is what he has to say about audio for virtual reality – and the unique possibilities and challenges it offers:
Hi Varun, where does virtual reality currently stand in terms of audio?
It is still early days, in terms of technology, workflow and design principles. While the art of designing sound would still be similar (as would the art of image composition, storytelling, etc), virtual reality as a medium needs exploration. In the past two years we have seen some incredible work on the visual front, with thousands of people around the world experimenting with the medium. Comparatively, there’s very little experimentation happening with audio and VR.
The technology is still catching up. Most VR users and developers use headphones, given that it is a personal experience. Binaural audio, which works great over headphones is a perfect fit for VR. Many of the long standing problems with real-time synthesised binaural audio can be overcome thanks to the head and positional tracking sensors and algorithms in VR devices.
Binaural or positional 3D audio makes a huge difference to the virtual reality experience. There was a lot of commercial work around binaural audio in the 90s but it has since retreated into academic and research spheres. Much of our work at Two Big Ears is making such technology (and many more) work really well across game engines, operating systems and hardware devices. It’s not just about having a straight up binaural panner, but also about advanced audio algorithms that can help create a more realistic experience.
From a design perspective, audio for VR is very much an open book. Many of the game design principles (especially those related to FPS games) crumble in virtual reality. I’m curious to see what can be uncovered for audio. For example, I’ve personally found the idea of depicting the scale of objects through sound very intriguing. With binaural audio, the space around the player becomes another parameter to play with. There’s lots that needs to be defined for mixing too. Creating a real-time binaural mix over headphones is very different to mixing on speakers.
How does sound for a 360 degree film work? Nobody is entirely sure! That’s the exciting part!
Outside the scope of games, there’s much work that needs to be done with VR films too. How does sound for a 360 degree film work? Nobody is entirely sure! That’s the exciting part!
What are some of the challenges presented by audio in virtual reality?
Designing visuals for VR needs to be treated as a different process (rather than upgrading existing content to support VR) and I’m convinced that the same applies for audio too. A lot of the techniques we take for granted, such as designing sounds for menu screens or HUD elements might not be applicable any more. Quite a few of the VR games are leaning towards user interface elements existing within the virtual space around the player. Could sound be used to draw attention to them?
Another area that has seen almost no discussion is music. How do we treat a background score? Must it be diegetic and exist within the scene? Could it just be a ‘normal’ stereo track in the background? Foley is an interesting area too. Could it be used to help achieve presence in a scene?
Technologically there are challenges too. Game audio needs a real push across the whole stack. We are working hard at it and so are our competitors =) The next few years are going to be an interesting time for audio.
As a sound designer for VR, how do you achieve presence and spatialization in your sound design?
For spatialisation we have to rely on technology. Binaural rendering and room modelling makes a massive difference. Although, the spatialisation effect itself can be greatly enhanced by the quality of the sounds. I don’t just mean recording quality, but the timbre and envelope of the sound. As humans, we’re better at localising sounds with higher mid and high frequency content. HDR mixing and ducking can help massively in clearing up a mix.
As for presence — I’m not entirely sure yet. Spatialisation helps, but I feel generative, procedural and dynamic content can greatly improve the experience. I’m personally interested in seeing how input devices for VR evolve. I’m always looking for ways to map sensor data to sound!
Project Morpheus is Sony’s take on virtual reality, announced in March 2014. This platform will be, at least initially, exclusively for the PlayStation 4.
Oculus VR – the company making the Oculus Rift – is also behind the Samsung VR, which offers a novel approach where a Samsung Galaxy Note 4 phone is used as the device display. Final pricing and availability has not been announced for any of the devices.
Outside of games, what are some of the areas where you envision VR being used – and how does audio fit into those scenarios?
There’s been a lot of talk lately about education and it is quite easy to see the impact VR can have on it. I also think it can redefine audio post-production, especially with VR movies. We would have to lean towards using game audio tools, but in many ways they can be inadequate for a structured and curated experience.
Being a long time advocator of procedural and generative audio techniques, I think there will be a strong need for those techniques too. Somebody just needs to create the tools that will help us move beyond vehicle, impact, rain and gun-shot models. It is tough work, but it needs getting done. If I had more time in a day, I’d be spending all of it on that!
Popular on A Sound Effect right now - article continues below:
If sound designers want to get started doing sound for VR, how do they go about it?
It is tough to design for VR without a device — so getting access to one would be a good place to start :) The audio tools exist and are constantly being improved upon. I think it would be great for sound designers to also jump into the video game development process, with game engines like Unreal Engine 4 that have a low price barrier — picking up those skills can help when communicating with the rest of the game development team too.
We definitely need more sound designers breaking new ground, failing, iterating and pushing the medium forward.
Another approach would be to work with the many VR enthusiasts around the world. There are lots of great indie projects around the web and I’m sure they would all value the contribution of a sound designer.
We’ve found that a number of the VR developers around the world care a lot about audio and they are willing to experiment with new technology and techniques.
We definitely need more sound designers breaking new ground, failing, iterating and pushing the medium forward.
Please share this:
License agreement for users of Sound Examples downloaded through A SOUND EFFECT (www.asoundeffect.com) (as “Distributor”).
This end user license agreement (the “Agreement”) is entered into between you, a single user natural person (the “Licensee”), who has downloaded one or more Sound Examples through the Distributor, and the creator or creators of these Sound Examples (the “Licensor”). For multi-user licenses, please contact firstname.lastname@example.org.
This Agreement covers one or more Sound Examples downloaded by the Licensee via the Distributor.
The Licensor is the creator or creators of the Sound Examples, stated as such in the downloaded file(s) (“File”) the Licensee receives after registering with a valid email address and name.
By downloading, the Licensee accepts this EULA and agrees to be bound by the terms and conditions set out in this EULA and the EULA’s with similar terms for each Licensor in the File. Any files or material included in the File not specifically mentioned in each Licensor’s EULA is covered by the terms below. By downloading the Sound Examples, you'll also receive the A Sound Effect newsletter from time to time. You can unsubscribe from this anytime.
1. Grant of License
In consideration for the download of the Sound Examples via the Distributor, the Licensor grants the Licensee a worldwide, non-exclusive, perpetual, royalty free license to use the Sound Examples (“Sounds”) on the terms and conditions set out in this Agreement.
2. Rights Granted
The license granted in this agreement allows the Licensee to:
a. install and use the Sound Examples on one workstation at a time, although the Licensee is permitted to make and keep backup copies of the Sound Examples on other storage devices, and
b. distribute and publicly perform reproductions of the Sounds, where these are incorporated in and synchronized with other media productions, which shall mean products that contains at least one additional media element to the Sounds (music, voice, image, etc.), including but not limited to radio and television broadcasts, film, music compositions, web sites, podcasts, mobile apps, advertising, multi-media presentations, video games and similar.
The Licensee is not permitted to distribute or perform reproductions of the Sounds where these are not incorporated in and synchronized with other media productions, including but not limited to in toys, product design, greeting cards, ringtones, applications such as soundboards, hardware devices, media authoring tools etc.
To the furthest extension permitted by law, the Licensee is prohibited from adapting, modifying or repackaging any Sounds, except as permitted in Clause 2.
4. Intellectual property rights
All rights to the Sound Examples are owned by the Licensor and other than the license rights granted in this Agreement all rights in the Sounds and Sound Effect Libraries remain the property of the Licensor. The Licensee must not claim ownership or authorship of the Sounds or the Sound Examples.
The Licensee’s right to use the Sound Examples will automatically terminate in the event of any breach by the Licensee of the terms of this Agreement. In the event of termination, the Licensee shall delete or destroy all copies of the Sound Examples which the Licensee has produced.
The Licensee shall indemnify Licensor and Distributor from, and against any and all claims, demands, suits, awards, damages, suits, injuries, liabilities and all reasonable expenses, including attorney’s fees incurred by the Licensor and the Distributor with respect to any matter that arises as a result of the Licensee’s breach of this Agreement.
Licensor and/or Distributor shall not be liable for any damages or for any loss of business or business profits, business interruption, or any other direct or indirect loss resulting directly or indirectly from the use of any of Licensor’s Sounds.
To the furthest extension permitted by law, the Licensee must not assign, license, sublicense, sell or otherwise assign the Sounds to any third party, except as set out in Clause 2.B.
9. Applicable Law
This Agreement is governed by the law of Denmark without giving effect to the Uniform Law on the International Sale of Goods and the Uniform Law on the Formation of Contracts for the International Sale of Goods.