Asbjoern Andersen


Virtual reality has been a LONG time coming. I remember reading about it back in the early 90s where it was being hailed as the next big thing – and then 20+ years passed, and nothing much happened, at least in the eye of the general public. However, with devices such as the Oculus Rift nearing completion, it looks like we’re finally getting there.

With this revolutionary new approach to entertainment and immersion just around the corner, what does this mean for the way audio is created and used?

I decided to reach out to Varun Nair for some insights. He’s co-founder of Two Big Ears, an Edinburgh-based company developing audio solutions for games and virtual reality. Here is what he has to say about audio for virtual reality – and the unique possibilities and challenges it offers:

 

Hi Varun, where does virtual reality currently stand in terms of audio?

It is still early days, in terms of technology, workflow and design principles. While the art of designing sound would still be similar (as would the art of image composition, storytelling, etc), virtual reality as a medium needs exploration. In the past two years we have seen some incredible work on the visual front, with thousands of people around the world experimenting with the medium. Comparatively, there’s very little experimentation happening with audio and VR.

The technology is still catching up. Most VR users and developers use headphones, given that it is a personal experience. Binaural audio, which works great over headphones is a perfect fit for VR. Many of the long standing problems with real-time synthesised binaural audio can be overcome thanks to the head and positional tracking sensors and algorithms in VR devices.

Binaural or positional 3D audio makes a huge difference to the virtual reality experience. There was a lot of commercial work around binaural audio in the 90s but it has since retreated into academic and research spheres. Much of our work at Two Big Ears is making such technology (and many more) work really well across game engines, operating systems and hardware devices. It’s not just about having a straight up binaural panner, but also about advanced audio algorithms that can help create a more realistic experience.

From a design perspective, audio for VR is very much an open book. Many of the game design principles (especially those related to FPS games) crumble in virtual reality. I’m curious to see what can be uncovered for audio. For example, I’ve personally found the idea of depicting the scale of objects through sound very intriguing. With binaural audio, the space around the player becomes another parameter to play with. There’s lots that needs to be defined for mixing too. Creating a real-time binaural mix over headphones is very different to mixing on speakers.

How does sound for a 360 degree film work? Nobody is entirely sure! That’s the exciting part!

Outside the scope of games, there’s much work that needs to be done with VR films too. How does sound for a 360 degree film work? Nobody is entirely sure! That’s the exciting part!

 

What are some of the challenges presented by audio in virtual reality?

Designing visuals for VR needs to be treated as a different process (rather than upgrading existing content to support VR) and I’m convinced that the same applies for audio too. A lot of the techniques we take for granted, such as designing sounds for menu screens or HUD elements might not be applicable any more. Quite a few of the VR games are leaning towards user interface elements existing within the virtual space around the player. Could sound be used to draw attention to them?

Another area that has seen almost no discussion is music. How do we treat a background score? Must it be diegetic and exist within the scene? Could it just be a ‘normal’ stereo track in the background? Foley is an interesting area too. Could it be used to help achieve presence in a scene?

Technologically there are challenges too. Game audio needs a real push across the whole stack. We are working hard at it and so are our competitors =) The next few years are going to be an interesting time for audio.
 

As a sound designer for VR, how do you achieve presence and spatialization in your sound design?

For spatialisation we have to rely on technology. Binaural rendering and room modelling makes a massive difference. Although, the spatialisation effect itself can be greatly enhanced by the quality of the sounds. I don’t just mean recording quality, but the timbre and envelope of the sound. As humans, we’re better at localising sounds with higher mid and high frequency content. HDR mixing and ducking can help massively in clearing up a mix.

As for presence — I’m not entirely sure yet. Spatialisation helps, but I feel generative, procedural and dynamic content can greatly improve the experience. I’m personally interested in seeing how input devices for VR evolve. I’m always looking for ways to map sensor data to sound!
 

Battle of the virtual reality platforms
There are several large platforms in development for virtual reality. Best-known is Oculus Rift, a Kickstarter-funded project that ended up being purchased by Facebook. Their latest ‘Crescent Bay’ prototype includes headphones.

Project Morpheus is Sony’s take on virtual reality, announced in March 2014. This platform will be, at least initially, exclusively for the PlayStation 4.

Oculus VR – the company making the Oculus Rift – is also behind the Samsung VR, which offers a novel approach where a Samsung Galaxy Note 4 phone is used as the device display. Final pricing and availability has not been announced for any of the devices.

 

Outside of games, what are some of the areas where you envision VR being used – and how does audio fit into those scenarios?

There’s been a lot of talk lately about education and it is quite easy to see the impact VR can have on it. I also think it can redefine audio post-production, especially with VR movies. We would have to lean towards using game audio tools, but in many ways they can be inadequate for a structured and curated experience.

Being a long time advocator of procedural and generative audio techniques, I think there will be a strong need for those techniques too. Somebody just needs to create the tools that will help us move beyond vehicle, impact, rain and gun-shot models. It is tough work, but it needs getting done. If I had more time in a day, I’d be spending all of it on that!
 


Popular on A Sound Effect right now - article continues below:

 

Latest releases:  
  • Nature & Countryside FOREST Summer Edition Play Track 110+ sounds included $75.60 $52.92

    FOREST summer edition of Japan is nature sounds and ambiances captured in different forests of Central Japan, from the Kanto region.

    This library contains recordings of deep forest ambiances and direct close cicadas & birds and creeks, streams, and waterfalls in a Japanese forest.

    All of these nature ambiances are recorded using different microphone set-ups. Ortf stereo set-up and xy & ortf combination set up brings variations to stereo images from the locations.

    30 %
    OFF
    Ends 1538258400
    Add to cart
  • Foley Ball Bounces Play Track 59+ sounds included $20 $17

    This library is a collection of balls bouncing off different surfaces, inside and outside. We can generally find sports being played but not the clean sound of the ball used. We have included unique sounding balls such as a toddlers play ball and a Yoga ball for that really specific sound.

    Ball types included:

    Soccer ball / football • Yoga Ball • Beach Ball • Tennis Ball • Kids ball
    15 %
    OFF
    Ends 1538172000
    Add to cart
  • Horror Industrial Terror Play Track 210 sounds included $24.99 $19.99

    INDUSTRIAL TERROR is a collection of dark, distorted, rumbling, ear-piercing, evolving industrial impacts and whooshes, designed to unsettle.

    Collection consists of 108 designed sound effects and 102 source field recordings.

    Unique processing of organic field recordings makes it sound rooted in reality and fits perfectly into horror genre.
    Ready to be used on their own or as a part of a more complex sound design.

    RECORDED WITH: Sound Devices MixPre 6 + Senhheiser MKH8060 + DPA 4060
    CREATED AND EDITED WITH: Pro Tools, Live 10, Reaktor, EOS2, Discord4

    20 %
    OFF
    Ends 1538085600
    Add to cart
  • Trains Distant Trains Play Track 68 sounds included, 200 mins total $59 $49

    Distant Trains Sound Library delivers impressive multi-channel recordings of trains that pass by in a distance of at least 300 meters up to 1.5 kilometres.
    Included are pass bys of freight, commuter and passenger trains plus single locomotives.
    In order to capture the impressive walls of sound caused by passing trains an arrangement of 6 microphones was used. This setup encompasses stereo-pairs of MKH3040-MS, MKH8090-ORTF and MKH8040-WideAB. All pairs are recorded to separate files that are now available as time-aligned Reaper sets.
    Each recording session took place at nights so that it was possible to capture extremely long approaches and tails that fade out to natural and calm night ambience without any disturbance by road traffic or other unwanted noises.

    17 %
    OFF
    Ends 1538258400
    Add to cart
  • Sci-Fi Static Space Play Track 155 sounds included $28

    Static Space is a collection of 155 space or Sci-Fi sound effects, focused on continuous, seamless loops. Instead of fly-by's and accelerations, you get the sounds of space ships going at cruise speeds or idling. If you need motion, just add panning, fades and doppler as you like.

    Here are plenty of loops for designing alien environments, including engine rooms, general room tones, strange transmissions and more. Additionally, I have made sure to add plenty of variety; both in terms of moods, frequency content and tonality. Some of these are unmistakably space sound effects, while others are more ambiguous, and could be used for other high tech environments as well.

    You will find both low- and high end sweeteners, so when ever you see the words “rumble” or “low end” in a file name; watch that master fader! Furthermore, I went for a strong, almost musical tone, in some sounds, while others are much more noise-based and abrasive.

    Add to cart

Need specific sound effects? Try a search below:
 

If sound designers want to get started doing sound for VR, how do they go about it?

It is tough to design for VR without a device — so getting access to one would be a good place to start :) The audio tools exist and are constantly being improved upon. I think it would be great for sound designers to also jump into the video game development process, with game engines like Unreal Engine 4 that have a low price barrier — picking up those skills can help when communicating with the rest of the game development team too.

We definitely need more sound designers breaking new ground, failing, iterating and pushing the medium forward.

Another approach would be to work with the many VR enthusiasts around the world. There are lots of great indie projects around the web and I’m sure they would all value the contribution of a sound designer.

We’ve found that a number of the VR developers around the world care a lot about audio and they are willing to experiment with new technology and techniques.

We definitely need more sound designers breaking new ground, failing, iterating and pushing the medium forward.

 

Please share this:


 

About Varun Nair
Varun has worked on over 400 projects in films, games and advertising and music. He’s the co-founder of Two Big Ears, a company developing advanced audio solutions for games and VR. They offer 3Dception, a unique binaural audio and room modeling solution for game engines and middleware.
 
 
 
THE WORLD’S EASIEST WAY TO GET INDEPENDENT SOUND EFFECTS:
 
A Sound Effect gives you easy access to an absolutely huge sound effects catalog from a myriad of independent sound creators, all covered by one license agreement - a few highlights:
 
 
  • Weather Wind In Broadleaf Trees Play Track 42 sounds included, 66 mins total $20 $15

    Wind In Broadleaf Trees is a collection of recordings focusing on wind through broadleaf trees. Recorded in a Swedish forest in October it is representative of the most common broadleaf trees in Europe and other areas with a temperate climate.

    In this collection I wanted to capture evocative and emotional wind sounds, from the soothing and relaxing soft winds, tense moderate winds, dramatic strong winds and the transitions between the different emotions or speeds of wind.

    The collection was recorded at 24 bits 96Khz with an A-B stereo setup. It contains both longer files, with two minutes and more in length and shorter files edited to be a seamless loop. The files also come with embedded metadata.

    25 %
    OFF
    Add to cart
  • Water & Oceans New York Gorges and Waterfalls Play Track 48+ sounds included, 70 mins total From: $19

    New York Gorges and Waterfalls has the hidden sounds of waterfalls from the secluded gorges of New York State. The gorge walls shoot upward 150 meters, creating beautiful stone sanctuaries of sound. Rushing water cuts deep through smooth shale outcrops, with the lack of pebbles giving these falls a uniquely polished and frictionless character. In this collection, you’ll receive sounds ranging from murmuring 1-meter cascades to thunderous 65-meter behemoths. If you need waterfalls distinct to the Appalachians, listen to New York Gorges and Waterfalls.

    2% for the Planet:
    Two percent of the price of this library is donated to an environmental cause. I view it as an “artist royalty” for the planet!

    Key Features:

    • Waterfall sizes ranging from 0 to 1, 1 to 3, 3 to 5, 6 to 25, and over 25 meters
    • Perfectly loopable clips with approximately 1 minute and 30 seconds of audio for each waterfall
    • Close and distant perspectives

    Gear Used:
    Sennheiser MKH 50/30 MS pair
    Sound Devices 702
    Rycote AG MS Blimp
    Mastered Stereo:
    • All 48 sounds included as decoded LR stereo files
    • Specs: 4.92 GB – 192 kHz / 24-bit WAV – 48 files, 48+ sounds, Approx. 70 min

    Mastered Stereo + RAW Mid/Side:
    • All 48 sounds included as decoded LR stereo files
    • All 48 sounds included as RAW Mid/Side files
    • The RAW MS audio files are the same sounds as the mastered stereo, just in their original uncut/unmastered/undecoded form.
    • Specs: 10.56 – 192 kHz / 24-bit WAV – 96 files, 48+ sounds, Approx. 140 min

  • Materials & Texture Glacier Ice Play Track 300+ sounds included
    Rated 5.00 out of 5
    $40

    Glacier Ice is a library containing over 300 high quality sounds of ice cracking, breaking, shattering in various sizes of blocks – recorded entirely in the Italian Alps over the course of two winters.

    The library contains sounds of all dimensions, from ice cubes being dropped in a drink to a designed iceberg collapsing.

    The majority of the material was recorded at 192 KHz with a Sanken CO100K and a stereo pair of Sennheiser MKH8040, making this library greatly flexible for pitch shifting and all sorts of heavy processing.

    A small section recorded at 96KHz features sounds recorded exclusively with contact microphones placed directly on the surface of a frozen water stream.

    Bonus: Two extra libraries included for free:
    This library also includes two additional releases from Mattia Cellotto - for free: Crunch Mode delivers 230 crunchy sounds made with a variety of vegetables, fresh bread, pizza crust and a selection of frozen goods. The Borax Experiment gets you 158 squishy, gory, slimy and gooey sounds.
    Add to cart
 
Explore the full, unique collection here

Latest sound effects libraries:
 
  • Nature & Countryside FOREST Summer Edition Play Track 110+ sounds included $75.60 $52.92

    FOREST summer edition of Japan is nature sounds and ambiances captured in different forests of Central Japan, from the Kanto region.

    This library contains recordings of deep forest ambiances and direct close cicadas & birds and creeks, streams, and waterfalls in a Japanese forest.

    All of these nature ambiances are recorded using different microphone set-ups. Ortf stereo set-up and xy & ortf combination set up brings variations to stereo images from the locations.

    30 %
    OFF
    Ends 1538258400
  • Foley Ball Bounces Play Track 59+ sounds included $20 $17

    This library is a collection of balls bouncing off different surfaces, inside and outside. We can generally find sports being played but not the clean sound of the ball used. We have included unique sounding balls such as a toddlers play ball and a Yoga ball for that really specific sound.

    Ball types included:

    Soccer ball / football • Yoga Ball • Beach Ball • Tennis Ball • Kids ball
    15 %
    OFF
    Ends 1538172000
  • Horror Industrial Terror Play Track 210 sounds included $24.99 $19.99

    INDUSTRIAL TERROR is a collection of dark, distorted, rumbling, ear-piercing, evolving industrial impacts and whooshes, designed to unsettle.

    Collection consists of 108 designed sound effects and 102 source field recordings.

    Unique processing of organic field recordings makes it sound rooted in reality and fits perfectly into horror genre.
    Ready to be used on their own or as a part of a more complex sound design.

    RECORDED WITH: Sound Devices MixPre 6 + Senhheiser MKH8060 + DPA 4060
    CREATED AND EDITED WITH: Pro Tools, Live 10, Reaktor, EOS2, Discord4

    20 %
    OFF
    Ends 1538085600
  • Trains Distant Trains Play Track 68 sounds included, 200 mins total $59 $49

    Distant Trains Sound Library delivers impressive multi-channel recordings of trains that pass by in a distance of at least 300 meters up to 1.5 kilometres.
    Included are pass bys of freight, commuter and passenger trains plus single locomotives.
    In order to capture the impressive walls of sound caused by passing trains an arrangement of 6 microphones was used. This setup encompasses stereo-pairs of MKH3040-MS, MKH8090-ORTF and MKH8040-WideAB. All pairs are recorded to separate files that are now available as time-aligned Reaper sets.
    Each recording session took place at nights so that it was possible to capture extremely long approaches and tails that fade out to natural and calm night ambience without any disturbance by road traffic or other unwanted noises.

    17 %
    OFF
    Ends 1538258400
  • Sci-Fi Static Space Play Track 155 sounds included $28

    Static Space is a collection of 155 space or Sci-Fi sound effects, focused on continuous, seamless loops. Instead of fly-by's and accelerations, you get the sounds of space ships going at cruise speeds or idling. If you need motion, just add panning, fades and doppler as you like.

    Here are plenty of loops for designing alien environments, including engine rooms, general room tones, strange transmissions and more. Additionally, I have made sure to add plenty of variety; both in terms of moods, frequency content and tonality. Some of these are unmistakably space sound effects, while others are more ambiguous, and could be used for other high tech environments as well.

    You will find both low- and high end sweeteners, so when ever you see the words “rumble” or “low end” in a file name; watch that master fader! Furthermore, I went for a strong, almost musical tone, in some sounds, while others are much more noise-based and abrasive.

 
FOLLOW OR SUBSCRIBE FOR THE LATEST IN FANTASTIC SOUND:
 
                              
 
GET THE MUCH-LOVED A SOUND EFFECT NEWSLETTER:
 
The A Sound Effect newsletter gets you a wealth of exclusive stories and insights
+ free sounds with every issue:
 
Subscribe here for free SFX with every issue

2 thoughts on “Exploring New Sonic Worlds: Sound for Virtual Reality

  1. Speaking of mapping sensor data to sound, you might be interested in the work being done on haptics and haptic-tactile broadcasting, esp. for live virtual reality events where the “feeling” of the remote event can be captured, added to the audio and video VR stream and then decoded and used by the remote VR participant (using the appropriate haptic-tactile hardware such as a platform, vest, gloves, etc.) adding significant presence to the event.

Leave a Reply

Your email address will not be published. Required fields are marked *

HTML tags are not allowed.