virtual reality sound Asbjoern Andersen


Alex Riviere has worked with all the major VR platforms - PSVR, Gear VR, Oculus Rift, Google Daydream & HTC Vive - as Audio Director on VR titles EVE Gunjack, and Gunjack 2: End of Shift. And below, he shares his insights on creating audio for VR:
Written by Alex Riviere
Please share:

Lots of my fellow game audio professionals are often asking me the question about the difference working with Virtual Reality compared to flat-screen games except for Binaural positioning. Well, my short answer is that to create an alternate reality that provides a natural listening experience in (gaming) Virtual Environments, our game audio creative approach needs to be adapted, and binaural positioning plays a major role into revisiting it. The long answer is below…
 

Multi-Sensory Integration

With the lack of full-sensory usage in Virtual Reality games (only Vision, Hearing, and Touch through haptics are integrated), the human brain has the power to adapt the focus or the perception of the senses that are integrated. Don’t get me wrong, you won’t become a super-hero with sharpen sense(s) by playing VR games, but the brain can rewire itself through training and learning, to enhance the use of integrated senses, with the goal to use information at its disposal to analyze and interact with the surroundings.

Your smell and taste are not turned off while playing VR games, so those two senses are providing contradictory information between what you see and hear from the game versus what you smell and possibly taste from your physical surrounding environment. Touch (and proprioceptors – sensory receptor which receives stimuli from within the body, especially one that responds to position and movement) would send mixed information between the virtual world and the real world.

The senses that are used need to be reproduced respectfully to immerse the player mentally in the virtual world, combining well-designed and controlled simultaneous visual, auditory, and haptic cues to create a believable Virtual Reality experience. The brain allows to react when subtle sensory signals that might not seems important on their own gets trigger simultaneously, and that’s the power of multisensory integration (MSI).

“A basic tenet of multisensory integration is the ability of one sensory modality to enhance or to suppress information from another sensory modality.”

(Calvert et al., 2004)

A good example would be the “Virtual barber shop” demo released about 10 years ago on Youtube.

Most people experiencing this binaural audio demo for the first time, if mentally immersed (close your eyes while listening to not be distracted by your surroundings, and take a seat), would feel someone is literally behind them touching their scalp, talking to them, and cutting their hairs.
 

Human Evolution & Game Audio

The modern world we’re living in as well as the interactive medias audio language that have been developed for decades have been bringing modern sounds and audio cues to us, and we are constantly analyzing them without even necessarily being conscious of it. Our ears and brain can distinguish thousands of sounds at the same time, being highly informative, sounds are giving us the ability to analyze situations or events and react to it instinctively.

Sounds bring physical and spatial information about objects and environments that our brain can analyze, so we can learn about those sound sources and understand better a situation of objects and events.

In VR, hearing is the only sense able to provide full spatial information going beyond our field of view

Our auditory system provides a lot of information about the world surrounding us. In real life, vision, audition, and our sense of smell provide information that helps us to identify object, situations and navigate in our environment. But in VR, hearing is the only sense able to provide full spatial information going beyond our field of view, including elevation, 360 degrees and depth, allowing us to guide our decisions and behaviors as well as understanding our virtual surroundings.

In other terms, Audio in VR is the only medium able to make you turn your head, grabbing your attention and orientation to possible scripted events, or pre-defined directions and paths.
 


Popular on A Sound Effect right now - article continues below:

 

Latest releases:  
  • Foley Switch Play Track 124 sounds included $20

    124 recordings of various switches and buttons, flipped, pressed and clicked. This collection contains household light and lamp switches, radio buttons and a variety of other activate and click presses.

    Recorded in 2018 and 2019 with a stereo shotgun microphone.

    All sounds were recorded and edited at 24-bit / 96Khz resolution and mastered to broadcast quality, with only the best final sounds selected for this collection. Each audio file includes embedded meta-data.

    Add to cart
  • Whooshes Effective Trailer Risers Play Track 106 sounds included, 26 mins total $34.99

    Over 100 cinematic riser effects and epic buildups featuring distorted synths, eerie whispers, sci-fi glitches, hardcore guitars, dramatic reverses, drilling leads, and profoundly evolving textures. This original library is from the personal collection of trailer music composer Federico Soler Fernández (“Valerian and the City of a Thousand Planets” – “Middle Earth: Shadow of War” – “Halloween – 2018” – “The Predator 2018”)

    Add to cart
  • Animals & Creatures Botswana Play Track 49+ sounds included, 136 mins total From: $60

    Botswana Faunethic sound library is a unique collection of 49 sounds recorded through several national parks of the country (Okavango, Chobe,…). All these sounds has been recorded and produced with high quality equipment in multichannel.

    This collection offers a wide diversity of soundscapes and animals sounds such as:

    -Bush, savanna and forest soundscapes.
    -Hippopotamus grunt and vocals.
    -Elephants vocals, showering and drinking.
    -Zebras and impalas fighting.
    -Lion chasing an elephant during one night.
    -Birds and insects at different perspectives.
    -Villages

    If you want to hear more about this field recording trip, feel free to check this post.
    The multichannel version contains 40 tracks in native 4.0 and 9 tracks recorded in stereo only.

    This library provides authentic and interesting sounds, recorded with DPA, MBHO and Neumann mics powered by an Aeta 4minX.
    All Faunethic tracks includes metadata carefully edited, compatible with Soundminer, Soundly and Basehead.

  • City Life Shenzhen Play Track 53 sounds included, 266 mins total $39.99 $19.99

    Shenzhen is the first ever sound library of the Silicon Valley of China. Jzoo Sound invites you on the first ever audiological journey to the unique locations in Shenzhen that only locals know.

    This is a unique collection of sounds captured of various atmospheres at different times of the day. From energetic local farmer’s market, to howling subway tubes, to local middle school performance day, to old alleyways of a forgotten part of town.

    The library also contains many rhythmic elements, such as meat chopping, construction hammering, and ominous beeps. Not to mention the echoing Chinese announcements between two schools, and the cacophony of crying and construction in a forgotten alley. I hope this library will even meet your sound design needs, and ultimately show you a local and unique side of China.

    All recorded in 24 bit/96 kHz. Some recordings contain detail information above 20khz. The library is passionately described emotive words, and all the recordings are fully tagged with metadatas compatible with Soundminer, Basehead and Soundly.

    Gear Used: Roland R26, four channel built-in microphones (OMNI and XY), four channels merged down to a stereo file.

    Shoot me an email if you have any questions: shiheng_xu@emerson.edu

    50 %
    OFF
    Ends 1561845600
    Add to cart
  • Twisted Reality – Drones & Other Worlds, brings you 50 dark, twisted, ominous and magical drones / underscores for all your sound design needs. Perfect to create that unworldly atmosphere in your theatre show, film or game. Made from original field recordings and twisted into something quite different, the styles vary from monotone drone to full soundscape. All styles covered from, the condemned man's last walk, to the Snow Queen's, Ice Palace, and with most of the files over 3 mins, there should be plenty there to cover your scene.

    50 %
    OFF
    Ends 1561845600
    Add to cart

Need specific sound effects? Try a search below:
 

Binaural or Hard Panning Stereo?

To create a sense of presence in new medias (feeling of being physically present in the virtual scene), besides the usual narrative, tactical and strategic immersion, spatial immersion has been added to immersive design rules. Does it mean we need to position every single sound with HRTF processing then? Well, not necessarily.

In VR, you have the choice between using binaural positioning in place of stereo positioning (and we have the choice for any flat screen games as well now, which would likely open the door to new possibilities and discussion for first-person games).

Though less real, stereo positioning creates a stronger sense of directionality

Binaural positioning allows us to create a more realistic experience, as it accounts for real world physics in the calculations. But due to the long history of using only stereo positioning in games, players are accustomed to it, and using stereo positioning may offer certain advantages over binaural positioning. Though less real, stereo positioning creates a stronger sense of directionality, as the sound is hard panned, making it more obvious where the source is positioned, relative to the listener.

Depending on your game, you might want to use different positioning systems for different components.

If your purpose is to completely immerse the player in the world and create a sense of connection with it, you could use the binaural positioning to establish a sense of realness. For game-play feedbacks, you could possibly use hard-panning, as players are more familiar with it, to enable them to focus their attention on the gameplay, creating immersion with the gameplay rather than the environment. The stronger sense of directionality offered by the stereo positioning provides more obvious cues for the player, allowing them to receive stronger audio feedback.

The decision between using stereo positioning and/or binaural positioning should be a creative one, rather than a technical one, and depends on your game needs per component and situation. It’s all about experimentation and choices.

Preferably, your choices should be made prior starting the audio production of your game, as your positioning system(s) will affect your technical pipeline on the implementation front, as well as the creative approach when building assets and implementing them.
 

 

Creative Audio Language

Your creative audio approach when starting to work on a VR game is then affected by (at least) all of the above. You should ask yourself: When to use audio for multi-sensory integration purpose(s)? How human evolution (realistic approach) and game audio language (gameplay feedbacks and players expectation) impact your creative decision and technical choices and pipeline?

But then, what do you do with any extra-diegetic sources? Let’s take a concrete example with Music. From an end-user perspective, people need to be guided to understand what to think of a scene, what’s the context, what to feel, or what’s the setting. Those are some of the primarily Music purposes. It really depends of your game creative approach, so aim for what works best for your game. VR needs to be interactive, and Players actions need to feel meaningful in the game. Adaptive music needs to be think of carefully and creatively to build something that works both for the VR medium, and the game itself.

On the other hand, players have expectation of realism when playing VR game or experiences, and will notice anything done wrong on the soundscape, anything that could take them out of the experience. Contents production value, audio repetition, and the degree of direct or indirect interactivity to be built in Virtual Worlds can be extremely challenging.

You need to decide your intimate zone, the zone where you could interact with object, the zone where you need to add a lot of details in your sounds, both for positioning and dynamic layering matters.

Interactivity is a big topic of itself for audio in VR. You need to decide your intimate zone, the zone where you could interact with object, the zone where you need to add a lot of details in your sounds, both for positioning and dynamic layering matters. Audio can also be used as an input in VR, which once again reinforce the multi-sensory integration with haptic and visuals, and could possibly suggest taste or smell, if the MSI moment is totally mastered. Concrete example, let’s say you could grab a cigarette in the game, take the controller to your mouth, inhale and exhale would trigger respectively the subtle and visceral sound of a cigarette being burned (plus its visual feedback) and the visual cue of the smoke VFX along the sound of your character exhaling, or the direct sound of yourself (from the VR microphone) going through the acoustic rendering of the virtual environment.
 

To sum up

The main rule when doing audio for VR is that…there are no rules! Creative language is currently being developed by pioneers in the industry, both for the VR medium itself, and the type of games that works well for it.

Rome wasn’t built in a day, and as for all great things, VR and VR audio need time to reach its full potential. We are at the beginning of the VR in terms of technology and development, and there is no perfect recipe when working on VR audio.

Audio is not limited to what’s on screen anymore and should play an even bigger role than in flat screen games, serving all gameplay feedbacks, emotional states, engagement, storytelling, spatial immersion, multi-sensory integration, player’s position relative to the game action (and to some extend body balance).
 

A big thanks to Alex Riviere for his insights on VR audio!

 

Please share this:


 

About Alex Riviere:

Alex Riviere is a French born audio artist that have been working on many projects, from record albums (awarded one platinum and one gold record), commercials, documentaries to video-games, including the Gunjack series, Final Fantasy XIV, Call of Duty Online, Transformers: Rising, Civilization Revolution 2 Plus, etc.

A pioneer in VR Audio, Alex has built his experience developing for all major VR platforms (PSVR, Gear VR, Oculus Rift, Google Daydream, HTC Vive) as Audio Director on the VR titles EVE Gunjack, and Gunjack 2: End of Shift.


 
 
THE WORLD’S EASIEST WAY TO GET INDEPENDENT SOUND EFFECTS:
 
A Sound Effect gives you easy access to an absolutely huge sound effects catalog from a myriad of independent sound creators, all covered by one license agreement - a few highlights:
 
 
  • Mechanical Gizmo Play Track 2500+ sounds included, 244 mins total
    Rated 5.00 out of 5
    $125

    Gizmo is a mechanism library for designing mechanical actions in machinery without the engines. This library covers the gambit of steam, ronks, clicks, clanks, clunks, servos and all sorts of mechanical actions from various props.

    Unlike most libraries, this one was recorded at 192 with a stereo pair of Sanken CO100k microphones for extreme lossless pitching, so you can take something small like a typewriter, and make it sound like a massive printing machine.

    Add to cart
  • Animals & Creatures Animal Hyperrealism Vol II Play Track Over 2000 sounds included
    Rated 5.00 out of 5
    $170

    Animal Hyperrealism Vol II is a library containing sounds themed animal vocalisations, from real to designed creatures totaling more than 2000 individual sounds in 283 files.

    The sounds were partly recorded with animals trained for media production, partly recorded in zoos and wildlife centers. The asset list includes but is not limited to: amur leopards, bottlenose dolphins, californian sealions, pacific walruses, red ruffed lemurs, owls, parrots, dwarf little fruit bats, hamsters, guinea pigs and many more.

    The content has been recorded at 192KHz with a Sanken CO100K plus a Sennheiser 8050 for center image and a couple of Sennheiser MKH8040 for stereo image.
    A special section of the library features samples recorded at 384KHz. For these sounds an additional microphone was employed, specifically the CMPA by Avisoft-Bioacoustics which records up to 200 KHz. This microphone was actually used to record most of the library but the 384KHz format was preserved only where energy was found beyond 96KHz not to occupy unnecessary disk space.
    All files are delivered as stereo bounce of these for mics, though in some instances an additional couple of CO100K was added to the sides.
    The resulting ultrasonic spectrum is rich and allows for truly extreme manipulation of the content.

    Bonus: Two extra libraries included for free:
    This library also includes two additional releases from Mattia Cellotto - for free: Crunch Mode delivers 230 crunchy sounds made with a variety of vegetables, fresh bread, pizza crust and a selection of frozen goods. The Borax Experiment gets you 158 squishy, gory, slimy and gooey sounds.
    Add to cart
  • Nature & Countryside Nature Roomtones: Deserts Play Track 44+ sounds included, 116 mins total
    Rated 5.00 out of 5
    $59

    • In Nature Roomtones: Deserts, get a collection of ambiences with zero wildlife and barely audible wind. Hear the nothingness of desert plains and open sparseness of mountain valleys resonated by barely audible wind. Hear wildlife-free, natural roomtones from 44 different locations perfect for your post-apocalyptic productions or as atmospheric beds for any type of environment.

    • This library offers you an extensive collection of ambiences with sparse flora and zero fauna from winter nights in the desert. Hear the majestic sparseness of desert nights perfect for your dystopian adventure story.

    2% FOR THE ENVIRONMENT & CARBON NEUTRAL:
    • Two percent of the price of this library is donated to an environmental cause, as an “artist royalty” for the planet!
    • Carbon offset credits were purchased to offset my field recording travel for this library.

    KEY FEATURES:
    • 2-5 minutes clips from 44 different locations
    • Mountain valleys
    • Mountain ridge tops
    • Barely audible wind
    • Light wind with distant winter grass
    • Nothingness
    • Zero wildlife and insects
    • Zero water
    • Sparse foliage
    FILE LIST & METADATA:
    • View larger version or Download CSV
    MORE INFO:
    • Read 40+ testimonials for Thomas Rex Beverly Audio
    • Read my Field Recording Mastering Rules and learn more about how these recordings were mastered.
    • Browse the Library Info Master List to compare specs on all my libraries.
    • Browse the Metadata Master List to search my entire catalog.
    • MD5 and SHA 256 Checksums are included for each zip file in my catalog. Use these hashes to check the integrity of your downloaded files.
    What's the different between “nature roomtones” and “quiet nature” sound libraries? 

    Nature Roomtones: These are natural soundscapes with zero fauna, sparse flora, and sparse water. These recordings are location nonspecific because there is zero wildlife. As a result, these sounds can be used as a foundation for any natural environment you create.
    Quiet Nature: These are natural soundscapes with sparse fauna, sparse flora, and sparse water. These recordings are geographically linked to a place because of the wildlife present in the recordings.

    GEAR USED:
    • Sennheiser 8040s in ORTF
    • Sennheiser MKH 8040 and MKH 30 in MS
    • Sennheiser MKH50 and 30 in MS
    • Sound Devices 702
    • Rycote ORTF Blimp
    • Rycote MS Blimp
    Add to cart
 
Explore the full, unique collection here

Latest sound effects libraries:
 
  • Foley Switch Play Track 124 sounds included $20

    124 recordings of various switches and buttons, flipped, pressed and clicked. This collection contains household light and lamp switches, radio buttons and a variety of other activate and click presses.

    Recorded in 2018 and 2019 with a stereo shotgun microphone.

    All sounds were recorded and edited at 24-bit / 96Khz resolution and mastered to broadcast quality, with only the best final sounds selected for this collection. Each audio file includes embedded meta-data.

  • Whooshes Effective Trailer Risers Play Track 106 sounds included, 26 mins total $34.99

    Over 100 cinematic riser effects and epic buildups featuring distorted synths, eerie whispers, sci-fi glitches, hardcore guitars, dramatic reverses, drilling leads, and profoundly evolving textures. This original library is from the personal collection of trailer music composer Federico Soler Fernández (“Valerian and the City of a Thousand Planets” – “Middle Earth: Shadow of War” – “Halloween – 2018” – “The Predator 2018”)

  • Animals & Creatures Botswana Play Track 49+ sounds included, 136 mins total From: $60

    Botswana Faunethic sound library is a unique collection of 49 sounds recorded through several national parks of the country (Okavango, Chobe,…). All these sounds has been recorded and produced with high quality equipment in multichannel.

    This collection offers a wide diversity of soundscapes and animals sounds such as:

    -Bush, savanna and forest soundscapes.
    -Hippopotamus grunt and vocals.
    -Elephants vocals, showering and drinking.
    -Zebras and impalas fighting.
    -Lion chasing an elephant during one night.
    -Birds and insects at different perspectives.
    -Villages

    If you want to hear more about this field recording trip, feel free to check this post.
    The multichannel version contains 40 tracks in native 4.0 and 9 tracks recorded in stereo only.

    This library provides authentic and interesting sounds, recorded with DPA, MBHO and Neumann mics powered by an Aeta 4minX.
    All Faunethic tracks includes metadata carefully edited, compatible with Soundminer, Soundly and Basehead.

  • City Life Shenzhen Play Track 53 sounds included, 266 mins total $39.99 $19.99

    Shenzhen is the first ever sound library of the Silicon Valley of China. Jzoo Sound invites you on the first ever audiological journey to the unique locations in Shenzhen that only locals know.

    This is a unique collection of sounds captured of various atmospheres at different times of the day. From energetic local farmer’s market, to howling subway tubes, to local middle school performance day, to old alleyways of a forgotten part of town.

    The library also contains many rhythmic elements, such as meat chopping, construction hammering, and ominous beeps. Not to mention the echoing Chinese announcements between two schools, and the cacophony of crying and construction in a forgotten alley. I hope this library will even meet your sound design needs, and ultimately show you a local and unique side of China.

    All recorded in 24 bit/96 kHz. Some recordings contain detail information above 20khz. The library is passionately described emotive words, and all the recordings are fully tagged with metadatas compatible with Soundminer, Basehead and Soundly.

    Gear Used: Roland R26, four channel built-in microphones (OMNI and XY), four channels merged down to a stereo file.

    Shoot me an email if you have any questions: shiheng_xu@emerson.edu

    50 %
    OFF
    Ends 1561845600
  • Twisted Reality – Drones & Other Worlds, brings you 50 dark, twisted, ominous and magical drones / underscores for all your sound design needs. Perfect to create that unworldly atmosphere in your theatre show, film or game. Made from original field recordings and twisted into something quite different, the styles vary from monotone drone to full soundscape. All styles covered from, the condemned man's last walk, to the Snow Queen's, Ice Palace, and with most of the files over 3 mins, there should be plenty there to cover your scene.

    50 %
    OFF
    Ends 1561845600
 
FOLLOW OR SUBSCRIBE FOR THE LATEST IN FANTASTIC SOUND:
 
                              
 
GET THE MUCH-LOVED A SOUND EFFECT NEWSLETTER:
 
The A Sound Effect newsletter gets you a wealth of exclusive stories and insights
+ free sounds with every issue:
 
Subscribe here for free SFX with every issue

Leave a Reply

Your email address will not be published. Required fields are marked *

HTML tags are not allowed.