Behind the superb sound of Zelda – with Nintendo’s game audio team

Getting the story behind Zelda’s fantastic game audio has been tricky, but thanks to a recent video from GDC we now have it – directly from the sound team at Nintendo!

To recap Junya Osada’s presentation on the sound aspect of the game, he said they wanted sounds to have natural decay and echoes that were relevant to the many different spaces in the game, such as open-air spaces, complicated terrain (with structures, forests, and other obstacles that absorb or occlude sounds), and enclosed spaces. To achieve this, they needed to create rules for the sound design that dictated how sounds would act in the 3D environments in-game.

They started by examining real world acoustic characteristics. As a general rule, SPL decreases by 6 dB for every doubling of the distance from the source. Air absorption also plays a role in the attenuation of sounds at distance, and the higher the frequency, the more it’s absorbed by air. A distance attenuation curve is used to show how quiet a sound is perceived to be at distances further from the sound source. This helped to create one of the rules they used for lowering the volume of sounds in-game. But lowering the volume is only one aspect. Different filters are added at different distances, like low-pass filters to muffle the sound, and reverb to help the sound gradually blend into the environment. This helps the sound to feel like it’s realistically getting further away.

Aside from processing, this effect can be achieved by cross-fading a close-up recording with a distant recording of the same sound source.

These tried-and-true metholds have been used with success on many game titles. But the concept is taken even further on Tears of the Kingdom because they wanted the player to have other information delivered via sound: how far away the source of the sound is, what direction it’s coming from, and what environment the source sound is coming from (a cave, open-air, a forest, etc.). To achieve this, sound parameters that change with distance – like volume, filters, and reverb processing – must be controlled by the same rules. When creating rules for sound, the team considered what characteristics they needed filters for, and the distance at which those filters should be applied.

Their interactive music system was the inspiration for their system for interactive sound effects. They created a system that allowed them to manipulate sounds so they played in the 3D game space, like changing the volume of sounds to match the character’s distance from its source, adding filters and reverb depending on the distance, panning sounds to match the sound source’s location on-screen, and processing sounds to match the sound source’s environment (i.e., sounds happening in a cave will have echo on them).

It’s also important to consider the characteristics of indirect sound. Sounds happening inside a house will sound different from ones happening inside a rock cave. Reverb size and length can help differentiate the two, but their approach to reverb in past games resulted in too many parameters to adjust and therefore, too much work. So for this game, the reverb parameters are automatically calculated using information about walls (e.g., direction of walls, distance to walls, room capacity, density of wall, and sound absorption rate of walls) collected by ray casting. This information is plugged into Eyring’s Reverberation Time Equation, and the results are used for reverb and echo parameters.

Since this game has more complicated terrain, like walls and cave, sound obstruction and occlusion are important considerations when creating realistic sounds. To depict a three-dimensional world in-game, the terrain contains voxel information. Each voxel contains terrain information, like whether it’s located indoors or outdoors, near water, near a forest, etc. Sound uses the voxels as well to find sound paths using an informed search algorithm. The sound system performs calculations based on the camera’s position and the sound source along a sound path. So, if a sound is coming from behind a wall, the sound path to the player is obstructed and the sound is occluded. As the player runs around the wall, toward the sound source, the path gets more direct and the sound changes accordingly. Doors dynamically affect the sound path. If a door is open, the player can hear a sound source inside the structure; if the door is closed, the sound cannot be heard.

The important aspect of sound attenuation and processing in the game is that all sounds follow the same rules. Among those rules, loudness is particularly important. If you create acoustic characteristics based on the loudness of a sound (how loud it is and how far away it can be heard), then assigning a loudness to each sound will allow you to hear them properly in the game space.

Instrumental music that isn’t a sound effect can also echo off of nearby walls or sound as though it’s coming from a canyon. It’s affected by the same rules and therefore plays naturally within the game space.

The sound team also needed to develop a system for handling non-dedicated sounds – sounds needed for player-created objects using ‘Ultrahand.’ This ability allows Link to grab, move, and rotate objects, and also attach things together to create bridges, and vehicles. How do you get sounds to play without dedicated implementation? The sound team created a system for combining existing sounds. For example, the sound of a wagon will combine the sounds of wheels rolling, short, repeated shaking sounds, creaking joints, and rattling chains. No dedicated ‘wagon’ sound was created for the game. There’s also no dedicated sound for the paddle boat. It’s just the combined sounds of wheels rotating on the water’s surface, and wooden boards splashing through the water. The sound changes due to the water resistance.

The sound team made a system based on the physics system (how rigid objects move). It analyzes the way rigid bodies – controlled by the physics system – will move and then bases the sound on an object’s size and material. As a result, various sounds will play without needing to have a dedicated program set up to play them. One example is the suspension bridges. Those are held together with just physics, without any dedicated suspension bridge program, and their wobbling and creaking sounds play automatically.

The non-dedicated sounds also adhere to the established sound loudness rules, so they play naturally within the game space.

This form of implementation – building a rule system for how sounds are made and react to the game world – resulted in high-quality sounds that even impressed the game’s sound designers. The system made sounds that the sound designers have no memory of creating. The game director noted that it’s basically a physics engine for sound.

The sound team didn’t set out with the goal of building such a system. It was the result of their efforts to use sound to make the expanded world of Hyrule more dynamic.

A big thanks to GDC for sharing the video, to Technical Director Takuhiro Dohta, Lead Physics Programmer Takahiro Takayama, and Lead Sound Engineer Junya Osada for the insights, and to Jennifer Walden for the recap!