Has AI Replaced Technical Sound Designers? An experiment by Brennan Anderson:

I was recently at a conference, and was having conversations about the state of AI in the games industry, specifically in my field of Technical Sound Design. It made me reflect quite a bit, and I wanted to share my thoughts here; I want to look at this more thoroughly without jumping to a panicked state of “AI has finally come for my job”.

AI is here

I think that is as good a place to start as any. The reality is that AI IS here, and it HAS affected all of our jobs. Sound designers are being asked to handle more complex audio implementation tasks while using AI to assist them. This certainly shifts the focus of what Technical Sound Designers are tasked to do. The window of responsibilities has shifted because of AI; more junior level people can accomplish higher level and more technical tasks, so it follows that the Technical Sound Designer’s role shifts to higher level thinking and engineering tasks. We may even have to start doing code reviews of our sound designer’s AI assisted output! This is the kind of democratization I only ever dreamed of as an up and coming sound designer. It was a fantasy, so much so that I convinced myself that I absolutely had to learn how to program to bring into reality the kind of work I wanted to do. To be in full control of what you hear in a game. That has always been the dream, and we have always made concessions and relied on collaboration and teamwork to make it happen. Even in the pursuit of that dream, the dream changed. The goal is no longer for one person to be in complete control of the audio, the goal now is for the audio team to be in complete control of the audio. Chasing the dream revealed something important that most everybody here should resonate with: making games is a team sport.

The Dunning Kruger effect of AI

After my conversations of fear, promise, and even bullish hope of what AI could achieve, I wanted to see what could actually be done with it. Now, full disclosure here, I am no stranger to AI, LLMs, etc.. I have been testing their limits in real production environments over the last year. They have amazed me, and they have frustrated me. But I wanted to test an AI on something so deceptively simple. Something that I have both failed and succeeded at in various ways over the years. Something any Technical Sound Designer worth their salt will attest kinda sucks… Footsteps.

Before I go into this, I am aware of many possible solutions to the footstep problem. The following is only one possible way to do it, and I like it because it has challenges to overcome.

The footstep problem

The problem: large scale AAA games with dozens of characters with unique animation sets with dozens to hundreds of animations all require footsteps to be placed on all of those animations. This is dozens of hours of mind-numbing work scrubbing through a timeline and placing a footstep at the right frame. You finally finish, and then, WHOOPS, the animation team decided to change every single animation by a couple of frames, and none of the producers remembered to tell the audio team that this was happening. Guess who has to adjust the footsteps in every animation? The intern, if you have one. If you don’t then it is probably YOU! The solution: create a system where the game automatically detects when the character’s foot is close enough to the ground to make a footstep. Simple, right? WRONG!

You can make a system like that, sure, and it will work great on flat ground. But if you don’t think through how it works, then you will very quickly encounter your mortal enemy… hills. Any uneven terrain that exists, and a character controller that does not predictively move the way a human does by picking its feet up higher for uneven terrain, will give you an uneven mess of footstep detection that just sounds bad as the foot sweeps and clips through hills. Now, maybe you have the budget to make a perfect character controller, but you really don’t. So this becomes a big problem. I was curious then, how would AI handle it?

What the AI came up with

It was able to get the standard detection implementation done really fast, and fairly efficiently. It tracked the distance between the foot and the ground using a spherecast, and played a footstep when the distance was small enough. It even put in logic to wait until the foot lifted far enough from the ground before it could trigger again. Great stuff, all stuff I would have done, and have done in the past. But, the first run didn’t even consider uneven terrain or hills. It very confidently did not even think about it.

Here is the result without any compensation for slopes:

The foot clips into the slope, causing early footstep triggering.

Not a huge problem, so I prompted it to come up with a solution for slope compensation. On its first attempt, it added functions to a math library to calculate a new contact plane based on the slope degree. Seemed reasonable; it sounded terrible. On its second attempt, it used the slope angle to adjust where the footstep probe started and ended, with larger slopes raising the probe higher. It also sounded terrible. At this point I had spent almost $200 in credits trying different things. I was purposefully letting the AI handle it with a little bit of pushing in various directions, but I wanted this to be a system the AI reasoned through to test how well it could cope. It really couldn’t. It very confidently invented new ways to solve the problem, all of which just sounded bad. No matter how many variables it gave me to adjust, no combination gave me a consistent footstep detection gait that would be anywhere close to acceptable in a shipping game. I almost wanted to open up the animator and add footstep markers!

Here is the result after it added slop compensation. It is a little better, but not by much. I wouldn’t ship this.

Still getting early footstep triggering uphill. Downhill is slightly better.

Drawing it out

So now it was time for me to step in. I thought about the problem. A problem I myself had gotten very confidently wrong in the past. So wrong that we actually did have to go back to manual or semi-manual tagging of hundreds of animations in a AAA game. This failure had haunted me for years, and here I was again, years more experience under my belt trying to solve the problem again. I drew pictures. I find that helps reason through spatial and sound problems.

I guess I had never drawn the problem out before, or maybe I just needed to revisit the problem again after some time, because after drawing half a dozen crude feet and hills and trajectories over about 15 minutes, the solution was immediately clear to me. I didn’t need or want any fancy math, no complex algorithms. The solution was as simple as a solution that I had ever thought up before, and when I implemented it, it worked nearly perfectly. Instead of detecting the footsteps directly beneath the feet, I moved the probe behind the foot by 6-12 inches. This causes the footsteps to behave identically on flat surfaces, then on uneven terrain, you don’t get the premature detection of the foot sweeping into the terrain, and the lagging probe will more correctly trigger when the foot comes to its natural resting position. If I wanted, I could make it a bit more complex, and move the probe further back depending on the slope angle, but I was tired so I wanted to call it a day and take the win.

Here is the result of my fix. It maybe still needs some tweaking, but I would be much more comfortable shipping this than the slope solutions the AI provided.

No more early footstep triggering up slopes, giving us a more consistent sound.

What can never be gained

My victory over this problem was done with things that AI just doesn’t have. Understanding. Inductive chain learning. Lived experience. Failure that haunts my memories. Disappointment. Joy of discovery and problem solving. Drive. With a tool as powerful as AI is, why could it not think of a good sounding solution in the hours of time it spent on the problem that took me 15 minutes? Well, to be fair, I guess it did take my local brain-AI several years to figure out the problem. But that process is exactly what makes people experts.

We know the questions we need to ask. We know what has and hasn’t worked and why. But we don’t only know the words to why something didn’t work. We also know the feelings of why something didn’t work. We know that even though we tried fifty different variations of something, it still didn’t work. We know the pain of that, and those experiences mold us into craft experts. So when people talk about whether AI will replace Technical Sound Designers, or any role really, I have to say that the answer is a resounding NO! AI as it stands cannot experience the world the way we do. It is pure hubris to say that we can create something in such a short amount of time that has the raw capabilities that billions of years of evolution has harnessed. Will we be able to someday? Maybe. Can AI currently outperform us in certain things? Absolutely! But there is still a human element that is unmatched by AI. It would be shortsighted to try to get rid of that, and foolish to believe that it is unnecessary in our pursuit of art, entertainment, and connection.