Audio Lead Tom Colvin on Creating Sound/Music Inside ‘Dreams’ on PS4

Game developer Media Molecule’s Dreams (published by Sony Interactive Entertainment for PS4) is a world building game like no other, because it’s more than just world-building; it’s game building. Users can create their own games, which they can share with other user. Or, they can play games created by other users. Or, a user can create assets to share with others — like specific objects or entire backgrounds, music, bass lines, sound effects, templates — you name it. The object of the game is creation not just consumption. Dreams coaxes out ones inner-artist and offers an expansive set of tools that foster creativity.

That description sounds vague, I know, because in Dreams anyone can create anything inside the game. It’s really that open-ended.

Looking at the sound side — and mainly at music creation within the game — Media Molecule‘s Audio Lead Tom Colvin (along with Feedback Manager Alex Perry) held a workshop at Sónar+D 2018, which you can view further down in this interview.

Colvin shows off some of the music-creation tools and even creates an original piece of music with the workshop guests. He takes it further by showing how the music can work in-game. For instance, he quickly creates a generic world with a basic character, and then connects the music to the character’s movement, so as the character speeds up the music speeds up.

Colvin’s demo of the game uses a version that was still in development, but it’s a great example of how easy it is to create and implement music into user-designed games/experiences.

A Sound Effect caught up with Colvin to find out how the music engine has evolved since that workshop in 2018, what key features and capabilities it now offers, what’s included in the Dreams DAW, and how to get the most out of it.

Dreams – Accolades Trailer

How has the music engine evolved since that workshop/demo at Sonar+D in 2018?

Tom Colvin (TC): The sound engine was actually very close to complete at that time, the main differences are in the additional supporting materials.

We have a large library of instruments and sound effects that are available. We spent a lot of time and effort on providing elements that people will find useful and tuning that library to what the strengths of the game are and what we feel users are going to do with it. As a sound designer and musician, I was often faced with those kinds of choices.

It’s more like a curation of sound materials. We went through a sound materials creation exercise to refine it. We’ve applied our own collective experiences of making sounds and music and thinking about the struggles we face, not technically but in terms of assembling material, assembling effective sounds and effective music. We had a lot of help from partners along the way.

We also made a large game called “Art’s Dream” to showcase the whole series of things you can do with Dreams. There are three different styles of gameplay. The whole experience is voiced by Art, who is going through a creative crisis. And his story is kind of like the story of what it was like to make Dreams — in our minds anyway. There is a lot of joining of the music and sound creation to the other game features. That is something that Dreams does really well. The powerful thing about it is that all of the tools are in one place.

All of the tools and systems that run in Dreams are integrated nicely; they’re all linked to our logic system. It’s simple to, let’s say, have the music control visual elements or rhythmic gameplay aspects. The two can link really easily. As a sound designer or musician, it’s easy to come up with musical concepts for the gameplay. We tried to do as much of that as possible in “Art’s Dream.” There’s a level where two characters have guitars and they are firing off notes. Those notes are in time, in key, and in sync with the music. It’s all very music-oriented and that was made possible because we have this underlying system that keeps everything linked together.

You not only created custom soft synths and virtual instruments for the game, but you also created your own DAW for use inside Dreams. That is a monumental undertaking! How did you accomplish this?

TC: One of the challenges we had while making Dreams was the decision to sometimes use paradigms that already exist in software. When I started working on Dreams, the brief was very wide. There were a lot of different ideas about how we could create a system for making sounds in the game. It wasn’t at all clear how we would do it. In fact, there were a lot of people who needed convincing that using a DAW would be a good way to go.

But, the way we generally tried to do things was to realize how different disciplines crossover. The example of the timeline is a very simple idea and it’s a really good one. You have time represented from left to right, and you have a line going across which represents your current time. Anyone who works in linear editing of any sort, from animators to musicians, all of these people will be familiar with a linear timeline. But we thought that if we bring that into Dreams, we’ll be able to do a lot of game design things with that as well. It was important for us to make the music/sound engine more than just a DAW; it can also do the sequencing of lights. Or, you can do the sequencing of gameplay events or animations. It all lives on the same timeline. It’s like you have Pro Tools and Maya and After Effects timelines all in one place. It’s all the same timeline, all running live in the game.

‘Dreams’ feels like a new chapter in how games are going to be made, and how interactive media is going to be made because of that unification of skills and disciplines.

It was a big technical undertaking but I didn’t have to do that part, fortunately. Cleverer people than me did that.

Dreams feels like a new chapter in how games are going to be made, and how interactive media is going to be made because of that unification of skills and disciplines.

There are two things I like about it. First, everyone finds their own flavor and style of creation within Dreams. You quickly recognize the different styles of different users. Here at work, it’s clear to me when something was created by Ed (sound designer/composer), or Lisa, or Ollie, or Todd, or the other people that work here. I can tell straightaway who was making something because their style is inside the set up. Dreams isn’t enforcing its own aesthetics (although it does have a lot of aesthetic-based choices within it).

Having the unification of all these different types of mediums in one place makes it easy to create anything. For instance, when I’m not creating sound stuff, I like to make visualizer physics puzzles. I’m working on a game which is a bit like being in space and you’re herding stars using gravity, like you’re a space shepherd, and there are a lot of musical and sound design aspects to it. In any other system, the bar for entry to do that would be just too high – especially because it’s a hobby that I like to do in the afternoon or evening. It wouldn’t be possible. So Dreams is lowering the bar of access to people and making it possible for people to be playful in their technical creativity. It’s so much less of a struggle to create things like that in Dreams.

Having the unification of all these different types of mediums in one place makes it easy to create anything…’Dreams’ is lowering the bar of access to people and making it possible for people to be playful in their technical creativity.

It’s funny how I’m so embedded in it now. My creative process is in this thing. I have conversations with other sound designers and composers who are using other engines and I remember what it was like to have to deal with issues, like having to wait 10 minutes to reboot the game and then what you hear doesn’t always sound the same with the tools on the PC as it did in the engine. In Dreams, you’re working directly within the platform that you’re creating for. It’s so immediate.

What are some key features/capabilities of the music or sound creation systems?

TC: Many things in Dreams are based on the idea of things being performative. So, gestural controls are in the art tools. The system is designed to work with the PlayStation controller movements. The way we apply that to sound is that we use the buttons on the controller as notes. There are eight face buttons on the controller and that maps nicely to the eight notes of a scale. You tip left or right to play lower or higher octaves.

Then, on screen, we have a flat panel called the Canvas and you move your avatar (called an Imp) around the space using the motion controls, and that’s like your mouse. When you press a button on the controller, you can draw onto the surface and the sound changes.

The system is designed to work with the PlayStation controller movements…

There are different areas on the surface which have an effect on the sound, kind of like guitar effects pedals.

If, for example, I play a synth note, I can draw through the user-configurable circles on the flat surface to change the sound of the note. The circles are like effects processing; so you can set one to control the pitch for example. As you draw through the pitch-control circle, you can bend the note in real-time.

There are eight face buttons on the controller and that maps nicely to the eight notes of a scale. You tip left or right to play lower or higher octaves.

But I’m not limited to any one effect. All of the DSP can be wrapped up into a single field, so if you want a circle to have +12 pitch and distortion then you can do that. It’s like a container for lots of different things that you might want to do with automation. Your movement through the field applies the effects you assigned to it.

You can record notes onto the Canvas. Then, once you finish recording them, you can look at them in a piano roll. The piano roll is a traditional-style roll where you can draw and edit notes.

You can look at them on the performance surface as well, which if you think of it in terms of a DAW, it’s where you would do your automation or sonic control of your plug-ins, or your changing of DSP.

The other thing to know is the entire engine is granular. All of the voices are grains all of the time. So it’s easy to pull sounds apart. You can create a stutter effect and play with the speed at which it plays back, and it’s all controllable by the fields. So it’s all performable. It’s easy to do synthesis and textural things with the sounds. It’s nice because it’s not just a software synth based on simple waveforms. It’s like a cross between a synthesizer and a sampler. Your base waveforms are recordings rather than synthesized elements. We do have loads of synths in the library, but we also have real instrument recordings as well. You can make interesting textures using the grain synth.

Another cool thing is that there’s a microphone so you can record in your own sounds or voices. So you can record in your own voices for the animations or games you’re creating, which is really fun.

…the entire engine is granular. All of the voices are grains all of the time. So it’s easy to pull sounds apart.

On the creation side, we’ve done a lot to make music composition easier for people that don’t know anything about music theory. The simplest thing you can do if you want to add music to your game is you can go in the library and stamp in someone else’s music. But if you want to get involved in composition, Dreams users are able to publish anything they make, from full tracks to specific elements. If you want to be the person that makes bass lines or drum beats, that’s great. You can publish those and let other people use your stuff, and that’s a totally valid thing to do.

All the user-created published content gets put into our Internet-based database called the Dreamiverse. We had 100,000 people in our community before the game launched. So there’s absolutely tons of content in there, plus the library that we’ve created.

You can go through the music clips section and stamp things onto your timeline. It’s a bit like other software you might have used. You can pull in pre-made clips of music but the difference is that, since all the music was created with our Dreams tools — our instruments and sound technologies — if you wanted to change any part of the music it’s all available to you; it’s all accessible.

Every single note you play has a unique instance of all of the different plug-ins that exist in ‘Dreams’ — that’s EQ, distortion, delay, filtering, grains, and other processing.

Another cool thing is how we’re handling all the DSP. Let’s say you were mixing something in a regular sound/music DAW; you’d have all your plug-ins after the instrument. So you have one instance of a chorus effect after a vocal part, for example. But in Dreams, because you are able to modulate between different DSP settings, all of the DSP is running a unique DSP chain per note. Every single note you play has a unique instance of all of the different plug-ins that exist in Dreams — that’s EQ, distortion, delay, filtering, grains, and other processing. The entire DSP chain is instanced for every note, which allows you to have one instrument but a different number of articulations of the same sound. It allows you to perform between them in real-time.

All of this is much easier to show than tell!

[tweet_box]Audio Lead Tom Colvin on Creating Sound/Music Inside ‘Dreams’ on PS4[/tweet_box]

It sounds like it requires a significant amount of processing power. It sounds like it could get really complex really quickly…

TC: We spent a lot of time streamlining everything and making it as fast as possible. It’s fast enough to make the game with. Our game is running all of the mixing, all of the instruments, all of the music, all of the sound effects for every level and it’s all running together all in real-time. The number of voices you can have depends on certain factors, but the system has been created in such a way that it’s very streamlined.

Again, cleverer people than me have done this. I can’t explain exactly how they did it but our technical director Alex Evans has always had a love of audio. He used to be a VJ for Warp Records at a time in his life when he was also doing graphics programming for the first Fable game. So some of his ideas for how Dreams works come from his designs for VJing software, like cross-fading between multiple different types of videos. He’s also a musician and a sound person. So he pulled inspiration from past projects and brought them together in Dreams.

…there’s this cross-disciplinary enthusiasm in the audio department, which I feel made it possible for us to achieve this.

We have another coder, Bogdan Vera, who comes from Romania. He’s a brilliant guitarist and keyboardist, so there’s this cross-disciplinary enthusiasm in the audio department, which I feel made it possible for us to achieve this. Quite often there is a divide between musicians and sound designers and coders who are implementing their desires. We needed to have people with cross-disciplines to really understand what works and what doesn’t. All of that somehow led to us having a very efficient sound engine.
 

TC: The reason it’s streamlined is for speed. In terms of the actual code, our EQ’s and IR-based reverbs all use very efficient algorithms. They sound good but they aren’t hyper-exciting. They’re all very streamlined, and it’s created specifically for PS4 architecture. We don’t have the facility to import third-party audio plug-ins. It would be a real challenge for someone to do it, but who knows. It’s probably not happening in the near future though.
 

Any recommendations on how to get the most out of the music and sound creation engines?

TC: We have a good suite of tutorials. That’s the best place to start. When you start Dreams and play it through as a new player, you’ll be guided through all of the processes. With Dreams, there are new paradigms for how to control something because we are using a PS4 controller. So it’s worth spending the time and taking it slow to begin with to get comfortable with the Dreams control scheme. There’s a lot of information in the tutorials about how everything works. It’s a simple and focused step-by-step guide with simple goals. You can learn how the tools work really easily through that system.

…there are new paradigms for how to control something because we are using a PS4 controller. So it’s worth spending the time and taking it slow to begin with to get comfortable with the ‘Dreams’ control scheme.

Then, once you’ve reached the end of those, we’ve done Twitch streams on different subjects and there are tons of people already in the community doing tutorials. There’s tons of information on YouTube. User tutorials can take you quite a ways. There are tons of different techniques and ways to use Dreams and people have documented those. We do our best to promote those in our Twitch streams as well.

We have a series of more in-depth tutorials called “Master Classes” and those are very specific. For example, our art director does one where he is assembling a human head using the Dreams sculpture tools. But, it’s more about, “What would you do using Dreams to achieve a certain result?” I fully expect that we will be doing audio versions of those in the future. We don’t have an audio one yet, but we will.
 

TC: The things that excite me the most are when people start doing things we didn’t expect. I didn’t expect people to use Dreams in a more conventional recording studio sense, but there are people doing that. There is a guy who released an album of music that he made in Dreams. We met a bunch of creators while on a promotional tour, and there were some Italian guys creating these insane music visualizers. There are animators creating their own characters and doing really entertaining sketch shows in Dreams. There are people doing algorithmic-based music using Euclidean mathematics. I don’t know what’s going to come next. It’s such a fast creation system and it encourages people to do silly, sketchy stuff. It’s always surprising to see with the community has been up to and that is the exciting thing for me. You don’t know what to expect; you don’t know what somebody is going to do next.

I didn’t expect people to use ‘Dreams’ in a more conventional recording studio sense, but there are people doing that.

Everyone should have Dreams in their lives. It’s really rewarding. Everyone has a little place in their heart where they want to make something. And some people have had that talked out of them. Going to school and getting a job were the focus, or they were told that creation wasn’t for them. We forget that everyone has creative ideas; everyone has ideas for making things. It’s just that it’s been suppressed or held back, but Dreams is meant for the fun of creation. You don’t have to make something that has to impress people. You can just entertain yourself by making a flower that sings a silly song and you can share that with your friends. It’s the joy of creation for its own sake. Everyone should have that in their lives.

A big thanks to Tom Colvin for giving us a behind-the-scenes look at the sound of Dreams and to Jennifer Walden for the interview!