From the makers of Little Big Planet comes a game that could literally change the way games are made. Media Molecule's Dreams โ out now on Sony PS4 โ is a game-building game that combines several artistic disciplines into one easy-to-use creation platform. From animation to music, Dreams allows anyone to create anything within a virtual space. Here, Audio Lead Tom Colvin discusses the power of Dreams from a music and sound creation standpoint.
Game developer Media Moleculeโs Dreams (published by Sony Interactive Entertainment for PS4) is a world building game like no other, because itโs more than just world-building; itโs game building. Users can create their own games, which they can share with other user. Or, they can play games created by other users. Or, a user can create assets to share with others โ like specific objects or entire backgrounds, music, bass lines, sound effects, templates โ you name it. The object of the game is creation not just consumption. Dreams coaxes out ones inner-artist and offers an expansive set of tools that foster creativity.
That description sounds vague, I know, because in Dreams anyone can create anything inside the game. Itโs really that open-ended.
Looking at the sound side โ and mainly at music creation within the game โ Media Moleculeโs Audio Lead Tom Colvin (along with Feedback Manager Alex Perry) held a workshop at Sรณnar+D 2018, which you can view further down in this interview.
Colvin shows off some of the music-creation tools and even creates an original piece of music with the workshop guests. He takes it further by showing how the music can work in-game. For instance, he quickly creates a generic world with a basic character, and then connects the music to the characterโs movement, so as the character speeds up the music speeds up.
Colvinโs demo of the game uses a version that was still in development, but itโs a great example of how easy it is to create and implement music into user-designed games/experiences.
A Sound Effect caught up with Colvin to find out how the music engine has evolved since that workshop in 2018, what key features and capabilities it now offers, whatโs included in the Dreams DAW, and how to get the most out of it.
Dreams โ Accolades Trailer
ย

How has the music engine evolved since that workshop/demo at Sonar+D in 2018?
Tom Colvin (TC): The sound engine was actually very close to complete at that time, the main differences are in the additional supporting materials.
We have a large library of instruments and sound effects that are available. We spent a lot of time and effort on providing elements that people will find useful and tuning that library to what the strengths of the game are and what we feel users are going to do with it. As a sound designer and musician, I was often faced with those kinds of choices.
Itโs more like a curation of sound materials. We went through a sound materials creation exercise to refine it. Weโve applied our own collective experiences of making sounds and music and thinking about the struggles we face, not technically but in terms of assembling material, assembling effective sounds and effective music. We had a lot of help from partners along the way.
We also made a large game called โArtโs Dreamโ to showcase the whole series of things you can do with Dreams. There are three different styles of gameplay. The whole experience is voiced by Art, who is going through a creative crisis. And his story is kind of like the story of what it was like to make Dreams โ in our minds anyway. There is a lot of joining of the music and sound creation to the other game features. That is something that Dreams does really well. The powerful thing about it is that all of the tools are in one place.
All of the tools and systems that run in Dreams are integrated nicely; theyโre all linked to our logic system. Itโs simple to, letโs say, have the music control visual elements or rhythmic gameplay aspects. The two can link really easily. As a sound designer or musician, itโs easy to come up with musical concepts for the gameplay. We tried to do as much of that as possible in โArtโs Dream.โ Thereโs a level where two characters have guitars and they are firing off notes. Those notes are in time, in key, and in sync with the music. Itโs all very music-oriented and that was made possible because we have this underlying system that keeps everything linked together.
ย

You not only created custom soft synths and virtual instruments for the game, but you also created your own DAW for use inside Dreams. That is a monumental undertaking! How did you accomplish this?
TC: One of the challenges we had while making Dreams was the decision to sometimes use paradigms that already exist in software. When I started working on Dreams, the brief was very wide. There were a lot of different ideas about how we could create a system for making sounds in the game. It wasnโt at all clear how we would do it. In fact, there were a lot of people who needed convincing that using a DAW would be a good way to go.
But, the way we generally tried to do things was to realize how different disciplines crossover. The example of the timeline is a very simple idea and itโs a really good one. You have time represented from left to right, and you have a line going across which represents your current time. Anyone who works in linear editing of any sort, from animators to musicians, all of these people will be familiar with a linear timeline. But we thought that if we bring that into Dreams, weโll be able to do a lot of game design things with that as well. It was important for us to make the music/sound engine more than just a DAW; it can also do the sequencing of lights. Or, you can do the sequencing of gameplay events or animations. It all lives on the same timeline. Itโs like you have Pro Tools and Maya and After Effects timelines all in one place. Itโs all the same timeline, all running live in the game.
โDreamsโ feels like a new chapter in how games are going to be made, and how interactive media is going to be made because of that unification of skills and disciplines.
It was a big technical undertaking but I didnโt have to do that part, fortunately. Cleverer people than me did that.
Dreams feels like a new chapter in how games are going to be made, and how interactive media is going to be made because of that unification of skills and disciplines.
There are two things I like about it. First, everyone finds their own flavor and style of creation within Dreams. You quickly recognize the different styles of different users. Here at work, itโs clear to me when something was created by Ed (sound designer/composer), or Lisa, or Ollie, or Todd, or the other people that work here. I can tell straightaway who was making something because their style is inside the set up. Dreams isnโt enforcing its own aesthetics (although it does have a lot of aesthetic-based choices within it).
Having the unification of all these different types of mediums in one place makes it easy to create anything. For instance, when Iโm not creating sound stuff, I like to make visualizer physics puzzles. Iโm working on a game which is a bit like being in space and youโre herding stars using gravity, like youโre a space shepherd, and there are a lot of musical and sound design aspects to it. In any other system, the bar for entry to do that would be just too high โ especially because itโs a hobby that I like to do in the afternoon or evening. It wouldnโt be possible. So Dreams is lowering the bar of access to people and making it possible for people to be playful in their technical creativity. Itโs so much less of a struggle to create things like that in Dreams.
Having the unification of all these different types of mediums in one place makes it easy to create anythingโฆโDreamsโ is lowering the bar of access to people and making it possible for people to be playful in their technical creativity.
Itโs funny how Iโm so embedded in it now. My creative process is in this thing. I have conversations with other sound designers and composers who are using other engines and I remember what it was like to have to deal with issues, like having to wait 10 minutes to reboot the game and then what you hear doesnโt always sound the same with the tools on the PC as it did in the engine. In Dreams, youโre working directly within the platform that youโre creating for. Itโs so immediate.
ย

What are some key features/capabilities of the music or sound creation systems?
TC: Many things in Dreams are based on the idea of things being performative. So, gestural controls are in the art tools. The system is designed to work with the PlayStation controller movements. The way we apply that to sound is that we use the buttons on the controller as notes. There are eight face buttons on the controller and that maps nicely to the eight notes of a scale. You tip left or right to play lower or higher octaves.
Then, on screen, we have a flat panel called the Canvas and you move your avatar (called an Imp) around the space using the motion controls, and thatโs like your mouse. When you press a button on the controller, you can draw onto the surface and the sound changes.
The system is designed to work with the PlayStation controller movementsโฆ
There are different areas on the surface which have an effect on the sound, kind of like guitar effects pedals.
If, for example, I play a synth note, I can draw through the user-configurable circles on the flat surface to change the sound of the note. The circles are like effects processing; so you can set one to control the pitch for example. As you draw through the pitch-control circle, you can bend the note in real-time.
There are eight face buttons on the controller and that maps nicely to the eight notes of a scale. You tip left or right to play lower or higher octaves.
But Iโm not limited to any one effect. All of the DSP can be wrapped up into a single field, so if you want a circle to have +12 pitch and distortion then you can do that. Itโs like a container for lots of different things that you might want to do with automation. Your movement through the field applies the effects you assigned to it.
You can record notes onto the Canvas. Then, once you finish recording them, you can look at them in a piano roll. The piano roll is a traditional-style roll where you can draw and edit notes.
You can look at them on the performance surface as well, which if you think of it in terms of a DAW, itโs where you would do your automation or sonic control of your plug-ins, or your changing of DSP.
The other thing to know is the entire engine is granular. All of the voices are grains all of the time. So itโs easy to pull sounds apart. You can create a stutter effect and play with the speed at which it plays back, and itโs all controllable by the fields. So itโs all performable. Itโs easy to do synthesis and textural things with the sounds. Itโs nice because itโs not just a software synth based on simple waveforms. Itโs like a cross between a synthesizer and a sampler. Your base waveforms are recordings rather than synthesized elements. We do have loads of synths in the library, but we also have real instrument recordings as well. You can make interesting textures using the grain synth.
Want to know more about the sound of Dreams? Check out this in-depth workshop with Audio Lead Tom Colvin (along with Feedback Manager Alex Perry), held at Sรณnar+D 2018:
Another cool thing is that thereโs a microphone so you can record in your own sounds or voices. So you can record in your own voices for the animations or games youโre creating, which is really fun.
โฆthe entire engine is granular. All of the voices are grains all of the time. So itโs easy to pull sounds apart.
On the creation side, weโve done a lot to make music composition easier for people that donโt know anything about music theory. The simplest thing you can do if you want to add music to your game is you can go in the library and stamp in someone elseโs music. But if you want to get involved in composition, Dreams users are able to publish anything they make, from full tracks to specific elements. If you want to be the person that makes bass lines or drum beats, thatโs great. You can publish those and let other people use your stuff, and thatโs a totally valid thing to do.
All the user-created published content gets put into our Internet-based database called the Dreamiverse. We had 100,000 people in our community before the game launched. So thereโs absolutely tons of content in there, plus the library that weโve created.
You can go through the music clips section and stamp things onto your timeline. Itโs a bit like other software you might have used. You can pull in pre-made clips of music but the difference is that, since all the music was created with our Dreams tools โ our instruments and sound technologies โ if you wanted to change any part of the music itโs all available to you; itโs all accessible.
Every single note you play has a unique instance of all of the different plug-ins that exist in โDreamsโ โ thatโs EQ, distortion, delay, filtering, grains, and other processing.
Another cool thing is how weโre handling all the DSP. Letโs say you were mixing something in a regular sound/music DAW; youโd have all your plug-ins after the instrument. So you have one instance of a chorus effect after a vocal part, for example. But in Dreams, because you are able to modulate between different DSP settings, all of the DSP is running a unique DSP chain per note. Every single note you play has a unique instance of all of the different plug-ins that exist in Dreams โ thatโs EQ, distortion, delay, filtering, grains, and other processing. The entire DSP chain is instanced for every note, which allows you to have one instrument but a different number of articulations of the same sound. It allows you to perform between them in real-time.
All of this is much easier to show than tell!
[tweet_box]Audio Lead Tom Colvin on Creating Sound/Music Inside โDreamsโ on PS4[/tweet_box]
ย

It sounds like it requires a significant amount of processing power. It sounds like it could get really complex really quicklyโฆ
TC: We spent a lot of time streamlining everything and making it as fast as possible. Itโs fast enough to make the game with. Our game is running all of the mixing, all of the instruments, all of the music, all of the sound effects for every level and itโs all running together all in real-time. The number of voices you can have depends on certain factors, but the system has been created in such a way that itโs very streamlined.
Again, cleverer people than me have done this. I canโt explain exactly how they did it but our technical director Alex Evans has always had a love of audio. He used to be a VJ for Warp Records at a time in his life when he was also doing graphics programming for the first Fable game. So some of his ideas for how Dreams works come from his designs for VJing software, like cross-fading between multiple different types of videos. Heโs also a musician and a sound person. So he pulled inspiration from past projects and brought them together in Dreams.
โฆthereโs this cross-disciplinary enthusiasm in the audio department, which I feel made it possible for us to achieve this.
We have another coder, Bogdan Vera, who comes from Romania. Heโs a brilliant guitarist and keyboardist, so thereโs this cross-disciplinary enthusiasm in the audio department, which I feel made it possible for us to achieve this. Quite often there is a divide between musicians and sound designers and coders who are implementing their desires. We needed to have people with cross-disciplines to really understand what works and what doesnโt. All of that somehow led to us having a very efficient sound engine.
 ย 

Because these sound tools are so streamlined, does that mean third-party audio plug-in makers wouldnโt be able to make tools for Dreams? Is your tech that proprietary?
TC: The reason itโs streamlined is for speed. In terms of the actual code, our EQโs and IR-based reverbs all use very efficient algorithms. They sound good but they arenโt hyper-exciting. Theyโre all very streamlined, and itโs created specifically for PS4 architecture. We donโt have the facility to import third-party audio plug-ins. It would be a real challenge for someone to do it, but who knows. Itโs probably not happening in the near future though.
 ย 

Any recommendations on how to get the most out of the music and sound creation engines?
TC: We have a good suite of tutorials. Thatโs the best place to start. When you start Dreams and play it through as a new player, youโll be guided through all of the processes. With Dreams, there are new paradigms for how to control something because we are using a PS4 controller. So itโs worth spending the time and taking it slow to begin with to get comfortable with the Dreams control scheme. Thereโs a lot of information in the tutorials about how everything works. Itโs a simple and focused step-by-step guide with simple goals. You can learn how the tools work really easily through that system.
โฆthere are new paradigms for how to control something because we are using a PS4 controller. So itโs worth spending the time and taking it slow to begin with to get comfortable with the โDreamsโ control scheme.
Then, once youโve reached the end of those, weโve done Twitch streams on different subjects and there are tons of people already in the community doing tutorials. Thereโs tons of information on YouTube. User tutorials can take you quite a ways. There are tons of different techniques and ways to use Dreams and people have documented those. We do our best to promote those in our Twitch streams as well.
We have a series of more in-depth tutorials called โMaster Classesโ and those are very specific. For example, our art director does one where he is assembling a human head using the Dreams sculpture tools. But, itโs more about, โWhat would you do using Dreams to achieve a certain result?โ I fully expect that we will be doing audio versions of those in the future. We donโt have an audio one yet, but we will.
 ย  

Now that youโve put this fantastic creation platform in the hands of the masses, what do you hope to hear coming out of the Dreams community?
TC: The things that excite me the most are when people start doing things we didnโt expect. I didnโt expect people to use Dreams in a more conventional recording studio sense, but there are people doing that. There is a guy who released an album of music that he made in Dreams. We met a bunch of creators while on a promotional tour, and there were some Italian guys creating these insane music visualizers. There are animators creating their own characters and doing really entertaining sketch shows in Dreams. There are people doing algorithmic-based music using Euclidean mathematics. I donโt know whatโs going to come next. Itโs such a fast creation system and it encourages people to do silly, sketchy stuff. Itโs always surprising to see with the community has been up to and that is the exciting thing for me. You donโt know what to expect; you donโt know what somebody is going to do next.
I didnโt expect people to use โDreamsโ in a more conventional recording studio sense, but there are people doing that.
Everyone should have Dreams in their lives. Itโs really rewarding. Everyone has a little place in their heart where they want to make something. And some people have had that talked out of them. Going to school and getting a job were the focus, or they were told that creation wasnโt for them. We forget that everyone has creative ideas; everyone has ideas for making things. Itโs just that itโs been suppressed or held back, but Dreams is meant for the fun of creation. You donโt have to make something that has to impress people. You can just entertain yourself by making a flower that sings a silly song and you can share that with your friends. Itโs the joy of creation for its own sake. Everyone should have that in their lives.
ย
A big thanks to Tom Colvin for giving us a behind-the-scenes look at the sound of Dreams and to Jennifer Walden for the interview!
