I’m on a quest! I want to integrate Pure Data with Unity to make a Virtual Reality game focused on performing music. I’m fairly new to game development. My prior experience with game dev was when I was in my pre teens making Flash games. For this quest, I want to focus on this project being catered to performers….at least for now. How that will look? I’m not sure but I have built a framework to start with. I’m very aware there are many iterations that are currently in development or have been around for some time, and I admire those projects immensely. This will just be my flavor…
Lets get some back story on how I got here. Music with event based controls over 3D Visuals or better yet, Virtual Reality worlds, has been a dream of mine since I started making music. I was heavily inspired by Summer Wars and dabbled in Blender. I learned to model by giving myself the task of recreating pieces of the Land of OZ from Summer Wars. Sadly I’ve lost the file so I cant do much for it anymore 🙁 This is my last remaining render:
After familiarizing myself with 3D modeling I set my sights on developing a concept for this project.This was around 2014 and at this time I created a music video using the cube in this render. I mapped the scaling animations to the peaks in the audio file for the song. Not much going on there but it technically was my first audio controlled world.
I began exploring Unity3D and found it to be very capable for sound purposes. I got sound running and instantiated audio reactive cubes across the spectrum.
I continued documenting the project via Instagram as I learned new things. I’ve explored going the iOS/mobile route and found it to be somewhat limiting in user input.
A year passed and while still learning, progress was pretty much at a stand still. I acquired a Vive and very quickly got my prior proof of concept loading. I then shifted to exploring controller input and how to attach it’s properties to other objects. This allowed for picking up objects. Small step but was very important.
Getting a little better at this unity stuff. Avoiding prefabs as much as possible. Here I'm just testing some of the physics properties and controller actions. Got all my controller states mapped out. Plan to move on to making animation change based on controller state (glow differently when you hold things)
Sound manipulating objects!
Using Keijiro’s many frameworks I was able to get sound manipulating a 3D object! I split out a song over three tracks (drums, synth, percussion) and began experimenting with what a live song would look/feel like.
Got live audio routing now. Working on improving the latency. These photos are the progression of this concept since 2014. The first render is where I'm at now. Photo 2-3 is the initial concept from '14 done in blender. The following photos was my progression bringing it into a headset on a budget. It's been so fun. Progress is so much quicker now that I have a production machine/ vr setup and especially since the foundation is now built. . . . #vr #virtualreality #unity3d #indiedev
I became obsessed with iridescent shaders in the process of learning this stuff (partially because my girlfriend turned me on to how cool iridescence looked) I found that this type of shader was extremely good for understanding how to script color and lighting.
Pure Data has been an eye opening experience. I had built very basic sequencer/osc a few years ago and stopped looking into it due to the complexity of the routings. I found recently that an open source framework called Automatonism simplified a lot of my problems with Pd.
Pure Data and Unity!
After experimenting for a day I was hooked. I took a weekend to learn everything I could. I found that there was an amazing C# library called libpd and compiled myself a 64bit version of the library for use in Unity3d. The results? INPUT AND CONTROL OVER SOUND! 😀 My most significant progress towards this dream.
Yessssss got my pure data generative patch loading in unity. Now I can make accompanying sequences and parameters changes over my static backing tracks for this VR album idea I've had. You'll be able to manipulate the song by interacting with objects 😀 . . . #gamedev #puredata #unity3d #vrdev #electronicmusic
Got my generative pure data patch talking with unity3d! First example is the bang triggering kick drum in pure data standalone, next is game loading the patch and sending the bang message on press of space bar. After that I route the same space bar message to the start clock input of the automatonism clock. 😀 . . . . #unity3d #gamedev #indiedev #puredata #electronicmusic #automatonism
Proof of Concept
After going on a code binge I was able to bring to life a very simple proof of concept using the same Pd patch shown above. This gives the basic building blocks for what I need.
It’s starting to look and feel like something.
After a few years of breaking through the learning curve of C#, Pure Data, and Unity3D, I finally have a working game that feels like something I would be excited to perform in.
I’ll be continuing to update my blog as new things develop on this project. 🙂