Browser Based Music Interface Design

I have an idea! I'm writing this to hold myself to it. If you've not been keeping up with my personal happenings lately, let me catch you up really quick.

The last few years I've been chipping away at this idea of building a music performance application for XR (Extended Reality). I hacked together a somewhat stable framework for demo purposes and over the last year, I've refined it to something stable that finally builds for Windows (more platforms eventually ?.) The result has morphed into a game that I call Broken Place. The game's audio relies heavily on an open source visual programming language called Pure Data, which is extremely popular for multimedia. You can build very complex pieces of music using this language and I have developed a way to interface with this from Unity UI components. One thing I particularly like about this language is the patch file size is usually kilobytes but still generates complex sounds. This makes it super easy to handle users uploading on a post editor and downloading from an XR headset.

*The actual source will be open sourced as soon as it is cleaned up and stable. 🙂

Before I get on with the idea, some more backstory about WordPress things

If you don't already know, my day job is WordPress development and I spend a great deal of that time contributing to the WordPress open source project. Recently, I found myself in a position to help as one of the handful of release leads heading up one of the largest changes in the history of the project. In this release, the old content creation process was rebuilt with new block based approach to managing post content. This change brings a considerable chunk of the web to an open future for content creation in a browser. Users of all skill levels are empowered to create content in interesting ways. This content does not have to be specific to a web browser. On to the idea....

Music interfaces designed in a web/browser based editor

I’ve been thinking through how I could use the new block based WordPress editor for music interfaces in my game. I imagine a user logging into a WordPress to a custom song post type with the option to upload a Pure Data music file. The editor could be used for making grid layouts. The block would have placeholder images representing the custom performance interface element (knobs, buttons, sequencers.) In the example below, we see a grid layout of knobs. The caption of each is defined by the user. These messages could be very important messages for something like a Pure Data patch which is listening for parameters associated with those messages. In the game, we would know how many cells to put across a row by querying the column block's number attribute. In 3D world space, the game would then populate the components associated with the attributes of each block queried from WordPress. Each block's attributes represents what type of 3D UI element drops in the cell and even detailed parameters about that element (maybe min/max ranges).

Blockers and a spiffy option

Currently WordPress Core out of the box cannot query via REST API to get individual block data. Roy Sivan is currently working on a plugin that would allow this to happen( Hi Roy!) Here's the github link if you want to test or contribute: https://github.com/royboy789/gutenberg-object-plugin

The plugin allows for block data to be exposed via REST API. This will be crucial as I start building a web editor for my game.

More to come as I find time to work on this. I'm open to dialogue with any that are interested. Feel free to leave thoughts below.