So a thing happened over the weekend. My scheme to mash up spatial computing and DJing became real. Here’s a clip:

So what’s happening here is this: The earlier Raspberry Pi with AudioInjector HAT now has a twin, with a cloned SD card of the first one.

There is a Pyle 444 preamp on each one:

This is necessary to unwrap the RIAA curve from the turntable signal and turn it into line level, which is what the Audioinjector HAT needs. In order to cure the abysmal latency brought on by the import process over X11, a single-deck instance of Xwax runs headless on each Pi, which are renamed deck0 and deck1.

On the Magic Leap, an app made with Unreal Engine 4 gives me the following:
– Hand Tracking
– Laser-pointer 3D VR mouse
– OSC communication with the Pi

There is a game object for each deck, which contains the track selection and navigation UI, and the status circles. The inner circle tracks the playback speed via pitch value from Xwax, and the outer circle represents the elapsed time.

The circle is at the origin of the deck object, and it is where the Persistent Coordinate Frame for the deck relocates after the menu is moved. This enables the menu to fire up in the same space when restarting the Magic Leap or the application.

Hand tracking enables using the menu buttons with a finger instead of the controller. Your hands need to be free to use the turntables and mixer like you normally would.

A few things:
– The status circles don’t line up with the record label in the video. This is because the capture camera is mounted on the Magic Leap at an offset from the render projection. It also has a godawful field of view.
– Recorded audio from the Magic Leap is present with the recorded audio off the DJ mixer, so there’s that weird phasing effect. It means I need to implement a way to disable audio recording in-session.
– The menu needs to be redesigned so it looks a little less crap. At this point, it works, and I’m OK with that.
– A better way to record what this looks like will be with a smartphone looking through the lens instead of the capture camera, because the offsets and the FOV really, truly make the whole thing look like shit.

TO DO:
– It would be nice to be able to start and stop Xwax from within the App
– It would be nice to implement multiplayer networking so the experience can be seen on multiple devices.
– It would be nice if UE4’s native OSC plugin supported Android, but for now, PC and Mac only. Will have to port this later.
– It would be nice for both decks to access the same storage volume instead of using duplicate USB sticks.
– It would be nice to make use of the cue point functionality and store those with the tracks.
– It would be nice to replace the outer status circle with a waveform display showing Elapsed/Remaining in representative colors, but that might violate the principle of having the Pi’s do the heavy lifting

All in all, it’s a success. I can DJ using this system. I can select tracks from disk, load them, and play them using the DVS functionality of Xwax. This will probably not become a product, but it’s at least a platform to experiment with different ways of blending virtual and physical DJ techniques.