Air VJ - CaseStudy

Overview

CyberArts Gallery in Boston for the Fresh Media Show

Air VJ is an immersive interactive installation where visitors could use gestures to manipulate dynamic visuals in sync with music. The installation was showcased at the CyberArts Gallery in Boston.

What did the visitors think?

“This felt like a bite of the future, so rad! Kinda want to incorporate this in my games too. I saw all age groups swaying to the beats and trying out weird gestures, sometimes even dancing with their dogs lollll”

- Drew Thomas, Game Developer, Massart

Project Context

A Snack, a YouTube Ad, and an Idea sparked - experimenting with NUIs

An everyday inconvenience - messy hands and touchscreen interactions—sparked an exploration into Natural User Interfaces (NUIs) and GUIs.  Why settle for basic utility? I envisioned a world where users could draw, create, and VJ using just their hands, blending functionality with surprise and delight.

A Glimpse at the process - bringing a speculative idea to life in a prototype

Phase 1

Research and DJ Insights

Phase 2

Crafting Audio Visuals

Phase 3

Mapping Gestures

Phase 4

Gallery Installation/Testing

Phase 5

Speculating the Future

Technology & Execution

Handpose detection with ml5.js

1. Leap Motion Controller → Captured hand and finger gestures.

2. TouchDesigner → Generated and manipulated visuals based on real-time input.

3. ml5.js Handpose Library → Processed and mapped gesture detection for musical triggers.
‍
4. Audio-Visu
al Sync → Gestures not only shaped visuals but also played sound, creating a full-body immersive experience.

Future Applications: Beyond the Gallery

Enhanced Video Calls - Manage settings like mute, screen sharing, and camera angles etc

Safe Vehicle Controls - Hands-free navigation and media control to enhance driver safety

Industrial Machinery Control - Operate machines without physical contact to reduce injuries

Medical Equipment Operation - hands-free to improve hygiene and reduce contamination

Interactive Presentations - Presenters control using natural gestures to captivate audiences

Gaming and Entertainment - Deeper immersion by controlling virtual environments with movements

Gesture-based interfaces open new frontiers across industries

What's the Next?

Air VJ is just the beginning. The potential for natural, touchless interactions spans creative expression, healthcare, education, gaming, and more. Future iterations can leverage higher fidelity gesture tracking, haptic feedback, and AI-powered intent recognition—moving us closer to technology that feels like an extension of the human body.