Motion Mirror: Turning a Dancing Crowd into Neon Fluid

What if a light show did not simply react to the music, but to everyone dancing in front of it?

That question became Motion Mirror: a browser-based installation that transforms the movement of a crowd into glowing skeletons and flowing neon color in real time. A camera observes the dance floor, a pose model extracts the movement of multiple people, and a GPU fluid simulation turns their gestures into light. Arms cut through color, fast movements release fluid impulses, and the entire image changes with the beat.

Motion Mirror is the software component of an interactive installation for live music events. It accepts a live camera or an uploaded video, then performs video capture, pose inference, multi-person tracking, motion smoothing, beat analysis, effect sequencing and rendering entirely on the client. There is no inference server and no cloud rendering pipeline: the finished system is a static web application.

Building it was less about choosing a fashionable model or graphics API and more about finding a combination that remained responsive, visually expressive and practical during a live installation. Between April and August 2026, I worked through a series of focused prototypes to find that combination.

Finding the right kind of tracking

The first experiments were not pose estimators at all. I tested general person detection with a YOLO26 Nano model, ONNX Runtime Web and a simple centroid tracker. The prototype assigned IDs, drew traces and counted people crossing a line. It was useful for comparing camera footage, model latency and browser inference backends, but it also exposed the limitation of bounding boxes: they can show where a person is, not how they are dancing. It can create a very abstract outline of the movement of a crowd, but I wanted to make it more immersive if I could.

I then tested landmark-based tracking with MediaPipe Pose Landmarker. That route worked well for a compact, offline-capable iPad prototype and proved that body landmarks could drive a fluid experience directly. It also helped answer practical questions around camera permissions, GPU and CPU delegates, cached model assets, service workers and operation without a reliable network connection. For the final crowd installation, however, I needed a pipeline designed around simultaneous poses rather than a single foreground performer.

The next prototypes moved to YOLO pose models. I benchmarked different input sizes, simplified ONNX exports, full-precision and FP16 variants, and multiple ONNX Runtime Web execution providers. Some early FP16 exports did not run reliably with the browser WebGPU runtime, which made model export and runtime configuration part of the development work rather than a one-click conversion step.

The production setup uses a compact YOLO26n pose model exported to ONNX/FP16 with a fixed 288 × 384 input. That portrait-oriented input is a deliberate optimization: the model receives enough information to recover a body pose without processing a large square texture for every frame. The final tracker is configured for crowded scenes and can accept up to 40 pose candidates, although the useful capacity of an installation naturally depends on camera position, occlusion and the available GPU.

Keeping inference out of the visual loop

A real-time installation cannot allow model inference to interrupt rendering or interface controls. Motion Mirror therefore runs ONNX Runtime Web inside a dedicated module worker and requests the WebGPU execution provider explicitly.

Each camera or video frame is transferred to the worker as an ImageBitmap. There it is letterboxed, converted into the model tensor and evaluated. The inverse letterbox transform maps the detected keypoints back into the original image. Only compact pose data returns to the main thread. If inference is still busy when a new frame arrives, that frame is dropped instead of creating a growing queue and steadily increasing latency.

This architecture keeps the camera, effect controls and render loop responsive. It also makes performance failures visible. The production build does not silently fall back to a much slower inference backend: if WebGPU is unavailable, the installation reports the problem rather than appearing to work while accumulating delay.

From detections to continuous dancers

Raw pose output is not yet a usable visual instrument. Detection order can change from one frame to the next, keypoints jitter, limbs disappear behind other bodies and a person can briefly drop below the confidence threshold. If those changes are fed directly into a simulation, the result flickers and explodes.

I built a lightweight multi-person tracker that matches new poses to existing tracks using pose distance, bounding-box overlap and center distance. Tracks survive short gaps, giving each dancer a stable ID and preserving continuity through momentary occlusion.

On top of that, a One Euro filter smooths the box and every keypoint. Unlike a fixed low-pass filter, it can suppress small tracking noise while remaining responsive when movement accelerates. The amount of filtering is adjustable from the installation interface, and filters reset after a longer tracking gap so an old trajectory is not dragged onto a newly detected body.

The renderer also knows how long a pose has been tracked continuously. This lets an effect fade in instead of firing at full strength the instant a noisy detection appears. These details are almost invisible when they work, but they are what make the projected bodies feel coherent rather than computational.

Developing a fluid body, not a fluid cursor

The physics went through its own sequence of experiments. I began with WebGL fluid studies driven by a pointer, then replaced the pointer with wrists, full skeletons and person-segmentation masks. I tested simulation resolution, dye resolution, dissipation, curl, pressure iterations, splat size and different ways of converting body motion into force.

A pose should not merely be drawn on top of a fluid background. The pose itself should be the fluid source.

For the production version I built a custom GPU fluid solver in TypeScript and WGSL. It maintains velocity, dye, pressure and divergence textures and processes them through a sequence of WebGPU render passes. Motion impulses are added first, followed by advection and damping, divergence calculation, iterative pressure solving and velocity projection. The resulting dye field is rendered together with the pose layer.

Every visible limb becomes a line segment with its own position, direction and velocity. Slow motion can produce a soft wash; quick changes inject stronger velocity and color. The head is represented by an organic oval rather than a loose collection of face keypoints. Fast arm and leg movements can create additional drop-like impulses that detach from the skeleton and travel through the field.

This made the image feel connected to the dancer without becoming a literal anatomical diagram. It also created a useful artistic control surface. Physics presets can produce dense floods, broad color-filled limbs, sharp impulses, short snaps or long trails. Separate pose modes can emphasize a full body, forearms, arms and shoulders, or the upper body. Color and canvas effects — such as zoom, shake and blur — remain independent, so the same movement can generate very different scenes.

Making the image dance with the music

Motion provides the shape of the effect; music provides its timing.

Motion Mirror analyses microphone input with aubio.js. Tempo and onset detection produce a smoothed BPM, beat confidence, audio level and individual beat events. A manual BPM and sync control remain available if the live installation can provide a stream of MIDI information.

The beat is treated as a continuous phase rather than a simple on/off trigger. Each beat produces curves for a sharp kick, a snap, a swell and an afterglow. Those curves can momentarily change velocity force, dye force, damping, opacity, impulse strength and advection. The result can hit hard on a kick and then dissolve naturally. This leaves fluid imprints of the crowd on the canvas on every beat. Like movement snapshots.

An eight-beat sequencing system coordinates the larger changes. Color palettes, physics modes, body selections and canvas effects evolve at different musical intervals: canvas accents may change after one or two beats, pose emphasis after several beats, physics after a longer phrase and color after an even longer section.

Controlled randomness keeps the installation from repeating a fixed animation, while the beat grid and debug interface make the sequence understandable and tunable during setup. A big amount of changing display settings was tested, but in the end only a small selection was kept. Some colors and physics just work best, so you want to maximize the amount they are displayed.

Designing for the venue

The last part of the work was turning an R&D prototype into something that could be operated and maintained on site.

The app supports live camera and local video sources, camera mirroring, fullscreen output and independent fluid, skeleton and camera layers. A diagnostic panel reports frame rate, detected people, inference latency, dropped frames, runtime and model information. The pose threshold, smoothing, simulation parameters, color modes and beat controls can all be tuned without rebuilding the application.

For the Lichtung Festival setup I also added simple remote-control key bindings. The camera image can fade briefly into the background to show to the audience what the camera really sees.

Running locally as a static application also reduces the number of systems that can fail during a show. Once the browser, model and assets are present, pose data and camera footage do not need to leave the machine. That is valuable both for reliability and for an installation filming a public crowd.

What the experiments taught me

The most important result of Motion Mirror is not any single model or shader. It is the conjunction between tracking, temporal filtering, physical simulation, musical timing and operational design.

There are a few trade-offs that need to be worked on for each individual project. First, weighing false positive vs. false negative tracking results: what error do you rather have? Then it is weighing smoothing vs. immediacy: do you want to track dance motions on the beat, or should the visualization be an after-image? Then choose the level of abstraction you want. A more directly outlined body shape helps identify the visual metaphor quickly; creating abstract motion physics will increase the engagement people have with it.

A faster model is only useful if its frames arrive without building latency. Stable tracking IDs matter because physics has memory. Smoothing must remove model noise without removing the character of a gesture. Beat detection becomes compelling only when it modulates physical parameters over time. And an expressive visual system still needs clear fallbacks and controls before it is ready for a venue.

The project developed through many small, disposable experiments: person detection and counting, offline landmark tracking, segmentation-driven fluid, wrist emitters, full-body fluid sources, model export tests, skeleton studies, audio-reactive prototypes and effect sequencers. Each one answered a specific question. Together they led to a system in which the crowd is no longer merely observed by the camera — it becomes part of the light show.

Motion Mirror demonstrates a broader use case for browser-native AI and GPU computing. The same architecture could support interactive stages, exhibitions, dance performances, brand installations or participatory public art. The browser is not just the control panel in this setup. It is the complete real-time visual instrument.