How finger gestures become music.

From camera to speakers

The browser handles five separate steps.

Camera capture, hand tracking, gesture confirmation, music mapping, audio, and rendering are handled separately. Desktops request landscape or portrait video that matches the viewport. Phones prefer a wider 4:3 camera view and show the full frame so the wrist and fingertips are less likely to be cropped. On a busy device, tracking can use a smaller image while keeping the same finger-count rules.

Finger counting

Each finger is measured against the hand itself.

MediaPipe Hand Landmarker estimates 21 three-dimensional points around the wrist, palm, and fingers for up to two hands. Gesture Synth does not use the recognizer's small built-in pose list. It evaluates each long finger from two joint angles, whether the whole finger is straight, and whether its tip genuinely moves away from the wrist.

The thumb uses a separate local-palm test because it opens sideways. It must be straight enough, reach beyond the index side of the palm, and stay clear of the folded-thumb region. The final number is normally the total of all extended fingers: thumb-only is 1, four long fingers with a folded thumb is 4, and a full open palm is 5. One deliberate exception makes the common OK sign playable: when thumb and index form a tight loop and the other three fingers are extended, the controller reads 3 rather than 4 or 5.

Why transitions do not become accidental notes

Changing from two fingers to five briefly creates shapes that look like three or four. A clear count is accepted after roughly 80 to 90 milliseconds. A fist must stay closed for about 200 milliseconds before it releases the sound, which prevents a passing shape from stopping a phrase. Palm movement is smoothed before it controls tone and stereo position.

Music mapping

Five counts carry fixed musical meanings.

Melody mode maps 1–5 to scale degrees 1–5 in a selected major key and octave. Chord mode maps the same shapes to I, vi, IV, V, and ii triads. The user can move anywhere in the frame without gliding through unwanted pitches because position does not choose pitch.

Confirmed count

Note or chord

One to five selects a fixed note or chord in the current key.

Hand height

Brightness and level

Raising the hand opens the filter and slightly increases volume.

Horizontal position

Stereo position

Moving a hand left or right moves that voice in the stereo field.

Bottom of the frame

Release

Lowering the hand stops the voice and lets the same count trigger again.

Sound engine

The audio is synthesized live in the browser.

Airglass layers FM and sine bodies; Softpad uses slower amplitude modulation; Pulsewire uses a narrow pulse oscillator; Velvet uses a short pluck. Bellveil adds a tuned FM bell, Reedline sustains an organ-like reed body, and Stringloom builds a softer bowed ensemble. Chord mode creates up to three independently filtered tones per hand.

All voices pass through the adjustable Motion, Echo, and Space effects, followed by EQ, compression, gain, and a final limiter. The limiter controls peaks for device speakers and recorded clips.

Song Quest assigns every chart a recommended voice and a matching effects profile. Fast phrases reduce reverb and echo; lyrical and sustained phrases use smoother motion and longer bodies. The complete comparison is in the seven synth voices guide.

Song Quest

The gesture cue and the melody note are stored separately.

Each Song Quest step stores a finger number, melody pitch, start time, and note length. Ranked Quest waits for a stable tracked hand before it starts the count-in. That hand controls the song; other visible hands stay silent. If the hand disappears or inference stalls, the game clock pauses instead of producing hidden misses, then resumes after tracking returns.

Practice waits after the count-in until the correct gesture is available, and it does not save a score. Quest keeps moving through the song. A fresh gesture inside the timing window earns PERFECT or GOOD, plays the note, and increases the combo. A wrong or late gesture produces a visible MISS, identifies the missed and next targets, and resets the combo. Holding the same count across a repeated note produces HOLD ASSIST audio without adding ranked score.

Quest results and best scores are stored in the browser. A run needs at least 70% accuracy and a nonzero score before it can create a local result card and challenge link. A challenge link contains the song and target result needed to open the invitation, but it does not start the camera or audio until the recipient accepts.

For the player-facing loop, read the Song Quest guide and Practice walkthrough. The gesture recognition guide explains how the camera distinguishes each hand shape.

Local recording

The browser combines the stage and synth audio into a video.

A hidden 720×1280 portrait or 1280×720 landscape canvas crops the WebGL camera treatment and effects layer into one frame, then adds the live finger label and optional score HUD. The final limited audio bus is copied to a Web Audio media destination.

MediaRecorder combines those tracks at 30 frames per second. The browser chooses MP4 or WebM according to its encoder support. The current result remains available for immediate download even if local storage is unavailable. When storage succeeds, My Works keeps up to 12 recordings within an approximate 350 MB browser budget. Gesture Synth does not upload the video.

Read the recording and My Works guide for duration, aspect-ratio, storage, download, and sharing behavior.