Overview / camera to speakers
Five stages, one continuous response.
A gesture-controlled synthesizer is not listening for a list of magic signs. It repeatedly measures where the hands are and how open they appear, then maps those values to musical parameters. Gesture Synth performs that loop many times per second.
The stages are deliberately separated. Camera capture remains on the main page, hand inference runs in a Web Worker, musical mapping filters the raw measurements, and the audio engine receives only compact control values.
Vision / landmarks, not identity
The tracker sees a hand-shaped coordinate system.
Gesture Synth uses the MediaPipe Hand Landmarker model to estimate up to two hands. For each detected hand, it returns 21 three-dimensional landmarks around the wrist, fingers, and palm. The application uses those points as geometry; it does not attempt to identify who the hand belongs to.
Why a Web Worker matters
Machine-vision inference can be expensive. Camera frames are converted to transferable image data and sent to a dedicated Worker, keeping model work away from interface rendering and audio control. If inference becomes slow, the visual quality can step down before musical responsiveness is sacrificed.
How openness is estimated
The app compares finger joint angles and fingertip reach relative to palm size. Four fingers contribute to an openness score. A short vote window and separate open/closed thresholds prevent a finger near the boundary from turning a note rapidly on and off.
Landmark coordinates and handedness confidence, not a recording and not a biometric identity.
Control layer / motion made playable
Raw coordinates need musical restraint.
Direct camera data jitters by a few pixels even when a hand looks still. Gesture Synth applies adaptive One Euro filters to palm X and Y values: slow movement is smoothed strongly, while intentional fast movement is allowed to respond more quickly.
Horizontal position becomes a tuned note
The mirrored X position is quantized into ten zones from D3 to C5. A small amount of hysteresis keeps the current note active near zone boundaries, reducing accidental note chatter. The available notes come from D minor pentatonic.
Vertical position remains continuous
Y position controls a low-pass filter from warm to bright and adds a restrained level change. Unlike pitch, this value is not divided into steps, so lifting a hand produces a continuous timbral motion.
Each palm keeps a voice identity
Detected palms are matched to recent palm positions. That lets two hands keep separate synth voices even when the detector changes its result order between frames. A hand that leaves the frame releases its voice after a short grace period.
- Mirrored X
- Pitch zone
- Ten notes in D minor pentatonic, with boundary hysteresis.
- Inverted Y
- Filter + level
- Higher is brighter and slightly louder; lower is warmer and softer.
- Openness
- Voice gate
- Open sustains the voice; closed releases it after stable votes.
- Palm path
- Voice identity
- Position matching keeps two independently moving voices coherent.
Four voices / one hand map
The sound is synthesized, not sampled.
Gesture Synth ships four finished synthesizer voices on the same D minor pentatonic map. Airglass is the default FM voice: a sine carrier and triangle modulation create a clean, glass-like center. Softpad blooms more slowly for held harmony, Pulsewire uses a brighter square edge for articulated runs, and Velvet favors a short plucked attack for sparse motifs.
Every active hand receives its own synth, filter, volume stage, and stereo panner. The voices then share a restrained feedback delay, reverb, compressor, and final limiter. Switching a voice rebuilds the per-hand synths while leaving the camera mapping unchanged, so musical positions stay familiar.
The audio module is not downloaded until the user starts the instrument. That keeps the initial page light and ensures browser audio begins from an explicit interaction, as required by modern autoplay policies.
Visible cause / audible effect
The screen explains what the ear receives.
A WebGL layer mirrors and grades the live camera texture. A separate two-dimensional effects canvas draws hand bones, fingertip trails, changing-note rings, and energy-reactive light. Keeping the layers separate lets the interface lower visual resolution without changing the hand-to-sound mapping.
If WebGL is unavailable, the camera can fall back to the normal video layer. The visual treatment is part of the performance, but it is not allowed to become a requirement for sound.
Boundary / local by construction
The performance loop never needs a server.
Camera frames move from the video element to the on-device Worker and are discarded as the next frames arrive. Hand landmarks remain in memory only long enough to control the current performance. Audio is generated locally and sent to the device output.
The site has no account, recording pipeline, camera upload endpoint, or cloud inference service. Aggregate site analytics cannot receive camera footage, hand landmarks, or synthesized audio. See the full privacy explanation for the site-data boundary.