How Gesture Synth counts your fingers.

The five counts

Show a number. Keep the shape stable.

Gesture Synth uses the camera to estimate the wrist, palm, and finger landmarks of a visible hand. It evaluates whether each finger is extended relative to that hand, then combines the accepted extensions into a count from one to five. The easiest route is index for 1, add middle for 2, ring for 3, and pinky for 4; extend the thumb beyond the index side for 5.

A thumb by itself can also read as 1. In Free Play, each confirmed count selects a fixed note or chord. In Song Quest, the number tells the game which hand shape you showed; the song chart supplies the melody pitch and timing.

Optional OK sign

An OK sign can intentionally mean 3.

Touch the tip of the thumb to the tip of the index finger to make a compact loop, then extend the middle, ring, and pinky. When the loop is visibly closed and those three fingers are clearly raised, Gesture Synth reads the shape as 3 instead of treating the loop as a straight index or a literal four-finger count.

Keep the loop and all three raised fingertips inside the frame until confirmation completes. If the tips separate, the index looks straight, or the raised fingers are hidden, the controller returns to its ordinary geometric count. This shortcut is optional; the simple three-finger shape remains supported.

Confirmation

Confirmation separates movement from intent.

Changing from one count to another briefly creates intermediate silhouettes. The recognizer waits for a candidate to remain stable for a short interval before committing it. The screen shows CONFIRMING while the shape settles and TRIGGERED when the instrument accepts it.

CONFIRMING

Candidate shape

Keep the hand still and let the bar complete. Moving again can restart the candidate.

TRIGGERED

Accepted gesture

The count has crossed the stability gate and can make sound or advance a chart.

Rest

Release and re-arm

Lower the hand to the bottom edge or hold a deliberate fist when you want silence.

Camera technique

Frame the whole hand, not just the fingertips.

1. Use front lightSoft light facing the palm separates fingers from the background.
2. Step backStay about an arm's length away so wrist, palm, and fingertips fit together.
3. Separate raised fingersSmall gaps make the extension geometry easier to read.
4. Keep the palm visibleAvoid crossing hands, cropping the wrist, or placing a bright window behind you.

For a full setup checklist, see browser and device support or troubleshooting.

Local processing

Camera processing stays on your device.

Hand tracking runs locally in the browser. The instrument uses temporary landmark coordinates, gesture states, and hand position to make music; it does not identify you. Camera permission is requested for video only, and recordings remain on the device unless you explicitly share or download one.