The shape of a voice

Kwami is a 3D companion you talk to. Its library ships five bodies, and the default one is a sphere whose surface moves as it speaks. None of that motion is keyframed. On every frame, the browser reads the agent’s voice, splits it into frequency bands and moves every vertex of the sphere from those numbers. This is how Kwami does it, from sound to surface.

From a voice to 1,024 numbers

The agent’s voice arrives as a LiveKit audio track. The library plays it through an audio element and hands the same track to a Web Audio analyser. The analyser takes the last 2,048 samples, applies a Blackman window, runs a fast Fourier transform and keeps 1,024 bins. A bin is the strength of one narrow slice of the spectrum, and its width is the sample rate divided by 2,048. In an audio context running at 48 kHz, a common rate, each bin is about 23 Hz wide, and together they reach 24 kHz, half the sample rate. Each strength is read in decibels, clipped between −90 and −10, and mapped to a byte from 0 to 255.

The bins are cut into four bands by position: the first 10%, up to 40%, up to 70%, and the rest. The cut is linear in frequency, while the ear hears in octaves, so at 48 kHz the low band runs from 0 to 2.4 kHz and holds both the pitch of a speaking voice and its first formant. Each band is the mean of its bytes, divided by 255. That is not an RMS of the waveform: the bytes are already decibels, so a band is an average of levels in decibels. The overall level weighs the four bands 0.4, 0.35, 0.2 and 0.05, from low to high.

const lowEnd = Math.floor(dataLength * 0.1)
const midEnd = Math.floor(dataLength * 0.4)
const highEnd = Math.floor(dataLength * 0.7)
let lowSum = 0, midSum = 0, highSum = 0, ultraSum = 0

for (let i = 0; i < dataLength; i++) {
  const value = (frequencyData[i] ?? 0) / 255
  if (i < lowEnd) lowSum += value
  else if (i < midEnd) midSum += value
  else if (i < highEnd) highSum += value
  else ultraSum += value
}

Smooth, but not too smooth

Raw spectra flicker. The analyser already smooths each bin: every new reading is 35% the previous one and 65% the fresh one. Then the low, mid and high bands, and the overall level, each run through an exponential moving average of their own, once per frame:

s ← s + α · (x − s)

Here x is the band’s new value and s what the body remembers of it. It is the simplest low-pass filter there is: a reading from n frames ago still weighs α·(1 − α)ⁿ, so the past fades geometrically and a single number holds all of it. A response-speed setting puts α between 0.12 and 0.47: at 60 frames a second, the past then halves in about 90 milliseconds at the slow end and 18 at the fast end. Smoothing alone would blunt the start of every syllable, so part of the raw band is mixed back in, from 15% to 65% with a transient-boost setting. The body reacts only when that mixed level crosses three quarters of the sensitivity setting. Below it, Kwami breathes.

Louder means more spikes

The body is a unit sphere of 180 by 180 segments by default, just under 33,000 vertices. No vertex shader moves them. The shape is computed on the CPU, written back and given fresh normals on every frame, at three to seven noise samples per vertex. Each vertex keeps the direction it had the first time the mesh was animated, and every frame starts again from those directions. An earlier version read the vertices back from the mesh instead, and a few seconds of drift baked a cone into the body.

Along that direction, the radius comes from simplex noise, Ken Perlin’s gradient noise on a grid of tetrahedra. Each sample blends the gradients at four corners instead of a cube’s eight, and returns a smooth value just inside −1 to 1. We sample it in 3D at the vertex’s direction, not in 2D over the sphere’s UV map, so the displacement has no seam and nothing pinches at the poles. The field is built once per page from a shuffled table and never changes. Time is added to the coordinates, so the sphere slides through it. At rest, two slow layers of that noise morph the surface. When the voice is loud enough, a push built from the low, mid and high bands, capped at 3, raises both the height and the frequency of a third layer:

const audioFreqBoost = 1 + audioPush * spikeDensity
…
const spikeNoise = noise3D(
  direction.x * baseFreqX * audioFreqBoost + tX * 1.3,
  direction.y * baseFreqY * audioFreqBoost + tY * 1.3,
  direction.z * baseFreqZ * audioFreqBoost + tZ * 1.3,
)
…
const audioAmp = audioPush * 0.25 * amplitudeMultiplier
…
const speakingDisplacement = idleAmp * idleNoise
  + audioAmp * spikeNoise
  + audioDetail * detailNoise

Louder does not only mean taller spikes. It means more of them. Multiplying the coordinates by audioFreqBoost makes the sphere sample a wider stretch of the field, so more of its bumps fit on the same surface. Past a push of 0.2, a fourth layer at twice that frequency fades in for fine detail. The sum is clamped between 0.55 and 1.45 times the rest radius, then smoothed per vertex by the same filter, with α from 0.3 to 0.7 while the voice is active and 0.25 otherwise. A filter that only mixes values inside that range can never leave it. Each vertex’s amplitude, finally, weighs the three axis amplitudes by |x|, |y| and |z| of its direction. On a unit vector, |x| + |y| + |z| runs from 1 to √3, so with equal amplitudes the diagonals move up to 73% more than the points on the axes.

Louder does not only mean taller spikes. It means more of them.

Listening, thinking, breathing

The voice agent publishes its state, and the body blends three fields by it. Speaking, like idle, uses the one above. Listening turns the idle relief inside out at 70% strength, so bumps become hollows, and keeps 40% of the audio spikes. Thinking, which lasts at most ten seconds by default, stirs three faster layers, two of which drift as it goes on. The whole pulses, and fades as 1 − p², where p is the share of that time gone. The blends ramp linearly and are scaled by the real time between frames, so by default a change takes a third of a second, at 60 Hz as at 120. In silence, the body breathes: its scale rises and falls once every 3.5 seconds, by about half a percent with the default setting.

Colour is the fragment shader’s job. The default skin, one of 22, weighs three colours by the angle θ around the body’s z axis, each by 0.5 + 0.5·cos(θ − 2πk/3) for k = 0, 1, 2. Three cosines 120° apart always sum to zero, so the weights always add up to 1.5, and dividing by their sum only rescales them. The voice moves each vertex along its own direction, which leaves θ as it was: the colours ride the surface instead of sliding over it. A Phong highlight on top gives the body its shine.