Talking to my assistant by voice · Lab

A browser voice client that passes spoken requests to my AI assistant and keeps talking while the work runs; it worked end to end, and I paused it in late September 2026 with two problems open.

What it is

FIG. 1One spoken request. The task runs in the background, so the conversation does not stop while the assistant works; the result comes back through the voice.
Source: voice-relay README and design note, 30 Jul 2026

I talk to a browser page; a live voice model listens, answers small things itself, and hands anything that needs my notes or tools to my AI assistant as a background task. The design note named the problem: the assistant only helped on the days I opened a terminal.

There is no transcript on screen. A cloud of particles follows the audio, scattering in silence and gathering when someone speaks, in one colour for me, another for the voice and a third while the assistant reads my notes.

What happened

The design depended on one question, so I tested it first: does the voice model keep listening and talking while a slow tool call is still running? It did, and the relay worked end to end from the end of July. In September I moved its backend to the assistant system I run now, with live progress from its tools and a queue for several tasks.

A review of one night's session found two problems I never fixed. Speech to text invented fragments in other languages, and a queue bug made the voice read out stale progress lines. I paused it in late September.

What I learned

Two things I believed about my own setup were wrong. My design note said the newer voice model had dropped background tool calls; it had not. And my permission callback turned out not to be a sandbox: allow rules in a settings file silently overrode it, so I wrote a check that proves the block on writes holds.