sotto

it lets you cut in

Hear, think, speak, and survive being interrupted. The conversational layer underneath everything else I run, pulled out of a hypnotherapy app because the loop was the reusable part, and the problems that only appear once a voice has to be listened to rather than read.

Python
FastAPI
Kokoro TTS
Self-hosted
3.2–4.6 s

to first audio on a conversational turn, against 6 to 7 s to wait for the full reply.

1.00

speed in every register of the pace ladder. Silence does the entire job.

22

voice-service kills before anyone noticed. Now the health endpoint counts them.

How it works

  1. Open microphone
    faster-whisper held resident, and a small neural voice detector on 32 ms frames for the whole session.
  2. Self-hosted 30B
    Streamed over an OpenAI-compatible endpoint. The hosted lane was deleted, not flagged off.
  3. Sentence by sentence
    An 82M voice model on CPU renders each sentence as soon as it exists, delivered in order.
  4. Stop when cut off
    Playback stops the moment the person speaks, and the record keeps what they actually heard.

Two services and a client library. Everything domain-specific sits behind an interface of seven properties and seven methods (voice, pace, what to say next, whether a reply is expected, screening, ending), so the therapy app and a plain assistant share the same loop and neither reimplements it.

Key decisions

Slow the silence, not the words

A text-to-speech speed control stretches vowels and drags consonants; the voice does not get calmer, it gets damaged. A model reading a sentence already pauses in the right places, so the sentence is rendered once at its natural rate and only its existing silences are lengthened, each in proportion to how long it already was.

Key decision · tempo
Slower speech, reached by lengthening only the silences
speech, never scaled
silence the model made
owed after the line
one sentence, five registersspeed 1.00 throughout
target
2.33
achieved
2.33
length
6.0s
Drag it down the ladder. The grey never changes width, because the words are rendered once. By the deep register the pauses hit the 1.5 second ceiling, and the rest of the silence is handed back as quiet after the line. Each register lands on its words per second at speed 1.00.

Punctuation is the lever: a comma buys about 0.28 seconds of natural pause, an ellipsis 0.34, a full stop 0.38, and a spaced dash none at all.

Never stretch inside a word

A stop consonant, like the p in open, is 40 to 120 ms of real silence. The first version stretched one by 3.1 seconds, mid-word. Now nothing under 0.20 s is stretched, no mid-sentence pause exceeds 1.5 s, and silence that cannot be spent is held as quiet after the line.

Speak before the thinking is finished

The stream is cut into sentences as they complete (a splitter that knows Dr. and e.g. are not endings) and audio starts on the first. When that still misses its budget, a pre-screened holding line, like mm, let me think about that, plays while the answer arrives.

Measured on the deployed boxTime
Brain, first token, warm0.9s
Voice, one sentence0.95 to 1.8s
A conversational turn, to first audio3.2 to 4.6s
The same reply, waited for in full6 to 7s

Keep reading after an interruption

Abandoning the stream once nobody is listening silently destroyed the first live session: the server could not tell which sentences were heard, so the transcript lost the line and marked the wrong one as cut. Now the audio stops but the stream is read to the end, so the model is never shown a paragraph the person did not hear.

Queue, don't shed

The voice service refused sentences under load because nothing bounded concurrent renders, each wanting about a gigabyte. The fix is two slots that queue: a sentence that waits is late, a sentence that is refused is never spoken.

Key decision · concurrency
Nine renders at once hit the 4 GiB cap, and none of the nine were delivered
peak memory, against a 4 GiB capsame nine sentences
9 sequential9 of 9 delivered
2.84 GiB
6 concurrent6 of 6 delivered
3.60 GiB
9 concurrent0 of 9 delivered, process killed
4.00 GiB
22 cgroup kills observed before anything reported it
A synchronous render route ran in a forty-thread pool with nothing bounding how many renders were in flight. In one three-turn conversation, five of nine sentences came back as errors. The fix is two render slots that queue rather than shed.

Apparent leaks were the allocator holding arenas, so it now releases after each render. It lasted because the process restarted cleanly and nothing reported it; the health endpoint now counts refusals, queued renders and restarts.

Same shape: the tensor library read the host's core count, not the container's limit. Matching them took one ten-word sentence from 6.7 seconds to 1.6.

Limits

It serves only its first consumer, and the HTTP surface has barely been called from another machine. Conversations live in memory, so it runs as a single worker, and authentication is one shared key. Screening is a substring match over a short list, explicitly an engineering scaffold and not validated clinical criteria. The fifteen tests cover the loop with fakes and none of the voice service, which is the part that has broken in production.

The open question cannot be measured: whether this is a voice anybody would want to be spoken to by is a listening decision, and it has not been made yet.

Timings come from the deployed box and from the farm over a tunnel, recorded in comments and commit messages when taken rather than by a re-runnable benchmark, which is a gap worth naming.