sotto
it lets you cut in
Hear, think, speak, and survive being interrupted. The conversational layer underneath everything else I run, pulled out of a hypnotherapy app because the loop was the reusable part, and the problems that only appear once a voice has to be listened to rather than read.
to first audio on a conversational turn, against 6 to 7 s to wait for the full reply.
speed in every register of the pace ladder. Silence does the entire job.
voice-service kills before anyone noticed. Now the health endpoint counts them.
How it works
- Open microphonefaster-whisper held resident, and a small neural voice detector on 32 ms frames for the whole session.
- Self-hosted 30BStreamed over an OpenAI-compatible endpoint. The hosted lane was deleted, not flagged off.
- Sentence by sentenceAn 82M voice model on CPU renders each sentence as soon as it exists, delivered in order.
- Stop when cut offPlayback stops the moment the person speaks, and the record keeps what they actually heard.
Two services and a client library. Everything domain-specific sits behind an interface of seven properties and seven methods (voice, pace, what to say next, whether a reply is expected, screening, ending), so the therapy app and a plain assistant share the same loop and neither reimplements it.
Key decisions
Slow the silence, not the words
A text-to-speech speed control stretches vowels and drags consonants; the voice does not get calmer, it gets damaged. A model reading a sentence already pauses in the right places, so the sentence is rendered once at its natural rate and only its existing silences are lengthened, each in proportion to how long it already was.
Punctuation is the lever: a comma buys about 0.28 seconds of natural pause, an ellipsis 0.34, a full stop 0.38, and a spaced dash none at all.
Never stretch inside a word
A stop consonant, like the p in open, is 40 to 120 ms of real silence. The first version stretched one by 3.1 seconds, mid-word. Now nothing under 0.20 s is stretched, no mid-sentence pause exceeds 1.5 s, and silence that cannot be spent is held as quiet after the line.
Speak before the thinking is finished
The stream is cut into sentences as they complete (a splitter that knows Dr. and e.g. are not endings) and audio starts on the first. When that still misses its budget, a pre-screened holding line, like mm, let me think about that, plays while the answer arrives.
| Measured on the deployed box | Time |
|---|---|
| Brain, first token, warm | 0.9s |
| Voice, one sentence | 0.95 to 1.8s |
| A conversational turn, to first audio | 3.2 to 4.6s |
| The same reply, waited for in full | 6 to 7s |
Keep reading after an interruption
Abandoning the stream once nobody is listening silently destroyed the first live session: the server could not tell which sentences were heard, so the transcript lost the line and marked the wrong one as cut. Now the audio stops but the stream is read to the end, so the model is never shown a paragraph the person did not hear.
Queue, don't shed
The voice service refused sentences under load because nothing bounded concurrent renders, each wanting about a gigabyte. The fix is two slots that queue: a sentence that waits is late, a sentence that is refused is never spoken.
Apparent leaks were the allocator holding arenas, so it now releases after each render. It lasted because the process restarted cleanly and nothing reported it; the health endpoint now counts refusals, queued renders and restarts.
Same shape: the tensor library read the host's core count, not the container's limit. Matching them took one ten-word sentence from 6.7 seconds to 1.6.
Limits
It serves only its first consumer, and the HTTP surface has barely been called from another machine. Conversations live in memory, so it runs as a single worker, and authentication is one shared key. Screening is a substring match over a short list, explicitly an engineering scaffold and not validated clinical criteria. The fifteen tests cover the loop with fakes and none of the voice service, which is the part that has broken in production.
The open question cannot be measured: whether this is a voice anybody would want to be spoken to by is a listening decision, and it has not been made yet.
Timings come from the deployed box and from the farm over a tunnel, recorded in comments and commit messages when taken rather than by a re-runnable benchmark, which is a gap worth naming.