Case Study · Contract engagement
I built this end to end, under contract, as the sole engineer, from CUDA-level model serving up to the web app the client operates: an orchestrator agent over a roster of specialists, a self-extending skills-and-tools layer with real-world integrations, self-hosted LLM serving, and a generative image and video fleet, all running on the client's own hardware.
The platform and its brand belong to the client. The engineering described here is mine.
The platform is a multi-agent system, not a single chatbot. One orchestrator agent talks to the operator, decides the shape of each request (a question, a platform change, engineering work, research, or a multi-step project), and routes it to the right specialist. Everything runs on the client's own boxes.
Orchestrator
The conversational front and router. Classifies intent, delegates to specialists, enforces governance and honesty rules, and unifies delivery back to whichever surface asked (web or messaging).
Engineering agent
A software-development specialist with its own sandboxed workspace: writes, runs, tests, and self-repairs code, then ships, a build → test → fix → deploy loop that streams progress to the dashboard.
Memory agent
A governed institutional memory: a deduped, versioned, semantically-searchable store of facts, decisions, and preferences that persist across chats, with recall and write tools the other agents call.
Generation + end-user agents
The image/video engine (below), plus end-user assistant surfaces, voice and text, on iOS and web, each with its own scoped identity and toolset over the same backend.
The agents share a library of ~100 skills, tools they can call to actually do things: send email, manage a calendar, run research, generate media, kick off a project. Two engineering problems dominated here.
Tool selection at scale. A self-hosted 14B model degrades badly past ~25 tool schemas in context, it starts picking the wrong tool or hallucinating arguments. So tools are loaded dynamically per turn: a routing layer surfaces only the handful relevant to the message, with signal-gated groups that unlock sensitive tools (self-modification, approvals) only on an explicit keyword match. I built a routing eval harness to measure this and drive it up from a 58% baseline.
Self-extension. When the platform lacks a capability, the orchestrator can call a build_tool skill that writes, deploys, and live-grants a brand-new tool mid-conversation, so the new capability is callable on the very next model turn without a restart.
Real-world reach comes through Composio (Gmail, Calendar, Slack and more) with per-operator connections, plus first-party surfaces for live web research (a SearXNG backend), one-shot deep research delivered asynchronously past the request timeout, recurring monitoring paths, and autonomous multi-step "missions" that plan and execute a whole project and report back.
The reasoning runs on a self-hosted 14B (Ollama) rather than a hosted API, so tool-use reliability was a serving problem, not a prompting one. I run it at num_ctx 16384 with a q8_0 KV cache and flash attention to fit the window on the card, and call it with stream:false when tools are in play. The biggest correctness fix was sampling: Ollama's default repeat_penalty 1.1 quietly corrupts JSON tool arguments, so tool turns are pinned to repeat_penalty 1.0, temp ≤ 0.3, top_p 0.8, top_k 20. A 31-case routing eval keeps regressions visible, and the roadmap is QLoRA on the platform's own tool-call traces to push selection accuracy further.
Serving is load-balanced: an nginx layer round-robins the LLM lane across GPU upstreams on different boxes with backup routing, so a single box dropping degrades gracefully instead of taking the operator lane down.
One agent is a self-hosted image and video engine, and it is where the hardest GPU work lives. A single 12GB card (Blackwell sm_120) has to host a 12B FLUX.1 diffusion transformer, a second 12B editing transformer, a 7B vision model, and a Wan 2.2 14B video pipeline, none of which co-reside. I built a VRAM arbiter around a reentrant GPU lock that evicts and hands the card off cleanly, so contended jobs wait or self-heal instead of OOM-ing.
Highlights: moving to Nunchaku NVFP4 4-bit quantization (nf4 is broken on sm_120) cut FLUX from ~11GB to ~7GB, improved quality, and took a render from ~55–70s to ~5s; fixing an enable_model_cpu_offload host-RAM leak with malloc_trim(0) stopped kernel OOM-kills; video runs through headless ComfyUI (Wan Q4_K_M dual-expert + a 4-step Lightning LoRA, ~1–3 min/clip) with audio generated off-GPU on another box. Caption edits route to a deterministic compositor, not diffusion, since an 8× VAE downsample shreds small text.
Three-box mesh
A render box, a language box, and a Dockerized backend, joined over a Tailscale mesh and coordinated so heavy work lands where the hardware for it lives.
Self-healing services
systemd user units with linger + warm-preload; recoverable GPU errors rebuild the pipeline; three consecutive OOMs trigger a clean os._exit(3) so systemd restarts fresh.
Unified delivery
Long-running work (missions, research, renders) started from any surface reports back to the originating chat, so a job kicked off on the web dashboard doesn't vanish because it finished 20 minutes later.
Wait, don't fail
The governing rule: slow is acceptable, a silent failure is not. Every contended path blocks on an arbiter or self-heals rather than returning a bare error a retry would have fixed.
A working multi-agent platform, orchestration, memory, autonomous engineering, research, real-world integrations, and generative media, running end to end on the client's own hardware with no third-party model API in the critical path. The through-line across all of it was the same: take capabilities built for large managed infrastructure and make them behave like one dependable product on self-hosted, undersized machines.