Infrastructure · Experiment

node

nothing is ever loading

An intelligence that lives in the building instead of a browser tab. It remembers what I was working on last week, and every layer from the inference engine upward is mine. This is the machine underneath it: one chassis, specified from parts to hold 384 GB of GPU memory, because the idea has exactly one hard requirement. Nothing is ever loading.

Threadripper PRO 9995WX
4 × RTX PRO 6000 Max-Q
512 GB ECC
1,790 W
384 GB

of GPU memory in one enclosure, so every model stays resident at once.

~80%

of requests never reach a generating model. They are matched by embedding, in milliseconds.

$44,958

for phase one: the whole platform and half the GPUs. Nothing is bought yet.

What it is for

Today I have a Claude on the Mac, a model on my phone, an assistant on a rented server and a dozen projects, and none of them know about each other. Every conversation starts from zero. What I want is one intelligence with several bodies, and continuity between them: I ask what happened overnight and it knows the deploy failed and why.

Two tenants share the machine. One is that personal layer. The other is the product work I already run and ration: image, video and agent pipelines that today wait for each other. The work layer comes first, because it needs no sensors and is where the friction is. The apartment layer earns its way in later.

How it works · most of it should never reach the big model

An ambient system fails on latency and unreliability, not intelligence. One that takes four seconds to turn off a lamp will not be used twice. So traffic splits sharply.

  1. Intent match, by embedding
    Embed the utterance, take a dot product against a registry, act. Deterministic, testable and answered in milliseconds.
  2. One validated tool call
    Decoding is masked to a grammar compiled from the real device registry, so a malformed call or a missing device is unreachable, not just unlikely.
  3. The reasoning model wakes
    Long-context work over my own code and notes. The only tier that needs a large model, and it must not evict anything when it runs.

The same grammar handles permission. Each trust tier compiles a different grammar, and the privileged tools are absent from the one the always-on voice path uses: the model cannot emit what it cannot reach. Proactive speech is limited to things with an owner and a deadline, like a failed deploy. Never an observation about me.

Why the hardware has to be like this

If a model has to load before it answers, the system is an app with extra steps. Every tier has to be resident permanently, including the one that runs 5% of the time. Today everything time-shares one card, down to a lease broker written only to arbitrate an image lane and a video lane that cannot coexist. So the machine is specified around a residency budget, not a benchmark.

Resident at the same timeVRAM
Always-on orchestrator, the intent kernel that never unloads16 GB
Speech to text and text to speech, both hot10 GB
Vision model for camera and doorbell events20 GB
Reasoning model, a 120B-class mixture of experts120 GB
Image generation40 GB
Video generation60 GB
Embeddings and reranker for the memory index8 GB
Left over, for training110 GB

Every row is something I have built or am building. Today they queue.

Key decisions

Each was checked against the manufacturer's own specification sheet, and two of them removed parts instead of adding them.

Max-Q, not the 600 W edition

Both carry the same 96 GB of ECC GDDR7. The Max-Q draws 300 W with a density-optimised cooler, so four fit on air in one enclosure. The 600 W card realistically caps the build at two, which halves the ceiling for identical memory per card.

240 V is a part, not an upgrade

The 3000 W supply takes 220 to 240 Vac and will not power on from a 120 V outlet, and no 120 V supply carries 1,800 W continuously. A dedicated circuit is a line item that must exist before the parts arrive.

Onboard M.2, no enterprise U.2

The board's SlimSAS ports are PCIe 4.0, so a 30 TB Gen 5 U.2 drive would run at roughly 40% of its rating, and an adapter needs a slot the GPUs already took. The four onboard M.2 slots are full PCIe 5.0. Dropping the drive made the build cheaper and faster at once.

Two phases, because slots outlive silicon

Buy what can never be retrofitted (lanes, memory channels, slots, power, volume) and add what can. GPU prices are swinging by more than 50% on a memory shortage, so phase one buys the full platform and half the cards. Phase two is two cards into two waiting slots.

ConfigurationVRAMPeak drawCost
Platform only, no GPUs0 GB670 W$26,958
Phase one, two GPUs (runs a 200B model)192 GB1,190 W$44,958
Complete, four GPUs384 GB1,790 W$62,958

Privacy enforced by the network

The compute network is default-deny outbound, with an allowlist. Enforced in software, “my data stays in the building” is a promise; enforced at the gateway it is a property of the network. The management controller sits on an isolated network, cameras get no internet, and the home automation controller runs on a separate small box, so a light switch never depends on the node.

What it can train, and what it cannot

Memory is not the limit: at about 16 bytes per parameter, 384 GB holds roughly 24B parameters of optimiser state. Compute is. At an assumed 35% utilisation the machine does roughly 4 × 1019 FLOPs a day, against a cost of about six times parameters times tokens.

Trained from scratchTokensBF16FP8
0.5 B parameters30 B2 days1 day
1 B parameters100 B15 days8 days
3 B parameters60 B27 days14 days
7 B parameters140 B147 days74 days
70 B parameters, as actually done15 T~430 years—

It does not train a frontier model, and the reason is the token budget: a 1.1B model trained the modern way sees three trillion tokens, about 500 days here. What it is good at is the intent kernel, a narrow 1B model on a corpus generated from my own device registry. A week, not a year. It would run on the inference engine I already wrote, about three thousand lines of C, so the whole column would be mine, from the matmul to the machine.

Limits

  • No NVLink. The four cards talk over PCIe 5.0 at roughly 64 GB/s, against 900 GB/s on a datacentre GPU. Invisible for inference, where weights load once. For training, tensor parallelism suffers badly, so the lane is sharded data parallelism with large gradient accumulation. No money spent elsewhere closes this.
  • One card at half width. Four dual-slot cards need eight positions on a seven-slot board, and one slot is x8. With four cards in, no expansion slots are left.
  • The room is part of the spec. A 1,790 W peak is about 1,989 W at the wall, 8.3 A on a 240 V, 20 A circuit. It is also about 6,100 BTU per hour of heat, more than a small window air conditioner removes. The UPS buys seconds for an orderly shutdown, not a way through an outage.

Nothing is bought. What exists is a specification: every line a part number, two purchase phases, the power ledger, the slot map, and every rejected part with its reason. The hardware buys custody and headroom, not validation; the whole hierarchy can be prototyped on servers I already run.

Part numbers, voltages, slot count and PCIe generations are checked against manufacturer documentation. The GPU price is a moving target. The 35% utilisation behind the training estimates is an assumption, and the first thing this machine would tell me is whether it was right.