Open Source · C
mote
A tiny language model, run from scratch in C. A complete transformer inference engine in a few hundred lines, with no libraries doing the work: no PyTorch, no NumPy, no BLAS, no framework of any kind. It loads a pretrained model and generates text, and the whole point is to shrink it until it runs somewhere it has no business running, a chip instead of a computer.
What it is
Most people run a language model by importing a giant library and calling one function. mote is the opposite: it is the function, opened up. It takes a model that has already been trained, a set of numbers, and does the actual arithmetic that turns a prompt into the next word, then the next, one token at a time. Every part of that is written out by hand so you can read it end to end in an afternoon.
It does not train anything and it does not try to be the fastest engine in the world. It is built to be legible, and to be small, small enough that the same code can eventually run on a microcontroller with a few megabytes of memory instead of a laptop.
See it run
Point it at a small pretrained model, give it the start of a sentence, and it writes the rest. This is the real output, generated by the from-scratch engine at about a hundred tokens a second on a laptop, single-threaded.
$ ./mote stories15M.bin -i "Once upon a time, there was a little robot"
Once upon a time, there was a little robot. The robot liked to crawl. One day, the robot went to the park. He saw a big tree. The robot wanted to crawl to the tree. He went up, up, up the tree. He saw a bird singing, and stopped to listen. Then the robot saw a big hill. He was scared, but he was brave, and he crawled to the top. There was a party waiting for him. The robot and the bird were friends. They played and had fun.
[158 tokens, 109.4 tok/s]
The model that fits is small, so it tells simple stories rather than holding a conversation. That limit is the whole charm of the project.
On a phone
The same engine runs inside a native iOS app, with the model living in the app itself. No server, no network, no API. Put the phone in airplane mode and it still answers, because there is nothing to phone home to.
The app is a small playground: type a prompt, watch it continue your text, and read the real tokens-per-second off the top of the screen. Same C engine as everywhere else, called from Swift.
How it works
A word comes in as a token, which is looked up as a row of numbers, its embedding. That vector then passes through a stack of identical blocks. Each block does two things: it lets the token look back at everything before it and pull in what is relevant (attention), and then it thinks on its own for a moment (a small feed-forward network). After the last block, the engine reads out a score for every possible next word and picks one. Feed that word back in and the loop repeats.
The details are the parts people usually let a library hide: RMSNorm to keep the numbers well-behaved, rotary embeddings so the model knows the order of the words, grouped-query attention with a cache so each new token reuses its history instead of recomputing it, and a SwiGLU feed-forward. In mote they are all just a few dozen lines of C each. There is exactly one loop that costs anything, the matrix multiply, and it is the only place the code reaches for extra threads.
The weights are memory-mapped straight off disk rather than loaded into a buffer, so the model appears instantly and the operating system pages in only what each step touches. It is the kind of detail that stops mattering on a laptop and starts mattering a lot on a chip.
Fast, and small
That one hot loop, the matrix multiply, is where all the speed lives, so it is the one place the code gets clever: hand-written SIMD and a spread across cores. On the same laptop that made the from-scratch engine several times faster, and because it uses the instructions a phone also has, the gain carries onto the device.
Tokens per second
Model size on disk, MB
Measured on one laptop, so the on-device numbers differ, but the shape holds: quantizing shrinks the model about four times, and the vectorized, threaded matmul multiplies the speed.
How small can it go
mote is being built in three steps, from the general to the almost absurd.
Step one · done
Inference in pure C
The full forward pass with no dependencies, generating coherent text. This is what runs above.
Step two · done
Quantize it
The weights pack down to 8-bit integers, with the compressed arithmetic written by hand: about four times smaller on disk and faster to run, with the output quality held. Going below eight bits is the part still open.
Step three · underway
Onto a microcontroller
The engine is already chip-shaped: it loads its weights from a plain array with no filesystem, and a small model fits a $6 microcontroller's memory. What is left is the board itself, and the real tokens per second and memory measured on it, no hand-waving.
The name
A mote is the smallest thing you can still see, a speck of dust caught in a beam of light. That is the target: a language model shrunk until it is barely there, and still works.
Commit history · live
from githubloading commits…