A tiny language model with a huge lookup table
Short answer: yes, the table is worth it. And the more the model trains, the more it's worth.
The two-minute version · by Leon Fechner · last updated 18 October 2026
I took a small language model (21M parameters) and gave it a table with up to 16.8 million learned vectors. For every token (roughly a word) it only looks up a few hundred of them, so it barely computes more than before. Then I trained normal models of different sizes on exactly the same data, to see how big a normal model has to be to keep up.
Newest result: six times the training
The obvious objection: with more training, the normal models catch up and the table becomes pointless. So I trained everything again on 3 billion instead of 500 million tokens. I wrote down the rule for "grows", "holds" and "shrinks" before the runs, and my own guess was "holds" or "shrinks".
It grew, for all three table sizes.
What else works
- The table doesn't have to sit on the GPU. From normal RAM or even an NVMe SSD the model still writes 114 to 154 tokens per second on my PC.
- It trains on a gaming card. The 16.8M table trained on my 16 GB Radeon RX 9070, with the table in RAM, and came out as good as on a big H200. It just took 7.7 times as long.
- It helps to remember facts. Facts about made-up people, seen only a few times in training, stuck clearly better with the table than in a normal model with twice the compute.
- Same code on AMD and NVIDIA. The kernels run unchanged on a Radeon RX 9070, an AMD Instinct MI350X and NVIDIA H100/H200.
The catch
- Training takes longer. On an H200 a normal model gets to the same quality in 19 to 27% less time. Looking things up in the table isn't free.
- Not faster in use yet. On an H200 it doesn't write text faster than a normal model of the same quality. It computes much less, but at this small size computing isn't the bottleneck.
- It mostly helps on text like its training data. On a differently formatted text set (WikiText-103) the advantage is much smaller.
- Still small. The biggest normal model I compared against has 400M parameters, and every model was trained once.
- Bolting a table onto a finished model didn't work (tried on Qwen3.5-0.8B).
What's next
The real question is whether this holds at a size people actually use: a model around 1 billion parameters with a table of tens of millions of rows. That needs bigger GPUs than I can rent. If you have some to spare, AMD Instinct or anything else, I'd love to hear from you: fechner.leon [at] protonmail.com, or open an issue.