openTPU: An Agent-Built LLM Accelerator on a Kintex-7 PCIe Card, and What Its Numbers Teach

Every few months someone posts a “from scratch” AI accelerator, and most of them stop at a simulator screenshot. openTPU, from Felipe Sens Bonetto (GitHub FeSens), doesn’t. It’s a complete inference stack in one Apache 2.0 repo: SystemVerilog RTL, an instruction set, a bit-exact Python simulator, a small kernel language with a compiler, a browser profiler, and host tools that drive a real PCIe card. The card produces the same tokens as the simulator, bit for bit, on models people actually run: Qwen3, Qwen3.5, LFM2.5, Gemma 4, SmolLM3 and Phi-4-mini.

The second claim is the one that got it passed around this week. The README calls it “an open-source AI accelerator, developed by AI,” built with the same agent loop the author used earlier on RISC-V cores. I’ll get to that. The part I keep coming back to is how plainly the numbers show where LLM decode time goes.

One naming note first: this has nothing to do with Google’s TPU. It’s an independent project that borrowed the name.

The hardware

The target is an Inspur YPCB-00338, a PCIe card built around a Xilinx Kintex-7 xc7k480t FPGA with two DDR3-1066 channels. Per the README, that gives 17.1 GB/s of peak memory bandwidth and 4 GiB of on-card DRAM. The production bitstream runs at 133.33 MHz. That clock isn’t arbitrary: it’s where the design’s 128-byte memory port matches what the two DDR3 channels can deliver.

The machine is deliberately simple. A sequencer issues one instruction per cycle to a handful of units:

• a DMA engine that moves data,
• a matrix unit (four-column systolic in the current image) that multiplies int8 weights streamed from DRAM,
• an fp32 vector unit,
• a quantizer that turns results back into int8.

There’s no cache and no hidden scheduling. Every data movement is an instruction, each instruction is eight 32-bit words (ISA doc), and the Python simulator is treated as the spec. Kernels are written in a Python-embedded language (@ol.jit) and compiled to that ISA.

The numbers, and why they look the way they do

The README publishes a table measured on the card between September 29 and October 1, using tools/qual/perf.py: 64 greedy decode tokens after a 512-token prompt, with the host’s argmax in the loop. A few rows, device-side decode:

Nobody will serve production traffic at 6 tokens a second. The useful column is the last one: across every model and format, the card reads DRAM at 82 to 94% of peak while it decodes. That’s memory-bound decode with nothing hiding it.

You can do the arithmetic yourself. Divide bandwidth by token rate and you get the bytes the card reads per token. For Qwen3-0.6B in int8, 14.4 GB/s over 21.6 tok/s is about 0.67 GB per token, roughly one full pass over the weights. Switch to 4-bit and it’s 13.9 over 31.3, about 0.44 GB per token. That’s the README’s “about a third” fewer bytes, and the speedup tracks it almost exactly. Bandwidth stays flat while bytes per token drop, so tokens per second rise.

The same reasoning applies to an H100 or an MI355X, only with more zeros. When I size a decode tier, the first question isn’t TFLOPS. It’s how many bytes each token pulls from memory and how close the stack gets to peak bandwidth. openTPU puts both on one line, which most vendor benchmarks don’t.

Prefill tells the other half of the story. The 512-token prompt runs at 92 tok/s for Qwen3-0.6B int8, and the README says plainly that prefill is limited by the matrix unit’s multiply rate. Against the previous image (a two-column matrix unit at 120.755 MHz), the current one with a four-column array at 133.33 MHz runs prefill 1.3x to 2.0x faster, while decode moved less than 2.3% in every configuration. Prefill is compute and decode is bandwidth, and here you can watch them come apart.

4-bit without hiding the cost

openTPU’s 4-bit format uses FP4 values with two-level block scales, which works out to 4.25 bits per weight, and it keeps the LM head in int8 for accuracy. That buys 40 to 45% faster decode. What I like is the validation tooling. tools/validate.py checks the device against a CPU golden built from the same Hugging Face checkpoint, in fp32, with openTPU’s quantization applied. It reports top-1 agreement and KL divergence against both the quantized golden and the plain fp32 model.

It also measures a floor: the golden against itself after a one-ulp nudge to its inputs. On the simulator the device sits at 0.82 to 1.16 times that floor’s KL. The cost of quantization shows up separately, and it isn’t small. For Qwen3-0.6B, 4-bit weights drop top-1 agreement with fp32 to 85.2% (mean KL 0.166), against 92.2% for int8. The README publishes that number instead of burying it. If you’ve ever had to argue about whether a quantized model “matches” the reference, that split between device error and format error is the right way to frame it.

On the card itself (build 84989047, October 7), six runs across three models in int8 and 4-bit matched the simulator’s tokens with bit-identical logits on every prompt.

MoE models bigger than the card

The 4 GiB limit gets a practical workaround. For mixture-of-experts models that don’t fit, the card does the routing and computes every expert, but it keeps a set of per-layer expert slots in DRAM. The host copies missing experts in from a pool file over PCIe. The results, from the README’s October 1 measurements:

• LFM2.5-8B-A1B (8.5B total, 1.7B active): 10.6 tok/s, 98.5% of expert uses hit the slots, 5.2 MB streamed per token.
• Qwen3.5-35B-A3B (34.7B total, 3.0B active): 3.95 tok/s, 62% slot hits, 153 MB streamed per token at 1.41 GB/s over PCIe.

By my arithmetic, 153 MB at 1.41 GB/s is about 108 ms of transfer per token, against roughly 253 ms per token at 3.95 tok/s. So PCIe accounts for something like 40% of each token. That’s the expert-offload trade-off in miniature: hit rate decides everything, and a model whose routing spreads across experts pays for it on the link. Anyone planning to put a large MoE on GPUs with too little HBM and lean on host memory should run this math first.

The “developed by AI” part

The method lives in docs/tourney.md and comes from the author’s earlier auto-arch-tournament project, which evolved an RV32IM CPU core. Each hardware component gets its own champion branch. Each round, a hypothesis agent proposes one change, an implementation agent makes it in an isolated git worktree, and the change has to pass gates in order: a sandbox check that only allowed files changed, Verilator lint, bit-exact RTL-versus-simulator tests, a board micro-architecture run, a deterministic Qwen3-0.6B decode proxy (no more than 0.2% extra cycles), and synthesis. Only then is it scored on area or clock speed.

The default agent configuration is Claude Opus 5.5 for hypothesis and implementation at high effort, with Codex as an option. Agents run with no network tools and a shell limited to lint, test and read-only commands. The first smoke round, on September 23, cost $1.12 and took 4.7 minutes: the agent replaced the GATHER unit’s chained address multiplies with running-sum registers, taking it from 9 DSP slices to 0 and its estimated fmax from 75 to 128 MHz. The doc budgets $1 to $5 per slot for bigger units.

Two details matter. Merging a champion into main is explicitly a human decision, and the gates do the real work. The agents aren’t trusted; the bit-exact tests are. An agent can propose something clever, but nothing that disagrees with the simulator by one bit gets through. That pattern is worth copying well beyond RTL.

What it isn’t

These are self-reported numbers on the author’s own board, and I haven’t seen an independent reproduction yet. There’s no GPU comparison in the repo, and there shouldn’t be. Timing barely closes, with +0.032 ns of worst negative slack at 133.33 MHz. The smaller models still read DRAM at only 82 to 85% of peak, which the README lists as the next thing to fix. The validation tool doesn’t cover the MoE models yet.

Why I’d spend an evening on it

Everything except the card runs on a laptop. Pull LFM2.5-230M from Hugging Face, run otpu-chat --model lfm2 --backend isa on the simulator, then open a trace in the Lens profiler and watch which unit is busy and which is waiting on DRAM, cycle by cycle. I don’t know a faster way to build intuition for why decode tiers get sized by bandwidth, where 4-bit pays off, and why expert offload lives or dies on hit rate. And if an agent loop with strict gates can keep improving a hardware design, that’s a more interesting result than any number in the table.

Create-only. Specs from the openTPU repository and README, architecture tournament doc, ISA reference, MoE expert offload doc, 4-bit format doc, and auto-arch-tournament.

Sources: openTPU repository and README · openTPU architecture tournament doc · openTPU ISA reference · openTPU MoE expert offload doc · openTPU 4-bit format doc · auto-arch-tournament.

Related: vLLM 0.31 Fast Restart · Dist-MoE · AICR v1.0.

Anpoo Sivanadi, Staff Software Engineer