Kolibri-1: 78.1B / 3.46B Open MoE — Hybrid SWA, 1M Context Claim, ~78 GB FP8

Last night’s Credo Weaver note was near-package memory fanout. Yesterday morning’s Lightning V2 on Tenstorrent was on-prem TTS on Blackhole. The night before, Olmo-core 3 answered an open MoE training ticket. Today’s board is different: a new open-weight serving MoE with hybrid sliding-window attention, a named B200 train topology, and an FP8 footprint a cluster operator can place — not another AliceAI / DeepSeek Flash / MiMo rematch. On October 3, 2026, Aleph Alpha released Kolibri-1.

Primary: Aleph Alpha blog · Hugging Face Aleph-Alpha/Kolibri-1 · tech report PDF (189 pp) · aleph-alpha-inference (vLLM plugin). Numbers below are Aleph Alpha’s unless noted. Treat benches, Pareto charts, and concurrency claims as vendor measurements under their harness, not an independent bake-off.

Spec sheet (named numbers)

That is a different purchase sentence from “another open MoE leaderboard dump.” This one is: can I load a documented open MoE with hybrid SWA, named B200 topology, and a ~78 GB FP8 weight file onto a small GPU set with a vLLM plugin that already knows the reasoning / tool parsers — and what does Aleph Alpha publish when they try.

What the architecture is selling

Olmo-core 3 was a trainer story. Naive-N0.5-Flash was a decode story — zero full-attention layers, SWA+DSA, native 1M. Kolibri sits on the serving side of that second camp, but it does not delete full attention. It keeps a 4:1 hybrid: forty layers look only 512 tokens back; ten layers see the full context. Decode FLOPs and KV traffic in the SWA layers stay bounded as context grows; the full layers carry long-range state. RoPE lives only on the SWA blocks, which is how they claim extension past the 256k trained window without YaRN-style scaling — validated, they say, to 1,048,576.

The MoE side is sparse capacity: 384 narrow experts, top-6, plus one shared expert, 3.46B active of 78.1B total. Blog sizing: they preferred many small experts over fewer wide ones, and sized down from a 123B candidate because two H100s held only 3 concurrent 256k queries versus 18 concurrent and 28% faster decode on 78B. That is a serving argument, not a leaderboard argument.

Train recipe with named topology

The durable systems detail is the topology. Pretrain: 768 B200s, EP8 inside the node (NVLink Gen5), FSDP16 × DP6 across the SuperPOD — dense shards over EP×FSDP = 128 GPUs per replica, experts over FSDP = 16. Mid and long-context drop EP and run FSDP128 / DP6. Token budget: 20T at 16k, 3.44T at 64k, ~201B at 256k. Optimizer: Muon on 2-D backbone weights; Exact Quantile Balancing for routing. HF lists energy ~9.5×10² MWh for pre/mid/LC (excludes SFT/RL) and knowledge cutoff EN/DE June 18, 2026. Post-train adds reasoning effort none / low / medium / high plus tool calling via the Kolibri parsers — cite SFT/RL scale claims as vendor process numbers.

Vendor benches — attribute, do not rematch

Aleph Alpha’s English table (highest reasoning effort, their harness) puts Kolibri at AIME 2026 96.0, GPQA Diamond 84.3, LiveCodeBench v6 85.9, TerminalBench 2.1 27.7, beside Qwen3.6-35B-A3B, Nemotron 3 Super, and Mistral Small 4. Their blog frames a quality-vs-throughput Pareto frontier for English and German at 8× B200. I am not claiming a rematch against AliceAI-Foundation-80B, DeepSeek-V4.1-Flash, or MiMo-V2.6-Pro. Kolibri’s useful distinction is the open train recipe + hybrid SWA serve path + ~78 GB FP8 residency.

Where it sits next to other boards

Do not collapse this into “another European open model press release.” The fleet-operator claim is narrower: documented hybrid attention, an FP8 weight file that fits a dual-H100 / single-B200 class box, a vLLM plugin with reasoning and tool parsers, and a tech report that names the train topology.

Caveats

1. Vendor harness. AIME / GPQA / LiveCodeBench / TerminalBench and the Pareto charts are Aleph Alpha’s. Independent reproductions are not this post.
2. 1M is validated extension, not the daily default. Trained native max is 262,144; they recommend ≤ that (~256k) for latency, throughput, and complex tasks. The 1,048,576 path needs the --max-model-len / HF override flags.
3. LC token rounding. HF card: 201B long-context tokens; blog and tech report often say 200B. Quote the source next to the number.
4. Not Naive’s architecture. Kolibri keeps 10 full-attention layers. Naive-N0.5-Flash deleted full attention entirely (SWA+DSA). Same long-context problem family; different mechanism.
5. Plugin pin. aleph-alpha-inference currently targets vLLM 0.29. Pin the container / wheel pair before you freeze a serve image.
6. EU / sovereignty framing is product marketing — orthogonal to whether the MoE hits your concurrency target.

How I would evaluate

1. Read Table 2 / §2.1 in the tech report and the HF architecture block before quoting 4:1 SWA or 3.46B active.
2. Bring up Aleph-Alpha/Kolibri-1 with the published vLLM flags on 2× H100 or 1× B200; measure TTFT and tok/s at 64k / 256k with reasoning none vs high.
3. Re-run your AIME/GPQA/code suite; do not paste their TerminalBench 27.7 into a capacity plan.
4. Stress KV at long context with FP8 KV on and off; watch residency next to the ~78 GB weight floor.
5. Keep this board separate from Credo’s beachfront math, Lightning’s TTS CapEx table, and Olmo-core’s trainer tok/s — this is open MoE serve residency + hybrid SWA.

Bottom line

On October 3, 2026, Aleph Alpha shipped Kolibri-1: an Apache 2.0 English–German MoE with 78.1B total / 3.46B active, 50 layers of 384 experts (top-6 + 1 shared), hybrid 4:1 SWA(512):full attention, native 256k validated to 1,048,576, and a published train recipe on 768 B200s (EP8 / FSDP16 / DP6, ~24T tokens). Serving is concrete — ~78 GB FP8 weights, FP8 KV, a vLLM plugin with Kolibri reasoning and tool parsers, and min footprints down to dual-H100 / single-B200 class. Vendor benches (AIME 2026 EN 96.0, GPQA Diamond 84.3, LiveCodeBench v6 85.9, TerminalBench 2.1 27.7) are theirs; treat the Qwen3.6 / Nemotron Super / Mistral Small 4 columns the same way. Next to last night’s memory-fanout board and yesterday’s TTS ASIC board, this is the open serving MoE ticket with a full train recipe — not another AliceAI / DeepSeek Flash / MiMo rematch.

Create-only. Specs from Aleph Alpha’s October 3, 2026 blog, Hugging Face model card, 189-page tech report, and aleph-alpha-inference README. Update when third-party serve traces, multi-node EP bake-offs, or independent bench reproductions ship.

Sources: Aleph Alpha Kolibri blog · HF Kolibri-1 · tech report PDF · aleph-alpha-inference.

Anpoo Sivanadi, Staff Software Engineer