Olmo-core 3: Open MoE Training Stack Hits 1.2T on B300 — DDP, Rowwise EP, MXFP8

This morning’s Everspin CXL PERSYST note was a persistent media ticket. Last night’s CoreWeave × Cognition Vera Rubin was neocloud production. Plenty of frontier MoEs ship weights; far fewer ship the trainer that made sparse capacity cheap in wall time. On October 1, 2026, Ai2 released Olmo-core 3 — a redesigned open mixture-of-experts training stack aimed at trillion-parameter MoEs — with a full technical report and code on GitHub. Tonight is that training-systems board.

Primary: Ai2 blog · HF mirror · tech report · GitHub. Numbers below are Ai2’s unless noted. Treat tok/s, TFLOP/s, and memory deltas as systems measurements under the stated configs (often random routing), not time-to-quality claims.

Spec sheet (named numbers)

That is a different purchase sentence from “another open MoE checkpoint.” This one is: can I train sparse capacity without paying a full-reshard FSDP tax every microbatch, and what do Ai2 measure on B300 when they try.

What actually changed

Olmo-core’s earlier MoE path inherited the FSDP habits of dense Olmo 3: gather and reshard weights around microbatches. For MoE that is painful — stored expert capacity grows with N, while useful arithmetic follows K. Ai2’s framing: a capacity tax (storage, grads, optimizer sync following P_total) plus a runtime tax (dispatch/combine, uneven GEMMs, host sync).

Olmo-core 3’s answer is a DDP-centered path: keep the rank-local partition resident across the accumulation window, shard experts with EP, layers with PP, and FP32 main weights/optimizer state with a distributed optimizer. Token rows move to experts; weights stop thrashing through all-gathers every microbatch.

The composed result they emphasize: grow the expert pool 8 → 128 at fixed top-4 and ~3.2B active, total capacity 4.6B → 47B, and keep throughput within 5%. On eight B300s, the 47B point lands at 52K tok/s/GPU vs 19.4K on the old path — about 2.7×. That is a systems comparison under their harness, not a claim that Megatron-Core disappeared.

Mechanisms worth reading the report for

Rowwise EP. Instead of packing peer-contiguous blocks for classic all-to-all, their NVSHMEM path writes each token row straight into the destination expert buffer (PUT/GET). Less permutation tax on the hot path.

GPU-resident routing + grouped GEMM. Routing metadata stays on device so the CPU is not waiting on host copies for every shape; device-scheduled grouped GEMM eats uneven per-expert batches without padding every expert to the same capacity.

MXFP8. On a controlled four-GPU B300 bench with uniform expert load, enabling MXFP8 where it helped most bought ~21% throughput vs BF16 and cut peak active memory 103 → 95 GiB — Ai2’s controlled-config systems deltas, not a fleet-wide quality claim. Gains skewed to FFN compute and expert traffic, not attention alone. Ultra’s 858 TFLOP/s/GPU at 1.20T is reported with MXFP8 and per-layer recompute in the report’s headline panel — MFU denominators use B300 dense BF16 peak even for MXFP8 runs, so do not read that as FP8-peak utilization.

Findings that survived the cut. Token gerrymandering: load-balancing loss can improve while real imbalance worsens. Comm/comp overlap on separate streams is not free — sometimes it slows the kernels it was meant to hide. GEMM time depends on operand values, not only shapes. Scaling expert LRs by sparsity did not help in their tested family.

Where it sits next to other open stacks

Do not collapse this into “another open model drop.” Weights without the trainer leave the capacity tax on your cluster. Megatron-Core remains the industry reference Ai2 itself cites; Olmo-core 3 is the integrated path behind the next Olmo MoE, measured end-to-end on B300.

Caveats

1. Random routing for systems refs. Several headline rates use random routing to approximate balanced load. Production routers are messier; expect lower tok/s until you measure your own routing histogram.
2. 2.38T is a capacity probe. DeepEP v2 short run — reachable scale, not a sustained Ultra recipe. Do not treat Ultra’s 858 TFLOP/s/GPU as a quality or time-to-convergence claim.
3. Not a Megatron bake-off. Ai2 positions against their own FSDP MoE path and discusses Megatron-Core / DeepSpeed-MoE / DeepEP trade-offs in the report. Bring your own apples.
4. B300 NVL8 topology. Numbers are for their audited hardware/software; your IB/NVLink map and EP-MP placement will move the operating point.
5. Next Olmo is still future. The stack is shipping; the next MoE Olmo weights are not this post. No ship date in the release.
6. Not an Everspin rematch, not a CoreWeave rematch. This morning was persistent CXL media; last night was neocloud NVL72 production. Tonight is the open MoE trainer board.

How I would evaluate

1. Clone OLMo-core and reproduce the eight-GPU capacity sweep shape (fixed active, growing N) on whatever GPU you have — even if absolute tok/s differs, watch the slope.
2. Read Section 14 of the report for the Tiny→Ultra matrix (batch, EP/PP, precision, recompute) before quoting 858 TFLOP/s/GPU.
3. Turn on MXFP8 only after you can match their loss/grad-norm continuation story — throughput without a numerical recipe is just a faster wrong step.
4. Instrument load-balance and hard expert histograms so you catch gerrymandering early.
5. Keep this board separate from inference disaggregation posts — this is training sparse capacity on B300 clusters.

Bottom line

On October 1, 2026, Ai2 shipped Olmo-core 3: an open DDP+EP+PP MoE trainer with rowwise NVSHMEM EP, GPU-resident routing, grouped GEMM, and MXFP8, measured on NVIDIA B300 up through 1.20T total / 58.4B active at 858 useful-model TFLOP/s/GPU on 512 GPUs, with a 2.38T DeepEP v2 capacity probe. The durable story is the capacity-tax fix — grow experts without collapsing wall-time throughput the way their old FSDP MoE path did (19.4K → 52K tok/s/GPU at 47B on 8× B300) — plus the report’s honest negative results on overlap and token gerrymandering. Code and report are public; next Olmo is planned as MoE on this stack. Those tok/s, TFLOP/s, and MXFP8 deltas are Ai2 systems measurements under the stated configs — often random routing — not a sustained Ultra quality claim and not a next-Olmo ship date.

Create-only. Specs from Ai2’s Oct 1 Olmo-core 3 release and accompanying technical report. Update when Ai2 publishes sustained Ultra training runs, next-gen Olmo MoE weights, or third-party reproductions on non-B300 fleets.

Sources: Ai2 Olmo-core 3 · HF blog · tech report · GitHub.

Anpoo Sivanadi, Staff Software Engineer