AMD’s 512-GPU MI355X MLPerf: 5.75M tok/s on RoCE — and the +38% ROCm Asterisk

Last night’s Vera Rubin NVL72 note was NVIDIA’s first peer-reviewed preview tray. This morning is the other column on the same MLPerf Inference v6.1 whiteboard: what happens when you put 512 Instinct MI355X GPUs on a standard RoCE Ethernet fabric, plus the round-over-round ROCm gains that landed on unchanged silicon.

On September 16, 2026, MLCommons published Inference v6.1. AMD and first-time submitter Crusoe filed the largest GPU count in MLPerf Inference history — 512 MI355X across 64 Crusoe Cloud nodes — and AMD separately published the software story: up to +38% GPT-OSS-120B throughput at 8 GPUs versus the same MI355X hardware in v6.0. Primary sources: MLCommons v6.1 announcement, the MLCommons supplemental PDF (AMD + Crusoe statements), StorageReview’s round write-up, and Futurum’s ROCm-velocity read.

Spec sheet (named numbers)

Preview and Closed Division results are peer-reviewed, but stack and topology still decide whether you can paste any multiplier into a capacity model. Treat AgentX / tok-per-megawatt columns from the NVIDIA Summit blog as a separate methodology from this RoCE table.

What the 512-GPU table actually says

Three levers matter if you already run or are quoting Instinct:

1. Aggregate throughput on Ethernet. 5.75M / 5.39M tok/s on gpt-oss-120b and 2.90M / 2.41M on DeepSeek-R1 are cluster numbers, not a single-node flex. The fabric callout is explicit: standard RoCE, not a proprietary scale-up switch domain. That is the operational contrast to an NVL72 tray — different landlord for the interconnect politics, different failure domain when a NIC or switch ASIC misbehaves.
2. How the model was carved. gpt-oss ran as 512 independent single-GPU vLLM replicas in native MXFP4. DeepSeek-R1 ran as 64 eight-GPU SGLang replicas with tensor + expert parallelism. If your fleet is one serving stack and one sharding recipe, do not collapse those two rows into one “MI355X is X×” cell.
3. The 72-GPU sanity check. AMD’s own 95% scale efficiency and ~1M tok/s GPT-OSS line at 72 GPUs is the size most people can actually buy and operate before they talk about 512. Near-linear from 1→64 nodes is the claim that “more GPUs ≈ more revenue” is not fantasy for that Crusoe submission — still verify with your own burn-in, not a spreadsheet paste.

This sits next to my Helios UALoE note the same way Rubin sits next to Groq 3 LPX and d-Matrix Raptor: open-protocol Ethernet scale-up / scale-out is the AMD product thesis; MLPerf v6.1 is the receipt that the thesis can put half a thousand GPUs on one Closed Division scoreboard.

The +38% is the software audit

Hardware did not change between v6.0 and v6.1 for the MI355X configs Futurum and StorageReview highlight: still 288 GB HBM3E and 8 TB/s per GPU. The moving variable is ROCm. Up to +38% Server and +28% Offline on 8-GPU GPT-OSS, +70% on Wan-2.2 SingleStream, and a 72-GPU cluster that out-throughput a 94-GPU v6.0 submission — those are the lines I would put in a capacity ticket as software appreciation, not a new BOM.

AMD has also committed to a six-week ROCm release cadence (TheRock build system; ROCm 10.0 on Aug 27 as the first clocked release). One round of large gains does not prove the trajectory continues — Llama 2 70B already shows the mature-workload pattern where vendors tie because the easy kernels are gone. What changed is measurability: Closed Division, fixed silicon, partner reproduction within ~4%. That is a better audit trail than a vendor-only “3.3× since ROCm 7” claim.

Precision disclosure matters here. AMD says the GPT-OSS runs used the OpenAI MXFP4 checkpoint with an FP8 KV cache — the standard high-performance configuration, not an exotic quantization dodge — and has committed to publish reproduction steps. Keep that footnote next to any +38% slide.

MI350P and the partner band

Same CDNA 4 family, different chassis politics: MI350P is the dual-slot PCIe card (144 GB / 4 TB/s / 4.6 PFLOPS peak MXFP4/MXFP6) making its first MLPerf appearance against selected RTX PRO 6000 Server and H200 NVL submissions. StorageReview also notes an eight-GPU MI350X holding roughly ~80% of MI355X throughput across several workloads while MI355X carries a ~40% higher rated TDP — useful when the facility is power-capped before it is FLOPS-capped.

Seven partners landing within ~4% of AMD’s reference (some slightly above) is the reproducibility band I care about more than any single leaderboard row. If your integrator cannot get within a few percent of that band on the published recipe, you have an ops problem, not a silicon problem.

Caveats I would write into any MI355X capacity ticket

1. 512-GPU ≠ your rack. Quote the Crusoe topology (64 × 8-GPU RoCE nodes, named stacks) separately from the 8-GPU and 72-GPU rows.
2. Stack-specific deltas. vLLM single-GPU replicas ≠ SGLang TP8/EP8. Match your serving path before you multiply.
3. Ethernet is the feature and the risk. RoCE scale-out is why this story exists next to Helios; it is also why NIC/firmware health and congestion control belong on the same ticket as HBM generation.
4. +38% is one cycle. Budget a re-run date under the six-week ROCm clock; do not freeze a software-only multiplier into a multi-year contract.
5. NVIDIA comparisons have footnotes. Some AMD vs NVIDIA slides still lean on v6.0 baselines where NVIDIA did not resubmit; prefer Closed same-round pairs (e.g. official 8-GPU B300) when you argue.
6. MI455X / Helios tray is absent. This round audits CDNA 4 (MI355X/MI350X/MI350P), not the Helios UALoE generation. Do not infer Helios multipliers from these tables.
7. Not a Rubin substitute. Vera Rubin NVL72 answers a different interconnect and software stack question. Keep both columns.

How I would use this tomorrow morning

If you already run MI355X, the actionable trio is: (a) pull the Closed Division gpt-oss and DeepSeek-R1 rows and compare your sharding recipe to Crusoe’s 512-replica vs TP8/EP8 split, (b) ask whether your ROCm train is on the six-week cadence before you assume the +38% has already landed in prod, and (c) put RoCE fabric health next to HBM generation on the same slide. If you are choosing among Helios UALoE, a full Rubin NVL72 tray, and a dense MI355X Ethernet pod, MLPerf v6.1 answers the open-fabric aggregate-throughput question; it does not answer who owns your scale-up switch.

Create-only. Specs above are from MLCommons’ Sep 16 announcement and supplemental (AMD/Crusoe statements), StorageReview’s Sep 16 round-up, and Futurum’s Sep 16 ROCm analysis. Update when AMD publishes reproduction instructions or when MLPerf Endpoints starts accepting rolling MI455X results.

Sources: MLCommons · Supplemental PDF · StorageReview · Futurum.

Anpoo Sivanadi, Staff Software Engineer