NVIDIA Groq 3 LPX: 3,431 tok/s at 100K — and the Batch-1 Caveat

This morning I wrote about AMD Helios UALoE — open scale-up on Ethernet while native UALink switch silicon catches up. The other half of an agentic serving stack is not another interconnect. It is decode interactivity: how fast one user (or one agent loop) gets the next token when the context is already long.

NVIDIA Groq 3 LPX is the rack NVIDIA put into full production on August 24, 2026 for that job. Artificial Analysis measured a median 3,431 output tokens per second on Gemma 4 31B at 100,000-token input context. That number is real, third-party, and useful. It is also concurrency 1. If you size a floor off the headline without writing the batch ceiling into the BOM, you will over-buy interactivity and under-buy slots.

Spec sheet (named numbers)

Primary sources: NVIDIA newsroom, NVIDIA: ultrafast interactivity at long context, NVIDIA: inside Groq 3 LPX, The Register on the AA number.

What LPX is actually selling

LPX is not “a faster Rubin.” Vera Rubin NVL72 stays the general-purpose training and inference workhorse. LPX is a second engine optimized for the part of decode where small batches and long KV caches make first-bit latency and coordination tax kill GPU tensor-parallel speedups.

NVIDIA’s framing is explicit: agentic sessions accumulate context across hundreds of turns. Late-session turns still have to attend over everything learned so far. Without long context you get a fast amnesiac. Without high tokens-per-second-per-user you get a smart agent that feels like batch overnight. LPX is the bet that interactivity at 100K+ is a product feature, not a lab demo.

The silicon behind that bet is SRAM-first. Each LPU keeps about 500 MB of compiler-managed on-chip SRAM as primary working storage — weights, activations, KV state placed explicitly, not guessed by a hardware cache. Rack-wide that aggregates to 128 GB of SRAM and the 40 PB/s on-chip bandwidth figure in NVIDIA’s rack table. The trade is classic Groq: enormous bandwidth, tiny capacity per die, so models spill across many chips over deterministic chip-to-chip links.

Why small-batch TP needs a different fabric

At high interactivity you want tiny batches. Tensor parallelism still helps on paper — split the matmul, combine the results — but the fixed cost of synchronizing many small tensors starts to dominate the bytes-on-the-wire time. NVIDIA’s tech blog puts the bottleneck on first-bit latency: deciding which link to use, synchronizing endpoints, arbitrating conflicts.

LPX’s answer is to remove runtime arbitration. The compiler sees all 256 LPUs, the 128 GB SRAM pool, and the 96×112 Gbps C2C mesh per chip, and emits a clock-cycle schedule before the workload starts. Send on the cycle the data is ready; consume on the cycle it arrives. On top of that, transfers and compute overlap at 320-byte vector granularity — start shipping a slice of a matmul as soon as that slice exists, instead of waiting for the whole output matrix.

That is why the AA 10K and 100K medians sit almost on top of each other (3,382 vs 3,431). Deterministic routing plus heavy TP keeps output rate roughly flat as context grows, which is the property you want for late-session agent turns.

The number you should not paste into a capacity plan alone

Artificial Analysis’s 3,431 tok/s is the median of sequential client requests on a private NVIDIA-hosted LPX endpoint, measured at concurrency 1, with a shared o200k_base tokenizer. Public serverless peers on the same leaderboard were multi-tenant. So the comparison is “dedicated LPX peak interactivity” versus “shared commercial endpoints,” not a fair bake-off of equally loaded racks.

Gemma 4 31B is also a dense 31B — every parameter lights up each token. That is a friendly shape for an SRAM fabric: The Register notes FP8 needs a bit over 31 GB, so the model fits in well under one 256-LPU rack (on the order of 64 LPUs for weights alone, with headroom for pipeline / data parallel replicas). Larger MoE models activate fewer parameters per token but thrash experts and need many more chips for the full weight set — DeepSeek-V3-class footprints are multi-rack in that analysis.

Capacity math at 100K context is the other footgun. Headline tok/s is a single-stream figure. On a 256-LPU rack, KV and working-set pressure at that context length push effective concurrency toward a small batch (operator reads put the ceiling near ~12 concurrent 100K streams). Size your concurrent agent sessions off that envelope, not off 3,431 × N.

NVIDIA’s own SPEED-Bench coding numbers (4,767 median / 5,520 P80) are useful as a same-model coding-loop check, and still vendor-run. Treat them as generation-over-generation color, not as a substitute for your ISL/OSL mix.

Heterogeneous decode is the real product

The AA scorecard is LPX-only. The shipping story NVIDIA wants you to buy is Vera Rubin NVL72 + LPX:

1. Prefill–decode disaggregation — Rubin builds the KV cache; LPX decodes from SRAM-resident weights.
2. Attention–FFN disaggregation (AFD) — Rubin keeps attention over the growing KV in DRAM; LPX runs FFN / sparse MoE experts; intermediate activations cross once per layer per token.
3. External-drafter speculative decoding — LPX drafts; Rubin verifies; only draft tokens cross the link.

NVIDIA Dynamo is the control plane that classifies requests, routes by latency target, and tries to keep interactive sessions out of throughput queues. That software layer is load-bearing. Without it, you own two racks and a research paper.

Vendor economics claims — up to 35× tokens/s per MW at a premium operating point versus GB200 NVL72, up to 10× revenue opportunity when LPX is paired in — stay in the footnote until independent rack suites publish. What I keep for planning is the architectural split: throughput path on Rubin, interactivity path on LPX, Dynamo in the middle.

Caveats I would write into any LPX BOM

1. Batch-1 ≠ fleet capacity. Write concurrency and context length next to tok/s before you order trays.
2. Dense 31B is not your MoE production model. Re-bench on the active-parameter shape you actually serve.
3. Cerebras CS-4 / Nexus is shipping into the same window. AA’s ~4× gap is vs prior Cerebras endpoints; CS-4’s three-wafer backpack rack and Direct Wafer Links will re-open that comparison. AMD Helios + Cerebras disagg was already public at Advancing AI.
4. SRAM capacity is the tax. 500 MB/chip forces model parallelism; ops ownership includes the compiler schedule and C2C spine, not just CUDA graphs you already know.
5. Nebius is a signal, not coverage. One Token Factory launch does not mean every region or every model SKU.

Bottom line

Groq 3 LPX is NVIDIA putting a named interactivity rack next to Vera Rubin: 256 LPUs, 128 GB SRAM, 640 TB/s scale-up, and a third-party 3,431 tok/s at 100K on Gemma 4 31B. Next to this morning’s Helios UALoE piece, the early-September systems framing is clean — open scale-up fabric on one side of the aisle, proprietary decode accelerator with a batch-1 asterisk on the other.

If you run agent fleets, put tokens per second per user at your real context and concurrency on the same line as HBM generation and switch tray. The demo number sells the rack. The concurrency ceiling decides whether the rack pays for itself.

Create-only. Specs above are from NVIDIA’s Aug 24 production announcement, the two NVIDIA technical blogs, Artificial Analysis methodology as cited there, and contemporary Register analysis; update when independent multi-tenant LPX suites or MoE-at-100K numbers publish.

Anpoo Sivanadi, Staff Software Engineer