On October 9, two open serving stacks published the first useful Vera Rubin numbers that are not a keynote slide. LMSYS brought SGLang and Miles up on two early-access Vera Rubin nodes (eight GPUs) and walked the Kimi K3 NVFP4 kernel changes that actually moved the needle. The same day, the vLLM blog (Inferact, NVIDIA, Red Hat, and the community) said Vera Rubin NVL72 already runs from daily CUDA 13.4 containers and posted early AgentX and MLPerf multiples versus GB200 / GB300.
I read these the same way I read Arista Etherlink SU-144 and Upscale Token Fabric: ignore the PFLOPS banner until someone names shared-memory limits, SM occupancy, collectives, and a workload.
What changed in the silicon (for kernel writers)
Vera Rubin sits after Blackwell Ultra (GB300 NVL72). For people who write attention and MoE kernels, the LMSYS team called out three limits that matter on their early nodes:
| Limit | Hopper / Blackwell | Vera Rubin (LMSYS early nodes) |
|---|---|---|
| Shared memory per CTA | 227 KiB | 327 KiB |
| SMs visible on their nodes | — | 212 |
| Scale-up fabric | NVLink 5-era | NVLink 6 |
NVIDIA’s own platform write-ups put a full Rubin GPU at 224 SMs, HBM4, and up to 3.6 TB/s bidirectional NVLink 6 per GPU in an NVL72 domain. The vLLM post frames the rack against GB200 NVL72 as roughly 5× NVFP4 FLOPS, ~2.4× HBM bandwidth, and ~1.7× bidirectional NVLink bandwidth, plus 2× FP32 and 4× BF16/FP16 exponential throughput for softmax. Softmax keeping up with faster MMA is not a footnote if your decode path is attention-bound.
Those are the knobs. Everything below is what people did with them on day zero.
SGLang on Kimi K3 NVFP4: where the percent points came from
Kimi K3 is a 2.8T-parameter hybrid with a 1M-token context: 93 attention layers (69 KDA linear-attention + 24 MLA), Attention Residuals banking every attention output, and a LatentMoE FFN with 896 experts, top-16 routing, in a 3584-d latent space. LMSYS served the NVFP4 checkpoint with RadixArk’s DSpark block speculative decoding (draft then verify every step).
Attention. On Blackwell, the FP8 MLA kernel had already packed 3 K stages and 2 V stages into 227 KiB of shared memory. Rubin’s 327 KiB bought a fourth K stage and two more V stages. At batch 16 and 128k context, that deeper pipeline is 16% faster with bit-identical output. Separately, reordering split-KV reduction to issue all loads before accumulating from registers made full FP8 MLA 20% faster at batch 1 / 128k. Doubling split-KV from 32 to 64 kept more of the 212 SMs busy on batch-1 decode (+1.9% end to end with FP8 KV). Saturating FP8 conversion (clamp to ±448 instead of NaN) made the KV/Q preparation kernel 2.5× faster.
MoE / collectives. Every MoE layer ended with four tiny kernels: finalize experts, add shared expert, all-reduce across 8 GPUs, RMSNorm. K3 pays that tax in 92 MoE layers. Folding them into one collective kernel removed 276 launches per decode step and gained 5.9% end to end. Push-style all-reduce stayed ahead of pull up to 256 rows on Rubin (crossover moved), worth 3.6% decode throughput at 32 concurrent requests. Moving the latent all-gather GEMM onto mma.m16n8k16 for 4–8 rows cut another 1.1% of step time.
KDA. The fused verify kernel was barrier-bound: every CTA copied conv weights into shared memory and waited. Keeping weights in registers per warp, dropping CTA barriers for in-warp handoffs, and using idle v-warp time for the output-norm gate made the production verify kernel 20% faster with bitwise-identical outputs. Teaching gated RMSNorm to read its strided gate without a silent PyTorch copy removed 2.6 µs × 69 copies per step (−1.8% decode step time).
None of that is a rack-level TCO claim. It is a profile of where Rubin actually pays off on one hard hybrid MoE.
Miles: RL on the tray, sandboxes on the Vera CPU
Miles (SGLang for rollout, Megatron for training) ran end to end from a single container once the Rubin-enablement PR landed. On a single tray (4 GPUs):
- Qwen3-30B-A3B on GSM8K (256 prompts × 8 samples, up to 1,024 response tokens): reward rose ~45% → 95% over 50 rollouts, matching the GB300 reward curve.
- DeepSeek-V4-Flash (4-layer, FP8) on GSM8K: FP8 rollout and training both completed end to end.
Agentic RL used NeMo Gym’s mini-SWE-agent on the tray’s Arm-based Vera CPU: 64 sandboxes beside GPUs serving/training Qwen3.5-35B-A3B, SWE-bench Verified arm64 images, 8 tasks × 8 attempts from a 64-task pool, up to 64K tokens per episode. Reward held ~0.6, episodes got ~30% shorter, and train–rollout KL stayed matched. That is the interesting platform claim: the CPU next to the GPUs is part of the training loop, not just a host.
LMSYS also notes Cognition has already deployed Vera Rubin NVL72 on its own SGLang-based stack and reported up to 4.8× total token throughput versus GB200 NVL72 — private fork and private settings, so treat it as a direction, not a rematch recipe.
vLLM: containers today, AgentX and MLPerf as the early scoreboard
vLLM’s day-0 story is usability plus early multiples. Nightly images (vllm/vllm-openai:cu134-nightly) ship on CUDA 13.4 / PyTorch 2.15. Rubin is a new compile target (sm107), but Blackwell-family (sm100f) kernels already run; FlashInfer 0.7.0 adds Rubin-tuned attention, GEMM, and MoE, plus a Rubin-tuned MiniMax Sparse Attention (MSA) prefill path. Model coverage named on day zero: DeepSeek, Kimi, GLM, MiniMax.
The architecture piece I care about is CUDA 13.4 locality domains. HBM is non-uniform; locality domains let you place MoE weight shards so each SM group reads the half of W in its own domain. On MiniMax M3 MoE shapes, localized FC1+FC2 was about 1.2× faster in small-token decode across TP2/TP4/EP2/EP4. Default domain creation can leave you on 200 of 212 SMs; backfill mode (cudaDevSmResourceGroupBackfill) uses all 212. Even the 200-SM path still won, because decode is weight-load bound.
Early scoreboard (vendor / project self-report — early access, expect more):
| Benchmark | Claim |
|---|---|
| SemiAnalysis AgentX, MiniMax M3 on Vera Rubin NVL72 | up to 7.84× throughput per GPU vs GB200 NVL72 at matched interactivity; 5.18× under a 150 TPS constraint |
| MLPerf Inference v6.1 VLM, Qwen3-VL-235B-A22B (vLLM backend, Dynamo frontend) | up to 3.7× throughput vs GB300 NVL72 across offline / server / interactive |
Operators who want the knobs today: FlashInfer CuTe-DSL NVFP4/MXFP4 GEMM and MoE backends, FP8 KV with FlashInfer / FlashInfer-MLA attention, and the MSA prefill cutlass path documented in the vLLM appendix. Next on their list: sm107 MegaMoE, fuller locality-domain MoE, Kimi K3 KDA/MLA on Rubin, DeepSeek-V4.1-Flash CSA/HCA, MSA decode, and counted-write MoE all-to-all.
How I would use this
If you are sizing a Vera Rubin tray or NVL72 for agentic serve, read the LMSYS post for where kernels spend time on a hybrid MoE, and the vLLM post for what is already in a container and what AgentX/MLPerf currently claim. The 327 KiB shared-memory bump, 212–224 SM occupancy, NVLink 6 collectives, HBM4 bandwidth, and locality-domain MoE are the real platform story — not another “next-gen AI factory” paragraph.
Cross-check Cognition’s 4.8× and vLLM’s 7.84× carefully: different stacks, different models (Cognition’s private SGLang fork vs MiniMax M3 on AgentX), different interactivity contracts. Both say the same directional thing. Neither replaces a bake-off on your traffic.
Sources fetched for this piece: LMSYS SGLang/Miles on Vera Rubin (Oct 9), vLLM Vera Rubin NVL72 preview (Oct 9), plus NVIDIA’s Vera Rubin platform and DGX Vera Rubin NVL72 pages for the full-SKU SM / NVLink framing.
Related: RTX Spark N1X: The Windows Numbers That Matter Are TDP, Memory Ceiling, and Prefill · Scality AI Inference Factory: Object Storage Over RDMA as the Shared KV Cache · AICR v1.0: NVIDIA Freezes the Contract for GPU Kubernetes Recipes.
Anpoo Sivanadi, Staff Software Engineer