NVIDIA NVHBM: Moving the Memory Controller Into the Stack

On August 26, NVIDIA expanded NVLink Fusion with NVHBM — a custom high-bandwidth memory path for partners building XPUs that still want to live inside an NVIDIA rack-scale domain. Five days later, the MediaTek deal made the integration story concrete: hyperscalers bring the compute die; MediaTek ships a package that already speaks NVLink.

I am reading this as a package architecture move, not a GPU launch. The interesting claim is not “more HBM.” It is where the memory controller lives.

Spec sheet (architectural targets)

Primary sources: NVIDIA blog, NVIDIA technical blog, NVIDIA × MediaTek newsroom, Tom’s Hardware, StorageReview.

What actually moved

In a standard HBM4e package, the memory controller sits on the accelerator die. That logic burns area you would rather spend on matrix engines, vector units, or SRAM. NVHBM inverts that: the controller (and a custom PHY) live in the base die of the HBM stack. The XPU connects through a narrower interface.

Three effects fall out of that inversion:

1. Bandwidth — NVIDIA says up to 30% more stack bandwidth than commodity HBM4e. For weight streaming and KV traffic, that is the denominator that decides whether your math units stay busy.
2. Power — up to 15% less HBM power. At one chip that is a footnote. Across a power-capped aisle it is headroom you can spend on clocks or more sockets. StorageReview’s facility-scale example (1 GW / ~2 kW XPUs → up to ~15k extra XPUs from the HBM power delta) is directional math, not a capacity plan — but the direction is why operators care.
3. Die area — up to 25% more of the main XPU die available for compute. Tom’s and StorageReview also flag PHY/support shrink and simpler interposer routing; treat secondary percentages as reporting on NVIDIA materials until a packaging teardown lands.

NVIDIA’s compound line is the one to keep honest: bandwidth + area + power savings → about 30% end-to-end XPU performance. That is a vendor stack claim, not a public InferenceX table.

Why this is not “just better HBM”

NVHBM is not a drop-in JEDEC replacement for every GPU on the market. It is a building block for NVLink Fusion partners. Commodity HBM4e remains the merchant path. NVHBM is the path where NVIDIA designs the base die with memory vendors, partners adopt a smaller PHY, and the package is meant to land in an NVLink scale-up domain.

That is the strategic half of the story. If a hyperscaler ships a custom XPU but still wants:

• a coherent rack-scale fabric next to NVIDIA GPUs,
• NVLink-C2C to host CPUs,
• MGX-style rack machinery,

then owning the memory base die plus the Fusion chiplet keeps NVIDIA in the bill of materials even when the compute logo is Trainium, Maia, MTIA, or someone else’s.

Amazon is the named first collaborator. Annapurna will support NVLink Fusion starting with Trainium4, and is working NVHBM into that collaboration. MediaTek’s role is different: become the integration factory so a customer does not have to qualify every SerDes and packaging corner against NVIDIA’s stack alone.

How I would read it next to UALink

The open counterweight is UALink (UALink Consortium — AMD, Broadcom, Google, Meta, Microsoft, and others; not NVIDIA). AMD’s Helios path leans on UALink-over-Ethernet until native UALink switch silicon is volume-ready. That is a different bet: open protocol on Ethernet PHY vs proprietary NVLink switch silicon.

As of early September 2026, the production-ready custom-XPU-into-proven-scale-up story is still NVLink Fusion’s window. UALink 2.0 exists on paper; commercial switch volume is the missing piece. Operators who want optionality will keep both programs alive. Operators who need a rack this year will price the dependency explicitly.

Caveats that matter for anyone sizing a cluster

1. Architectural, not measured. Every 15/25/30 number above is NVIDIA’s pre-silicon / early-program target. No independent lab has published an NVHBM-equipped XPU suite yet. Cite it as a target.
2. Captive ecosystem. NVLink switches and PHY control stay with NVIDIA. Fusion is heterogeneous compute on a NVIDIA-defined fabric, not a multi-vendor switch market.
3. Not Rubin shipping silicon. Benefits do not auto-apply to NVL72 boxes already in the field. This is for next custom XPU / future GPU generations that adopt the base die.
4. Pair with Jalapeño-style thinking carefully. OpenAI’s captive inference ASIC optimizes tokens per megawatt inside one company’s fleet. NVHBM optimizes who can build a custom die that still docks into NVLink. Same industry pressure (memory + power), different product shape.
5. Timeline. No public Trainium4 + NVHBM ship date beyond “next-generation” language. Plan capacity on named SKUs, not on the blog diagram.

Bottom line

NVHBM is the quietest consequential line in the late-August stack: move the controller into the memory, free the compute die, sell the result only inside the Fusion partnership. For people who run heterogeneous racks, the actionable read is simple — when you evaluate the next custom XPU, ask whether the HBM path is JEDEC-generic or NVHBM-locked, and what that does to your interconnect choice for the following five years.

Create-only. Specs above are from NVIDIA’s August 26 materials and contemporaneous technical coverage; update when Annapurna or a memory vendor publishes a measured stack.

Anpoo Sivanadi, Staff Software Engineer