Maia 300: Microsoft’s Inference ASIC Puts 288 GB of HBM4 and a 12,288-Chip Ethernet Superpod on the Table

Most of the OCP Global Summit noise this week is about open racks and faster fabrics. Microsoft used the same stage to put numbers on Azure Maia 300, the third generation of its in-house inference accelerator and the first Maia built as a multi-chiplet part.

I care about this drop because it is not another “AI demand is huge” slide. It is a hyperscaler publishing a full inference stack — silicon, tray, rack, and scale-up network — with concrete memory, power, and topology figures, and saying Maia becomes the primary inference system for Microsoft products starting with multi-gigawatt deployments in 2027. Whether you will ever rent a Maia instance is a separate question. The design choices are worth reading if you run long-context or agentic serving on any accelerator.

What shipped on paper

Scott Guthrie’s announcement and the Azure architecture deep dive line up with the StorageReview spec table compiled from the same materials. The headline silicon:

That is a 2.5× memory-bandwidth jump and a 33% capacity bump for about a 27% TDP rise and a 37% FP4 compute rise. The generation is intentionally skewed toward feeding decode, not toward topping FLOPS charts.

On the rack side, Microsoft doubles the accelerator count and more than doubles the power envelope:

Microsoft’s claims against Maia 200 are “up to 7×” rack-scale token throughput in the full 96-accelerator config and “more than 2.5×” performance per watt. Against “leading GPU-based systems,” the company quotes 40% more tokens per megawatt. That last comparison does not name the GPU, the model, or the serving config, so treat it as a positioning number, not a reproducible bench.

Why the architecture is interesting

Maia’s programming story is Software Defined Locally Accessed Dataflow Architecture (SDLA). Control-path programming is separated from data-path execution; software explicitly places and moves data across local SRAM, DMA engines, compute engines, and the network. Maia 200 introduced that model. Maia 300 stretches it in three ways that matter for long-context decode.

First, memory layout moves from uniform access to a NUMA organization across chiplets. Each compute region has nearby memory, and SDLA’s DMA and network-on-chip controls are how you keep KV and activations from ping-ponging across the package. StorageReview’s breakdown of the package is a main N2P compute die, two I/O dies, four memory dies, and eight HBM4 stacks — more surfaces for locality bugs if software gets it wrong, and more room to get it right if the toolchain lands.

Second, precision gets narrower. Enhanced MXFP4 and hardware data-type conversion are aimed at cutting bytes moved per token for both compute and KV-cache storage. Kernels can also treat on-die SRAM as a software-managed scratchpad so HBM is not the only working set. That is the same bet a lot of serving stacks are making in software; Microsoft is baking it into the ISA and the memory hierarchy.

Third, the network is not a bolt-on NIC story. The on-die AI network controller speaks a custom AI transport over Ethernet. Software sees one communication model from a tray pair all the way to the superpod. Latency domains are explicit: a directly connected accelerator pair is cheapest, anything else in the 96-accelerator rack is one switch hop, and anything across pods is three hops. An accelerator can drive up to 750 GB/s to a peer in the superpod when only a subset of links go cross-rack, and the in-rack versus cross-rack bandwidth split is a job-packing parameter rather than a fixed PCB design.

That last point is the one I would steal for any MoE serving review. Tensor parallelism can stay inside the rack; expert and pipeline parallelism can span racks; prefill and decode workers can sit on different trays and move KV over the same scale-up fabric instead of bouncing through a host-mediated scale-out NIC. Hot-spare capacity in another rack can absorb a failed tray without leaving the fabric. Whether those semantics survive real failure modes is an operations question Microsoft will have to prove in 2027. On paper, they closed the usual “two networks, two programming models” tax.

How the rack is put together

The physical packaging matches the software story. Each of the 24 trays carries four accelerators plus host CPU, memory, storage, and cooling — no separate host trays stealing U-space. Six high-radix switch ASICs live in three liquid-cooled switch trays for in-rack connectivity. Power is dual-feed with high-capacity shelves and a 50 V busbar; an automatic transfer switch keeps the rack up on one feed during service. Liquid cooling takes more than 80% of the thermal load from accelerators, voltage regulators, and switch ASICs; air covers memory, storage, and ancillary parts. Microsoft says the rack can attach to facility water or reject heat through a heat exchanger in an air-cooled hall.

None of that is exotic for a 167 kW AI rack in 2026. What is unusual is how tightly Microsoft couples it to one inference ASIC and one Ethernet-based scale-up story instead of mixing GPU SKUs and a separate InfiniBand or Spectrum-X domain.

What is already running, and what is not

Maia 200 is in production across the United States, Europe, and Australia. Microsoft says it is serving Copilot and the MAI model team today, with GitHub Copilot and Microsoft Foundry planned by year end, and that it runs OpenAI models, Microsoft’s own MAI models, and leading open-weight families. The company attributes up to 1.7× more tokens per watt on built-in Teams / Copilot / Cowork AI features versus its own GPU infrastructure for those workloads. Again: Microsoft-owned comparison, not a public MLPerf line.

Maia 300 production deployments start in 2027. It is not offered as an Azure VM SKU you can rent. NVIDIA and AMD remain in Microsoft’s training fleet. So the readers who can validate these numbers first are Microsoft’s own serving teams and anyone who gets early access through a joint product — not an independent neocloud tomorrow morning.

How I am reading it

Put next to yesterday’s open switch-platform work and the rest of the OCP networking slate, Maia 300 is the other pole of the same week: a closed, co-designed inference factory with open Ethernet as the scale-up medium, versus open platform management on merchant switch silicon. Both are answers to the same pressure — long-context and agentic decode that starve on memory bandwidth and drown in cross-domain networking glue.

If you size clusters for decode-heavy MoE or multi-step agents, the useful takeaways are concrete:

• Memory bandwidth is the generational bet (2.5×), not FLOPS (roughly +37% FP4).
• 96 accelerators in one Ethernet scale-up domain, with a documented path to 12,288, is the topology Microsoft wants software to target.
• Prefill/decode disaggregation and cross-rack spare trays are first-class fabric features, not scheduler hacks on top of a second network.
• The efficiency claims against GPUs are directional until someone outside Microsoft publishes matched configs.

I will watch for the Ignite follow-ups Microsoft teased for next month, and for whether any of the SDLA / MXFP4 / AI-transport pieces show up in public kernel or collective libraries. Until then, the architecture post and the announcement are the primary sources; the StorageReview write-up is the cleanest secondary table I have seen.

Related: OSPA: Meta, Microsoft and Alibaba Want Every AI Fabric Switch to Boot the Same Way · Arista Etherlink SU-144: Ethernet Moves Inside the Scale-Up Domain, and the Rack Becomes the Unit · Upscale Token Fabric: SkyFabriX at 115.2 Tbps Meets Spectrum-X Scale-Out Under One OS.

Anpoo Sivanadi, Staff Software Engineer