Meta published Inside Prometheus on October 19, 2026. It describes the supercluster in New Albany, Ohio that trained the Muse models, written by Kevin Lee, Yashar Bayani, Ahmed Sharif and Yee Jiun Song. The headline is roughly 1 million GPUs at full scale. The useful part is smaller and rarer: real reliability numbers for GB200 and GB300 NVL72 at fleet scale, and how Meta plans capacity around them.

If you run NVL72 racks, or are about to, these are the numbers I'd write down.

The hardware, briefly

Prometheus runs two NVL72 platforms today, with a third coming.

  • Catalina (GB200 NVL72, Ariel variant). Two racks of 36 GPUs form one 72-GPU NVLink domain. Each Blackwell GPU gets its own Grace CPU and two ConnectX-7 400G NICs, so 800G of scale-out per GPU. Meta lists 13.4 TB of HBM3e (192 GB per GPU), 720 PFLOPS peak FP8, 130 TB/s of NVLink bandwidth and about 120 kW across the pair.
  • Clemente (GB300 NVL72). 288 GB of HBM3e per GPU, 20.7 TB per domain, ConnectX-8 SuperNICs at 800G per GPU natively, a 2:1 GPU-to-CPU ratio, and the full 72-GPU domain in one rack at about 135 kW. Meta quotes a 5 to 10% gain over GB200 NVL72.
  • San Miguel (Vera Rubin NVL72), next. 288 GB of HBM4 per GPU at 19.2 TB/s, 2.52 EFLOPS peak FP4 per rack, 216 TB/s of NVLink (3 TB/s bidirectional per GPU), and more than 170 kW per fully liquid-cooled rack.

The Ariel choice is the interesting design call. Meta's reasoning was that GB200 scaled FLOPS, HBM bandwidth and NVLink domain size well past H100, while standard scale-out would have stayed at 400G per GPU. They expected that imbalance to cost performance, so they paid for a 1:1 CPU-to-GPU layout to get 800G per GPU early. Even so, the gap is wide. Clemente's 1.8 TB/s bidirectional NVLink per GPU works out to about 7.2 Tb/s each way, against 0.8 Tb/s of scale-out. That's roughly 9 to 1 (my arithmetic), which is why the parallelism plan has to keep the heavy collectives inside the rack.

Power per GPU slot keeps climbing too. Dividing Meta's rack figures by 72 gives about 1.67 kW per GPU for Catalina, 1.88 kW for Clemente and more than 2.36 kW for San Miguel, before cooling.

The reliability numbers

  • More than 85% goodput on Muse Wonder pre-training runs and more than 90% on RL runs.
  • Wall-clock mean time between interruptions (MTBI) of about 47,000 GPU-hours during those runs, raised to more than 54,000 GPU-hours since then through fleet changes that include firmware fixes.
  • A peak of nearly 1,000 interruptions per 8,000 GPUs per day in mid-2025, during the first phase. Meta says a year of work cut that by 99%.
  • A stated target of two interruptions per 8,000 GPUs per day, which Meta equates to about 100,000 GPU-hours MTBI.

MTBI in GPU-hours is easy to misread, so it helps to turn it into wall-clock time for a single job. Divide MTBI by the job's GPU count. At 54,000 GPU-hours, a 32,768-GPU run sees an interruption about every 99 minutes. A 100,000-GPU run would see one about every 32 minutes. Even at Meta's 100,000 GPU-hour target, the 32K-GPU job is interrupted roughly every three hours. (That's my arithmetic from Meta's figures; the post doesn't break it down by job size.)

At these rates a restart isn't a rare event. You make restarts cheap, and you design the cluster so most faults don't need one.

Why 64 of every 72 GPUs do the work

This is the part I'd copy first. NVL8 on H100 meant an eight-GPU tensor-parallel domain. NVL72 makes it 72, which Meta describes as a 9x larger fault blast radius. A single GPU fault can now take out a whole rack's worth of work.

Meta's answer is to treat the rack as the unit of allocation. Training parallelism maps most cleanly onto power-of-two group sizes, so each NVL72 domain runs a 64-GPU slice and keeps the other 8 GPUs as in-rack spares. When a GPU fails, a spare takes over and the job restarts from its last checkpoint without waiting for a repair. For whole-rack faults, Meta holds about 2% extra racks in reserve.

Meta's worked example is a 32,768-GPU run provisioned on 523 NVL72 racks, or 37,656 GPUs. The post calls that "about 13% above the active footprint," with 11 points coming from the 64-of-72 geometry. One small note if you're reusing this for your own capacity plan: 4,888 idle GPUs is 13% of the 37,656 provisioned, but it's about 15% on top of the 32,768 active. The 11% matches 8 spares out of 72. Either way, the point stands. Most of the overhead is a geometry choice you'd likely make anyway, not a tax for unreliability.

Two recovery paths

Meta describes two mechanisms, chosen by how far a fault reaches.

Parallelism-aware fault tolerance (PAFT). If the fault stays inside one data-parallel replica, that replica goes offline, the rest of the cluster repeats the step and keeps training, and the failed slice restarts on spare hosts and resyncs. A full halt turns into a short dip in throughput.

Checkpointing. If the fault reaches past one replica, the whole job rolls back. Meta checkpoints model state, including gradients and optimizer data, asynchronously to multi-tier storage, using delta checkpoints and in-memory staging to keep the cost off the training step.

The arithmetic above explains why both are needed. With an interruption every 99 minutes on a 32K-GPU job, the classic Young approximation for checkpoint interval, the square root of two times checkpoint cost times MTBI, gives about 14 minutes if a blocking checkpoint costs one minute. That's an illustrative assumption, not a Meta figure. The lever Meta is pulling is to make checkpoint cost close to zero with async and delta saves, and to keep as many faults as possible on the PAFT path where nothing rolls back.

What actually broke

The breakdown by cause reads like a fleet bring-up checklist.

  • NVLink signal integrity was the first bottleneck on GB200. One NVL72 domain has 5,184 tuned differential pairs at 200 Gbps, each crossing GPU SerDes, tray PCBs, cable cartridges and switch tray interfaces. Across more than 600,000 NVLink connections, marginal links caused frequent recovery cycles. The fix was continuous telemetry, automated flagging, and L11 diagnostic burns to find weak links before production.
  • Memory faults in HBM and LPDDR became the leading cause as GB300's larger memory arrived. The mitigations are refined HBM firmware settings, memory scrubbing, a better LPDDR error correction algorithm, and pattern-based prediction to retire degrading devices early. Meta says it is still measuring the impact.
  • NVLink switch and fabric health is watched at millisecond resolution, and back-end link failures are rerouted in hardware, both without interrupting the job.

Meta's firmware automation can push driver and firmware fixes to more than 100,000 GPUs in a few hours, and the P90 time to upgrade one pod fell from more than seven hours early in GB200 bring-up to two hours. Every repaired system sits in quarantine until it passes validation before it rejoins the pool.

The power swings

Synchronous training moves power in lockstep. Meta reports load swings of 30 to 40% as tens of thousands of GPUs flip between compute and communication, and tens of megawatts shifting in seconds when a big job starts or pauses. Its Dynamo Dimmer framework lowers GPU power caps gradually as the cluster nears its electrical limit instead of stopping work, backed by rack batteries, capacitor trays, facility battery storage and coordination with the utility. The back-end network story (a 20,736-GPU building fabric on 51.2T switches, and an aggregation layer spanning up to 100 km) is in the post too, and worth its own read.

What I'd take from it

Most of this scales down to a few dozen racks:

  1. Plan capacity in whole NVL72 domains, with in-rack spares, and make sure your scheduler can express that.
  2. Track MTBI in GPU-hours, then convert it to minutes per job so the checkpoint interval is set from real numbers.
  3. Keep as many faults as possible on a replica-level recovery path, and make checkpoints asynchronous.
  4. Burn in NVLink before production and treat signal integrity as an ongoing telemetry problem.
  5. Expect memory faults to rise with GB300's larger HBM, and plan for scrubbing and predictive retirement.
  6. Measure firmware rollout time per pod. Seven hours versus two matters when the fix targets your top interruption cause.

The caveats: these are Meta's own numbers on Meta's own stack, and the memory mitigations are still rolling out. But a 99% cut in interruption rate, with the failure categories spelled out, is about as close to a public operations runbook for NVL72 as anyone has shared.

Sources

Anpoo Sivanadi, Staff Software Engineer