A first look at NVIDIA Cluster Readiness Engine

I’m a Staff Software Engineer. I spend a lot of time thinking about the gap between “the control plane says this node is fine” and “this cluster can actually finish a distributed GPU workload.” That gap is exactly what NVIDIA Cluster Readiness Engine (NVCRE) targets.

A Kubernetes node can be Ready while the distributed AI stack on it is not. The kubelet is healthy. The API server can reach it. A device plugin advertises nvidia.com/gpu. None of that proves that GPUs across several nodes can form an NCCL communicator, move data at the expected rate, finish a multi-node training job, or tell you which node is at fault when they cannot.

This note is a practitioner reading of the product from the official v0.1.0 README and first-certification guide. It is not a lab write-up of a cluster I ran, and it is not a contribution recap.

What NVCRE is

NVCRE is an Apache-2.0 Go Kubernetes controller that certifies GPU clusters before production workloads land on them. New GPU clusters often contain faulty nodes, and those faults show up under real distributed load. NVCRE runs real training and communication workloads across topology-aware node groups, measures goodput and bandwidth, detects hardware failures, and reports every bad node with a reason. Quarantine stays with your platform: NVCRE never cordons, taints, or otherwise modifies a node.

The audience is platform and infrastructure teams that bring up, validate, or resell GPU clusters.

From the v0.1.0 feature list:

• A certification catalog with NCCL tests and multi-node training
• Platform and GPU architecture detection
• Goodput from training logs via configurable LogProfile patterns
• Per-bus bandwidth from NCCL logs
• CEL node health checks while workloads run
• Per-node failure reporting with a reason
• Topology-aware grouping and adaptive fault isolation
• Checkpoint restart for training jobs
• WorkloadRun for ad-hoc training, NCCL, or custom workloads
• The nvcrectl CLI for setup, run, report, and cleanup

What v0.1.0 actually runs

The first-certification catalog has eight categories. You can list them with kubectl nvcre certification list-categories.

Platform detection covers AWS, GCP, Azure, OCI, nscale, TogetherAI, Mistral, Forge, and on-prem. GPU architecture detection covers GB200, GB300, H100, A100, L40S, and L40. NVCRE tunes workloads from the nvidia.com/gpu.product label and the Node providerID.

How certifications compose

The APIs compose the way Deployment, ReplicaSet, and Pod do. From the README:

Certification ──one Workflow per catalog category──▶ Workflow
WorkloadRun   ──one Workflow from inline spec──────▶ Workflow
                                                     │
                                                     ▼
                                                    Job
                                                     │
                                                     ▼
                                      TrainJob (Kubeflow Trainer adapter)
                                                     │
                              ┌──────────────────────┴──────────────────────┐
                              ▼                                             ▼
                    GoodputMeasurement                          BandwidthMeasurement

A Certification creates one Workflow per catalog category. A WorkloadRun bypasses the catalog: it creates one Workflow directly from an inline workload spec. That is what nvcrectl workloadrun run uses for ad-hoc work. Each Workflow creates a Job. The Job creates the workload through an adapter, for example a Kubeflow Trainer TrainJob. Measurement resources parse pod logs with LogProfile regex patterns and compute goodput and bandwidth.

While those workloads run, CEL expressions watch node health. When a node fails, the certification result records it with a reason. The first-certification guide names three: WorkloadFailed (the workload exited badly), ThresholdViolation (a performance threshold was missed), and HardwareFailureDetected (the node health monitor fired mid-run).

Evidence, not repair

NVCRE stops at evidence. It reports failed nodes. It does not cordon, taint, drain, or repair them. Quarantine is your platform’s job. Because cordoned nodes are skipped on later runs, kubectl uncordon is what brings a repaired node back into the next certification.

The README is also clear about non-goals. NVCRE is not:

• A continuous monitoring system. It watches nodes only while its workloads run.
• A general workload scheduler or production training platform.
• A benchmark leaderboard. Measurements exist to find faults, not to rank hardware.

How to run it

The steps below follow the official README quickstart and Your first certification. Plan for roughly 30 to 60 minutes; most of that is the workload.

Prerequisites

Enable standalone DCGM when you need the level-4 diagnostic:

kubectl patch clusterpolicy cluster-policy --type=merge \
  -p '{"spec":{"dcgm":{"enabled":true}}}'

1. Install the CLI

curl -fsSL https://github.com/NVIDIA/cluster-readiness-engine/releases/latest/download/installer | bash

To pin v0.1.0:

curl -fsSL https://github.com/NVIDIA/cluster-readiness-engine/releases/download/v0.1.0/installer | bash -s -- -v v0.1.0

The installer puts nvcrectl on your $PATH and creates a kubectl-nvcre symlink, so both nvcrectl and kubectl nvcre work.

nvcrectl --version
kubectl nvcre --version

2. Preflight the cluster

kubectl nvcre cluster info
kubectl nvcre setup status

cluster info shows the detected platform, GPU product, ready node count, and topology labels when present. setup status tells you what is still missing. Install the GPU Operator yourself if labels are absent.

3. Install NVCRE

kubectl nvcre setup init

That installs Kubeflow Trainer, the NVCRE CRDs, the controller, and the built-in LogProfiles. Confirm with kubectl nvcre setup status.

Prefer Helm and want an explicit pin to v0.1.0:

helm install nvcre \
  oci://ghcr.io/nvidia/cluster-readiness-engine \
  --version v0.1.0 \
  --namespace nvcre \
  --create-namespace

The controller image is ghcr.io/nvidia/cluster-readiness-engine/manager:v0.1.0.

4. Certify one category

Start with communication/nccl-all-reduce. It works on any node count and finishes in minutes on a healthy cluster:

kubectl nvcre certification run \
  --category communication/nccl-all-reduce \
  --wait

The CLI creates a namespace named nvcrectl-<timestamp> and a Certification in it. Note both names from the output. With --wait, it prints status updates and then the report.

5. Print the report again

kubectl nvcre certification report <name> -n <namespace>

The report command defaults to the default namespace. The run created its own, so leave out -n and you find nothing. That is the common first-day mistake called out in the guide.

The sample report in NVIDIA’s README is example output for an aws / gb300 / 16-node run, not a cluster I operated:

╔════════════════════════════════════════════════════════════════╗
║                      Certification Report                      ║
╚════════════════════════════════════════════════════════════════╝

  Name:      nvcrectl-20260806-162730
  Platform:  aws
  GPU:       gb300
  Nodes:     16

┌────────────────────────────────────────────────────────────────┐
│  communication/nccl-all-reduce                                 │
├────────────────────────────────────────────────────────────────┤
│  Status:    Succeeded                                          │
│  Runtime:   3m 56s                                             │
│  Scale:     full-scale                                         │
│  Nodes/Job: 16                                                 │
│  Jobs:      1                                                  │
│  MNNVL:     Enabled                                            │
│                                                                │
│  Bandwidth:                                                    │
│    Size       AlgBW        BusBW        Samples                │
│    16 GB      473.44 GB/s  932.09 GB/s  9                      │
└────────────────────────────────────────────────────────────────┘

┌────────────────────────────────────────────────────────────────┐
│  Summary                                                       │
├────────────────────────────────────────────────────────────────┤
│  Categories:   1/1 passed                                      │
│  Failed Nodes: none                                            │
│  Result:       PASSED                                          │
└────────────────────────────────────────────────────────────────┘

Pass and fail are informational unless you set thresholds. No thresholds ship by default. To enforce them, use a Certification YAML with options.thresholds and run with --cert-file.

Ad-hoc WorkloadRun

For a single custom run without the catalog, write a WorkloadRun YAML. This is the README’s NCCL all-reduce example:

# nccl-all-reduce.yaml
apiVersion: nvcre.nvidia.com/v1alpha1
kind: WorkloadRun
metadata:
  name: nccl-all-reduce
spec:
  image: nvcr.io/nvidia/pytorch:26.01-py3
  numNodes: 4
  framework:
    mpi:
      binary: /usr/local/bin/all_reduce_perf_mpi
      mpirunPath: /usr/local/mpi/bin/mpirun
      args: ["-b", "8", "-e", "32G", "-f", "2", "-n", "100"]
  bandwidthMeasurement:
    logProfileRef: nccl-bandwidth
    testType: all_reduce

numNodes is the node count per job group, not the cluster total. On a 16-node cluster, numNodes: 4 produces four 4-node jobs.

kubectl nvcre workloadrun run nccl-all-reduce.yaml --wait

Cleanup

kubectl delete namespace <namespace>
kubectl nvcre setup reset

setup reset also removes Kubeflow Trainer. Pass --skip-phases=deps to keep it.

Related reading

Saiyam Pathak published a hands-on lab the day after the v0.1.0 cut: NVIDIA Cluster Readiness Engine on vCluster Private Nodes. He ran NVCRE inside a tenant cluster backed by two A100 workers joined as vCluster Private Nodes. That is his lab and his report. I am not claiming his cluster, bandwidth numbers, GKE setup, or PASSED result as mine. Read it if you want an end-to-end private-node burn-in path on top of the same engine described here.

References

• NVIDIA/cluster-readiness-engine
• v0.1.0 release (1 September 2026)
• Your first certification
• Run NCCL Benchmarks
• Nemotron 5 WorkloadRun guide
• Saiyam Pathak: NVCRE on vCluster Private Nodes