A first look at NVIDIA Cluster Readiness Engine

I’m a Staff Software Engineer. I spend a lot of time thinking about the gap between “the control plane says this node is fine” and “this cluster can actually finish a distributed GPU workload.” That gap is exactly what NVIDIA Cluster Readiness Engine (NVCRE) targets.

A Kubernetes node can be Ready while the distributed AI stack on it is not. The kubelet is healthy. The API server can reach it. A device plugin advertises nvidia.com/gpu. None of that proves that GPUs across several nodes can form an NCCL communicator, move data at the expected rate, finish a multi-node training job, or tell you which node is at fault when they cannot.

This note is a practitioner reading of the product from the official v0.8.2 README, install and first-certification guides, catalog, certify-a-cluster docs, platform-detection, MCP and certification CLI references, metrics, deployment guide, CHANGELOG, and the v0.8.1 and v0.8.2 release notes. It is not a lab write-up of a cluster I ran, and it is not a contribution recap.

What NVCRE is

NVCRE is an Apache-2.0 Go Kubernetes controller that certifies GPU clusters before production workloads land on them. New GPU clusters often contain faulty nodes, and those faults show up under real distributed load. NVCRE runs real training and communication workloads across topology-aware node groups, measures goodput and bandwidth, detects hardware failures, and reports every bad node with a reason. Quarantine stays with your platform: NVCRE never cordons, taints, or otherwise modifies a node.

The audience is platform and infrastructure teams that bring up, validate, or resell GPU clusters.

From the v0.8.2 README and release notes:

• A certification catalog with NCCL tests and multi-node training
• Platform and GPU architecture detection, including B300 and RTX PRO 6000 Blackwell
• GPU architecture from DRA gpu.nvidia.com ResourceSlices when no nvidia.com/gpu.product label exists
• Goodput from training logs via configurable LogProfile patterns
• Per-bus bandwidth from NCCL logs, plus record-only BandwidthMeasurement.status.transport (IB or Socket)
• CEL node health checks while workloads run
• Per-node failure reporting with a reason, plus failedNodeDetails on the report
• Topology-aware grouping and adaptive fault isolation
• Checkpoint restart for training jobs
• Optional spec.gangScheduler for gang-aware placement
• Queue label on the submitted workload object, plus workloadMetadata.labels
• Workload environment variables on a Certification, globally at spec.env and per category at options.env
• Cordoned-node opt-in via target.taintSelectors
• Phase-transition Events on Certification, Workflow, Job, and WorkloadRun
• WorkloadRun for ad-hoc training, NCCL, or custom workloads, including orchestration.placement: Unpinned
• The nvcrectl CLI for setup, render, run, report, cleanup, read-only mcp serve, and rbac print
• nvcrectl setup init takes chart values (--values, --set, --set-json) and keeps them across runs
• Optional namespaced controller write access (rbac.scope)
• Configurable DCGM host for diagnostics/dcgm-level4
• Mokka local environment for running NVCRE without GPUs
• nvcre_certification_status, nvcre_workflow_status, and nvcre_job_status gauges (timed-out Jobs report failed)

What v0.8.2 actually runs

v0.8.2 is Latest (21 October 2026). v0.8.1 shipped the day before. Neither patch adds a catalog category, a platform, an nvcrectl command, or an API. The catalog, how Certification / WorkloadRun / Workflow / Job compose, and the signed-installer plus digest-pinned Helm path are the same as v0.8.0. What the patches actually change: the MPI sshd bootstrap is bounded and retried; Certification spec.imagePullSecrets now renders on every catalog entry, launcher included; images and binaries are built with Go 1.27.3.

The first-certification catalog still has eight categories. You can list them with kubectl nvcre certification list-categories. The categories themselves are unchanged from v0.8.0 (and from v0.7.0). ADR-091’s per-GPU GEMM entry is still a design document, not a shipped catalog category, so it is not in the table below. How those same entries are configured is what v0.8.0 already shipped: a Certification can set workload environment variables, dcgm-level4 can point at a DCGM host other than the GPU Operator default, the Nemotron 5 entries pick their attention kernel through --attention-backend, and NVLS is no longer force-enabled in the base NCCL environment.

Platform detection covers AWS, GCP, Azure, OCI, nscale, TogetherAI, Mistral, Forge, and on-prem. GPU architecture detection covers GB200, GB300, H100, H200, B200, B300, A100, L40S, L40, and RTX PRO 6000 Blackwell. Catalog overrides name GB200, GB300, H100, H200, B200, and B300. NVCRE tunes workloads from the nvidia.com/gpu.product label and the Node providerID. On a DRA GPU stack that never wrote that product label, it can also read gpu.nvidia.com ResourceSlices. On-prem is the fallback when providerID is empty. On-prem GB200/GB300 still get the override from v0.3.0: arm64 and GPU-present taint tolerations, portable InfiniBand NCCL env without HCA pinning, and a NIC resource request that is auto-detected when exactly one candidate (rdma/* or nvidia.com/mlnxnics) is allocatable at the resolved mlnxPerNode count on every target node. On-prem x86 HGX B200/B300 get a smaller override: GPU-taint toleration (any value, including on the MPI launcher) and the same NIC request, with no NCCL environment. Set nicResourceName when detection is ambiguous, or for offline nvcrectl certification render without --dry-run. Nodes with a metal3:// providerID detect as Mistral, not on-prem; pass --platform onprem on render if you need the generic override. An x86 HGX site provisioned that way matches neither the Mistral block (gated on GB300) nor the HGX block (gated on onprem).

v0.6.0 added GCP overlays for RTX PRO 6000 Blackwell Server Edition on NCCL all-reduce, all-gather, all-to-all, and Nemotron 5 8B, with an eight-GPU architecture default. Those remain. Smaller G4 node shapes need an explicit gpusPerNode. Without that, nodes advertising fewer than eight allocatable GPUs are excluded by the capacity filter.

The Nemotron 5 training entries now select the attention kernel through --attention-backend ${ATTENTION_BACKEND:-flash} instead of pinning the NVTE_*_ATTN variables. megatron-core derives those three from --attention-backend and asserts when one is already set to a conflicting value, so on AWS H100 and GB200 every rank failed in GPTModel.__init__ before the first step and JobSet restarted the job indefinitely. The AWS H100 and GB200 overrides now set ATTENTION_BACKEND=auto, which keeps the fused kernel they were after. Every other platform still renders flash.

The base NCCL environment no longer sets NCCL_NVLS_ENABLE=1. NCCL’s own default of 2 probes CU_DEVICE_ATTRIBUTE_MULTICAST_SUPPORTED; at 1 the probe is skipped and the NVLS path is entered on hardware that may not support multicast, and on NCCL older than 2.29 a failed multicast bind aborts the job where the default logs a fallback and proceeds. Four fragments with no recorded measurement behind the value also stop setting it: on-prem InfiniBand, TogetherAI, and the GB200 base and training profiles. On-prem GB200/GB300, TogetherAI, OCI GB200, and Azure GB200 renders lose the variable. Azure H100, AWS GB300, Nscale, and GCP H100 keep it and now cite the reason. A site that needs NVLS off still sets it through spec.env or mpiArgs.

Each NCCL variable has one owning fragment on OCI. Three OCI arms used to forward a variable to mpirun twice, and on the GB300 arms the two values disagreed: NCCL_IB_SPLIT_DATA_ON_QPS was sent as both 0 and 1. The shared profile is split at the point GB300 diverges from it, so every existing consumer renders the same names and values, and six duplicate pairs disappear. ADR-100 is the contract.

Certifiable platforms

v0.7.0 added three places to certify that remain in v0.8.2: Nscale NKS, x86 HGX B300, and OCI GB200. OCI GB300, GCP H100 image selection, and on-prem HGX B200 ship in that same set.

Nscale NKS is platform nscale, detected from the nscale:// providerID, on B200 and GB300. Catalog entries and the WorkloadRun path claim GPUs through a gpu.nvidia.com ResourceClaimTemplate instead of nvidia.com/gpu, and RDMA NICs through an rdma.nscale.com claim sized from mlnxPerNode (8 on B200, 4 on GB300, omitted at 0). diagnostics/dcgm-level4 (hostNetwork) and communication/nccl-loopback-nvswitch (NVLink only) claim GPUs alone; level-4 talks to nvidia-dcgm-dra.nvidia-platform.svc:5555 unless dcgmHost is set. The NCCL profile is InfiniBand with no HCA names pinned and NCCL_NET left unset, so the HPC-X plugin keeps SHARP rather than NCCL’s built-in IB transport. Nscale runs no GPU Feature Discovery, so GB300 uses topology.nks.nscale.com/accelerator-domain wherever other platforms use nvidia.com/gpu.clique. nvcrectl cluster info counts GPUs from ResourceSlices on nodes that advertise no nvidia.com/gpu. v0.6.0 detected Nscale from openstack:// plus an nscale.com/rdmashare allocatable; that path is gone, so those nodes now get on-prem overlays. Detail is in ADR-085.

The x86 HGX B300 architecture is b300, product label NVIDIA-B300-SXM6-AC. The catalog defaults table sizes it at eight GPUs and eight NICs per node, --gpu-arch b300 is accepted, and the Nemotron 5 8B and 56B entries carry a b300 parallelism row. An on-prem override for x86 HGX B200/B300 tolerates the nvidia.com/gpu taint with any value, including on the MPI launcher, and makes nicResourceName and mlnxPerNode request RDMA devices, with NIC auto-detection widened to those four architectures. It sets no NCCL environment: an HGX chassis has no multi-node NVLink, so the NVL72 tuning does not apply. A missing architecture falls back to four GPUs and no NICs; the controllers emit a Warning, reason GPUArchitectureDefaults, and every nvcrectl render path plus nvcrectl cluster info print the same message. ADR-094 is the design.

OCI GB200 no longer requests nvidia.com/mlnxnics by default. The platform default is mlnxPerNode: 0, and zero now omits the resource and the k8s.v1.cni.cncf.io/networks annotation instead of rendering a count of zero. A site that runs the SR-IOV device plugin sets its own count. ADR-087 is the opt-out. The five NCCL entries and both Nemotron training entries render the RoCEv2 GID index those queue pairs need (NCCL_IB_GID_INDEX=3, and UCX_IB_GID_INDEX=3 on the MPI launcher). Clusters that expose Mellanox devices only as host nodes under /dev/infiniband set rdmaDeviceAccess: hostPath on the Certification, or per category under options. That mounts the directory and runs the worker privileged; unset still means devicePlugin. ADR-088 covers the mount.

OCI GB300 ships an NCCL topology file in a ConfigMap mounted at /etc/nccl and sets NCCL_TOPO_FILE to it, so each GPU pairs with its own rail. It also stops forcing NCCL_DMABUF_ENABLE=0 when MNNVL is off.

GCP H100 reads the GKE network names from the target nodes instead of assuming gpu-nic0 through gpu-nic7, and picks the NCCL and training images and the tcpxo-daemon from the TCPXO plugin version on those nodes. The release notes say that path was validated on an AICR A3 Mega cluster at plugin v1.0.15 and v1.0.16 with no overrides. An explicit image still replaces the chosen one; set it per category under categories[].options.image so a mixed certification does not also replace diagnostics/dcgm-level4’s DCGM image. I did not run that cluster.

GPU architecture without a product label

Every architecture decision used to read nvidia.com/gpu.product, which GPU Feature Discovery writes. A cluster that claims GPUs through Dynamic Resource Allocation runs no GFD, so no node carried the label and certification failed with cannot determine GPU architecture. From v0.6.0 the architecture can come from the gpu.nvidia.com ResourceSlices the DRA driver publishes, normalized to the exact string GFD would have written, so a mixed GFD and DRA fleet still passes the homogeneity check. Nscale NKS still uses that path. Nothing is written back to the Node. nvidia.com/gpu.present=true is still required; a cluster running only the standalone DRA driver has to set that label itself.

The manager role gains get/list/watch on resource.k8s.io resourceslices. A missing grant leaves those nodes unlabeled rather than failing the reconcile. Offline nvcrectl certification render and workloadrun render have no cluster to read slices from, so they take --gpu-arch (for example gb300, b300, or NVIDIA-GB300). That flag cannot be combined with --dry-run, which detects from real nodes. With --dry-run, NIC detection (on-prem GB200/GB300 and HGX B200/B300) and GKE TCPXO network and plugin version detection (GCP H100) run as they do in the controller.

nvcrectl certification run --category no longer persists the GPU product it detected into spec.target.nodeSelector. Only nvidia.com/gpu.present: "true" is stored, so node discovery is not pinned to what nodes reported at creation time. That also matches DRA-only nodes.

How certifications compose

The APIs compose the way Deployment, ReplicaSet, and Pod do. From the README:

Certification ──one Workflow per catalog category──▶ Workflow
WorkloadRun   ──one Workflow from inline spec──────▶ Workflow
                                                     │
                                                     ▼
                                                    Job
                                                     │
                                                     ▼
                                      TrainJob (Kubeflow Trainer adapter)
                                                     │
                              ┌──────────────────────┴──────────────────────┐
                              ▼                                             ▼
                    GoodputMeasurement                          BandwidthMeasurement

A Certification creates one Workflow per catalog category. A WorkloadRun bypasses the catalog: it creates one Workflow directly from an inline workload spec. That is what nvcrectl workloadrun run uses for ad-hoc work. Each Workflow creates a Job. The Job creates the workload through an adapter, for example a Kubeflow Trainer TrainJob. Measurement resources parse pod logs with LogProfile regex patterns and compute goodput and bandwidth. Bandwidth threshold verdicts are judged on the launcher’s complete log, read once when the Job ends, rather than on a partial NCCL size sweep. BandwidthMeasurement.status.transport records which NCCL network carried the collective, IB or Socket. The report and --results-file show it per category, group, and clique. Record-only: no verdict reads it.

While those workloads run, CEL expressions watch node health. When a node fails, the certification result records it with a reason. The first-certification guide names three terminal ones: WorkloadFailed (the workload exited badly), ThresholdViolation (a performance threshold was missed), and HardwareFailureDetected (the node health monitor fired mid-run). A scheduling stall is a different state: the run is still InProgress, reported as blocked rather than running.

v0.8.0 records that same per-node reason and message on the report as failedNodeDetails, next to the failedNodes name list that keeps its shape. A timed-out Job that actually ran now seeds failed nodes; it used to report none. A Job that timed out while its pods were unschedulable still records nothing, because nothing ran on those nodes. A CEL detector’s own message survives instead of being replaced by one constant. NVCRE still relays the attribution it already computed; it does not classify a fabric fault from log text. A failed bandwidth category that recorded none on a multi-node group prints a next-step hint naming testScale: diagnose, which bisects the group to separate a node-local fault from one on the path between nodes. ADR-099 is the design.

Unpinned placement

Default WorkloadRun placement is Pinned: every eligible node is partitioned into groups of numNodes, and each job carries a required kubernetes.io/hostname affinity naming its group. On a 16-node cluster, numNodes: 4 still produces four 4-node jobs.

orchestration.placement: Unpinned on a WorkloadRun, or on a hand-written Workflow, runs a single job of exactly numNodes nodes and leaves the rest of the target untouched. The scheduler places the pods inside the target. The size is honored or the run fails with reason PartitionError; it is never clamped or auto-selected. Pair it with testScale: full-scale when you want one job. intra-node and intra-rack are rejected beside it. The report prints a Placement line with how many of the target nodes were exercised, and no Scale line.

spec:
  numNodes: 4
  orchestration:
    testScale: full-scale
    placement: Unpinned

A Certification has no placement field on purpose. Its verdict covers every node it targets: a PASSED certification means every node you pointed at was tested. Unpinned on that tier would be the first way a Certification could report PASSED while leaving targeted nodes untested. WorkloadRun makes no coverage claim; it reports whether the workload passed. On a contended cluster a multi-node Unpinned job should set spec.gangScheduler so the gang schedules whole or not at all. In both modes the target selector now travels onto the pods as a required node affinity, with an nvidia.com/gpu.product In term built from the products discovery kept. ADR-089 is the contract; the WorkloadRun quick start shows the single-job YAML.

Phase transitions as Events

Certification, Workflow, Job, and WorkloadRun each emit a Kubernetes Event when a phase becomes true. A transition into that resource’s Failed phase is a Warning; every other phase is Normal. kubectl describe on the object shows the run’s history without anyone reading controller logs.

Blocked versus running

A Job whose pods the scheduler has rejected (PodScheduled=False, reason Unschedulable) for longer than spec.schedulingStallGraceSeconds (default five minutes) reports WorkloadSchedulingBlocked with the scheduler’s own diagnosis. The Workflow mirrors it as JobSchedulingBlocked, and nvcrectl certification report prints a Blocked: line (JSON statusDetail) on a Running category. SchedulingGated pods are held on purpose and do not count. Previously the Job reported WorkloadRunning while those pods sat Pending.

The blocked interval pauses timeoutPerJob and the startup and training stall budgets, so a run no longer times out against hardware that never ran. A single blocked episode longer than timeoutPerJob still times the Job out. On recovery the runtime used before the block still counts. startupStallTimeoutSeconds on the Certification, or per category under options, sets the first-step budget the training entries previously hardcoded at 1200 s. nvcrectl certification run --startup-stall-timeout-seconds sets the spec-level value and is rejected with --cert-file.

Gang scheduling

Distributed workloads can deadlock under the default scheduler when only some of their pods fit: the placed pods hold GPUs while waiting for peers that never arrive. Optional spec.gangScheduler on Certification opts every category’s resolved TrainingRuntime pod templates into a gang-aware scheduler such as KAI Scheduler. The scheduler holds the pods until the full gang can be placed.

schedulerName is required. queue is optional and defaults to default-queue. queueLabelKey is optional and defaults to kai.scheduler/queue. The rewrite is certification-wide: after catalog and platform overrides resolve, it injects the scheduler name into every pod template and writes the queue value under that label key on the submitted workload object (the TrainJob), on each replicated job’s template metadata, and on its pod template metadata — including when a catalog entry hardcodes default-scheduler. Kueue and KAI Scheduler read the submitted object before any pod exists.

Under KAI Scheduler, the MPI launcher dependsOn gate and the JobSet startupPolicy are dropped, so both sub-groups are populated and the gang is admitted. The ordering barrier moves into the launcher pod, which waits for every worker to answer sshd. Non-KAI paths render byte-identically.

apiVersion: nvcre.nvidia.com/v1alpha1
kind: Certification
metadata:
  name: gang-scheduled-cert
spec:
  target:
    nodeSelector:
      nvidia.com/gpu.product: NVIDIA-GB200
  enableMNNVL: true
  gangScheduler:
    schedulerName: kai-scheduler   # required
    queue: high-priority           # optional; defaults to "default-queue"
    # queueLabelKey: kai.scheduler/queue   # optional; this is the default
  categories:
    - domain: communication
      variant: nccl-all-reduce
    - domain: training
      variant: nemotron5-8b

On NVIDIA Run:ai, name that scheduler and its queue label, and set queue to an existing Run:ai queue — the default default-queue is not a real Run:ai queue:

  gangScheduler:
    schedulerName: runai-scheduler
    queueLabelKey: runai/queue
    queue: team-a   # must name an existing Run:ai queue

nvcrectl certification render applies the same rewrite, so the rendered manifests show what the controller will create. It also names Workflows with the options suffix the controller uses (nodesPerJob, enableMNNVL, testScale, enableCheckpoint, maxSteps, exitDurationMins, gpusPerNode, and rdmaDeviceAccess), so a rendered name matches what the controller creates. Full detail is in the certify-a-cluster guide.

The same submitted object can carry arbitrary labels via workloadMetadata.labels, set globally at spec.workloadMetadata or per category at spec.categories[].options.workloadMetadata. These are not pod labels. Labels merge per key: catalog, then global, then per-category. The gang-queue insert does not overwrite a different value already on that key — the run fails instead. Cap is 32 labels after merge.

spec:
  workloadMetadata:
    labels:
      environment: burn-in
  categories:
    - domain: communication
      variant: nccl-all-reduce
      options:
        workloadMetadata:
          labels:
            kueue.x-k8s.io/queue-name: nccl-queue

That nccl-all-reduce category’s TrainJob receives both environment: burn-in and kueue.x-k8s.io/queue-name: nccl-queue. A category can change or add a key but cannot remove a global one.

Workload environment on a Certification

A Certification can now set environment variables on the workload it runs. WorkloadRun already had spec.env. From v0.8.0 the same idea exists on the catalog path: globally at spec.env and per category at spec.categories[].options.env. The two levels merge per name rather than replacing wholesale, so a category that adds one variable does not drop a site-wide one. A category can change or add a name but cannot remove a global one. Cap is 32 entries. valueFrom is rejected, because a variable reaches MPI ranks by value on a command line.

Where a variable lands depends on how the entry launches its ranks. The three communication/nccl-* MPI entries get -x NAME=VALUE pairs in trainer.args, because their ranks start under sshd with a fresh environment and a trainer env var would never reach them. Every other entry gets a trainer env var. An MPI entry with no -x pair to anchor an insertion fails rather than guessing. An entry pushed past Kubeflow’s 128-item trainer.args cap is named instead of producing a TrainJob the API server refuses. A Certification that sets nothing renders byte-identically to before. The resolved variable names, never their values, print on the category card.

spec:
  env:
    - name: NCCL_NVLS_ENABLE
      value: "0"
  categories:
    - domain: communication
      variant: nccl-all-reduce
      options:
        env:
          - name: NCCL_IB_HCA
            value: mlx5_0,mlx5_1

Catalog and platform overrides resolve first. Then spec.env replaces by name, then the per-category options.env. An NCCL_MNNVL_ENABLE set in env wins over enableMNNVL, because that field interpolates into an ordinary catalog value before env runs. ADR-097 is the design.

Cordoned node burn-in

Cordoned nodes are skipped by default. To certify hardware you have held out of production, add a target.taintSelectors entry for the node.kubernetes.io/unschedulable taint with effect NoSchedule. That opts those nodes in and clears nodeHealthMonitor for the run, so a cordon is not reported as HardwareFailureDetected.

A cordon is often the right hold-out: DaemonSets already tolerate unschedulable, so kube-proxy, CNI, and GPU Operator operands stay scheduled without extra taint tolerations.

apiVersion: nvcre.nvidia.com/v1alpha1
kind: Certification
metadata:
  name: gpu-node-1-certification
spec:
  target:
    nodeSelector:
      nvidia.com/gpu.product: NVIDIA-GB200
    nodeNames:
      - gpu-node-1
    taintSelectors:
      - key: node.kubernetes.io/unschedulable
        effect: NoSchedule
  nodesPerJob: 1
  enableMNNVL: true
  categories:
    - domain: communication
      variant: nccl-loopback

Caveats from ADR-081: taintSelectors is an AND, so listing the unschedulable taint also excludes every uncordoned node — one Certification cannot target both. Clearing the health monitor means no node in that run can be marked HardwareFailureDetected. Selection requires the listed taints to already be present; a node cordoned but not yet carrying the taint is excluded. Listing that taint also drops the blanket taint toleration workload pods get by default, so an extra unrelated taint on a selected node can leave pods Pending.

Air-gapped and mirrored installs

The controller itself needs no outbound network at runtime: catalog entries are compile-time embedded. Workloads and install still pull images and charts unless you point them at a mirror.

nvcrectl setup init can pull both Helm charts from a registry the cluster can reach:

kubectl nvcre setup init \
  --chart-ref oci://registry.example.com/mirror/cluster-readiness-engine \
  --trainer-chart-ref oci://registry.example.com/mirror/kubeflow-trainer \
  --image registry.example.com/mirror/manager:v0.8.2

--chart-ref is used for the release install and for CRD extraction (helm show crds). The mirror must still host the NVCRE chart at this CLI version and Kubeflow Trainer at 2.2.1. Authenticate a non-GHCR mirror with helm registry login before init; --image-pull-secret is scoped to ghcr.io. From v0.8.0, --image with a digest sets manager.image.digest, so the chart renders repo@sha256:… rather than the invalid repo:sha256:… tag it used to. The flag cannot be combined with values that set manager.image.repository, .tag, or .digest on the same run.

When no registry is reachable at all, the deployment docs cover preloading images into each node’s container runtime instead of pulling them: export the tar, serve or copy it onto the nodes, import with ctr -n k8s.io images import, and reference an explicitly tagged image. Never :latest: that tag defaults to Always and would send the kubelet to a registry this cluster cannot reach.

Catalog workloads can override the container image. Set image on the Certification spec, or per category under categories[].options (per-category wins). The override replaces the trainer image, the primary workload container, and init containers whose image matched the primary before the override (fix-ssh-permissions, megatron-clone). Distinct inits such as GCP’s tcpxo-daemon keep their catalog image. On AWS H100 and GB200, the platform override already lands NCCL workers on an nccl-tests image that ships the EFA OFI plugin; replacing that image means you own the plugin. Pair the image with spec.imagePullSecrets when the registry is private. Until v0.8.2 that field reached only the two training categories: the five communication entries and diagnostics/dcgm-level4 rendered no pull secret on any pod. Every catalog entry now renders it on its node pods, and the three multi-node NCCL entries render it on the launcher too, because the launcher runs the same image. Unset, the render is unchanged. nvcrectl certification run --workload-registry creates a Secret and appends it to that list; before v0.8.2 the pods still could not pull.

AWS H100 and GB200 keep the EFA userspace stack. Earlier overrides matched platform: aws with no GPU guard and stripped /opt/amazon on every AWS architecture, which sent NCCL to TCP over eth0 while EFA devices stayed attached unused. That cleanup is now scoped to GB300 (RoCE, no EFA). H100 and GB200 get FI_PROVIDER=efa, FI_EFA_USE_DEVICE_RDMA=1, FI_EFA_FORK_SAFE=1, and IPC_LOCK on both the catalog and WorkloadRun paths.

Multi-node MPI workers install openssh-server with apt-get unless /usr/sbin/sshd is already executable. Prebake sshd into the workload image for air-gapped clusters; the install then no-ops. From v0.8.1 the install is bounded and retried: apt-get update, apt-get install, and the retry’s dpkg --configure -a each run under timeout -s KILL 120, three attempts, with Acquire::Retries=3. A hung apt-get update used to sit until timeoutPerJob; now one attempt costs at most four minutes, a retry at most six, and three exhausted attempts sixteen. If all three fail, the worker exits with sshd install attempt 3 of 3 failed or timed out. An image that needs the install but has no apt-get or timeout exits at once. Images that already ship /usr/sbin/sshd still skip apt. Those workers still request SYS_CHROOT alongside IPC_LOCK, and the readiness probe opens a real loopback SSH session instead of a TCP connect to port 22. On CRI-O, which does not grant SYS_CHROOT by default, every SSH session used to die preauth while the listener stayed up and mpirun failed with the generic ORTE banner. An admission policy that pins the capability list needs to allow the new one. An image without an ssh client keeps the previous listening check. Loopback NCCL categories are single-node and have no sshd bootstrap. AWS H100/GB200 (nccl-tests) and AWS GB300 (command replacement) already skip the install. ADR-093 is the design.

On the mpi framework a WorkloadRun’s spec.volumes, spec.volumeMounts, spec.config, and spec.initContainers were silently dropped: the rendered TrainingRuntime carried none of them while torch and exec honored them. They now land on the MPI worker pods, where the ranks run, and spec.config is also mounted read-only on the launcher pod, where mpirun runs. A new mpi WorkloadRun that already set any of them now mounts them, so a volume that cannot attach, such as a ReadWriteOnce claim across nodes, holds the pods in ContainerCreating where it used to be ignored. Existing objects are unaffected because spec is immutable. The launcher pod also receives spec.imagePullSecrets, which only the workers used to get. Reusing a built-in volume name or mount path (dshm, /dev/shm, config-volume, /config, and on MPI mpi-ssh-auth) fails with BuildFailed naming the entry, and nvcrectl workloadrun render refuses it.

Training categories resolve Megatron-LM at pod start: an existing workspace at /mnt/workspace/megatron-lm, then source shipped in the image at /opt/megatron-lm, then a clone. The entry-defined default is https://github.com/NVIDIA/Megatron-LM.git, branch core_v0.15.2. For an internal Git mirror, set sourceRepo on the spec or per category (https:// or ssh:// only). Baking the pinned checkout into the image is the documented first choice when workers have no GitHub egress.

spec:
  sourceRepo: https://git.example.com/mirrors/Megatron-LM.git

Evidence, not repair

NVCRE stops at evidence. It reports failed nodes. It does not cordon, taint, drain, or repair them. Quarantine is your platform’s job. Cordoned nodes are still skipped by default. Opting them in with taintSelectors is a targeting choice; NVCRE still never cordons, taints, or otherwise modifies a node itself. After you repair one, kubectl uncordon is what brings it back into a default (no-taintSelector) run.

The README is also clear about non-goals. NVCRE is not:

• A continuous monitoring system. It watches nodes only while its workloads run.
• A general workload scheduler or production training platform.
• A benchmark leaderboard. Measurements exist to find faults, not to rank hardware.

Run it locally without GPUs

From v0.8.0 NVCRE ships Mokka, a Kind environment that simulates NVML on ordinary workers so the device plugin, GPU Feature Discovery, and DRA driver run, and NVCRE’s pods actually schedule and bootstrap MPI. Nothing in it runs CUDA. Bring it up with make mokka-up PROFILE=<h100|b200|gb200|gb300>. Kind gives every node the provider ID kind://, so NVCRE detects the platform as onprem. Cloud-specific overrides never apply. Real NCCL needs CUDA, so the environment provides a stub workload image that replays a recorded nccl-tests log. A Certification that passes here says nothing about CUDA, bandwidth, goodput, or DCGM. Those still need hardware. The design is ADR-096; the run locally without GPUs guide is the walkthrough. I did not run that cluster.

How to run it

The steps below follow the official README quickstart and Your first certification. Plan for roughly 30 to 60 minutes; most of that is the workload.

Prerequisites

Enable standalone DCGM when you need the level-4 diagnostic:

kubectl patch clusterpolicy cluster-policy --type=merge \
  -p '{"spec":{"dcgm":{"enabled":true}}}'

By default dcgm-level4 talks to nvidia-dcgm.gpu-operator.svc:5555 (on Nscale NKS, nvidia-dcgm-dra.nvidia-platform.svc:5555). To use a DCGM Service in another namespace, or on another port, set dcgmHost to host[:port] on the spec or per category. Each per-node Job must reach the hostengine on its own node; a Service that load-balances across nodes reports the wrong node’s GPUs. nvcrectl setup status still checks only the GPU Operator’s gpu-operator/nvidia-dcgm Service, so it reports DCGM as missing when only your own endpoint exists.

categories:
  - domain: diagnostics
    variant: dcgm-level4
    options:
      dcgmHost: nvidia-dcgm.gpu-system.svc:5555

1. Install the CLI

v0.8.2 artifacts are signed with Sigstore and carry SLSA provenance. The release verify gate refuses a cosign_version below v3.1.3. Preferred path: download the installer with its Sigstore bundle, verify with cosign, then run it pinned to the tag. Digests and exact cosign identity strings live on the v0.8.2 release.

REL=https://github.com/NVIDIA/cluster-readiness-engine/releases/download/v0.8.2

curl -fsSLO "${REL}/installer"
curl -fsSLO "${REL}/installer.sigstore.json"

cosign verify-blob-attestation \
  --bundle installer.sigstore.json \
  --type https://slsa.dev/provenance/v1 \
  --certificate-identity "https://github.com/NVIDIA/cluster-readiness-engine/.github/workflows/attest.yml@refs/tags/v0.8.2" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  installer

bash installer -v v0.8.2

Piping releases/latest or a pin-by-tag installer into bash still works, but it is weaker: the script runs before anything has checked the script itself. TLS proves you reached GitHub; it does not prove the asset was untampered.

# weaker: newest stable, no local cosign check of the installer itself
curl -fsSL https://github.com/NVIDIA/cluster-readiness-engine/releases/latest/download/installer | bash

# weaker: pin by tag, still pipes into bash
curl -fsSL https://github.com/NVIDIA/cluster-readiness-engine/releases/download/v0.8.2/installer | bash -s -- -v v0.8.2

The installer puts nvcrectl on your $PATH and creates a kubectl-nvcre symlink, so both nvcrectl and kubectl nvcre work. From this release, the installer also verifies each binary against that release’s Sigstore bundle before installing it (unless you pass --skip-verify).

nvcrectl --version
kubectl nvcre --version

2. Preflight the cluster

kubectl nvcre cluster info
kubectl nvcre setup status

cluster info shows the detected platform, GPU product, ready node count, and topology labels when present. setup status tells you what is still missing. Install the GPU Operator yourself if labels are absent.

3. Install NVCRE

kubectl nvcre setup init

That installs Kubeflow Trainer 2.2.1, the NVCRE CRDs, the controller, and the built-in LogProfiles. Confirm with kubectl nvcre setup status.

From v0.8.0, setup init takes chart values for the NVCRE release with --values/-f, --set, and --set-json, in Helm’s syntax and precedence, and keeps the values set on the installed release across later runs. A memory limit raised for a large cluster now survives an automated re-run. The controller image’s repository, tag, and digest are the exception: they are worked out on every run from this run’s values, --image, or the CLI version, so a digest pinned on an earlier run cannot hold the controller on an old image after an upgrade. --reset-values drops the kept values except rbac.scope and rbac.runNamespaces. -f - needs --auto-approve, and URLs are rejected. Before the confirmation prompt, and with --auto-approve too, preflight prints the paths and sources of the values the release will be given, without their contents. ADR-098 is the contract.

kubectl nvcre setup init \
  --set manager.resources.requests.memory=4Gi \
  --set manager.resources.limits.memory=8Gi

Prefer Helm and want an explicit pin to this release’s chart digest. The v0.8.2 signature covers digest sha256:75458557b0ff2610c7eea01e0b315ae0355cc6b2baa816756a2dba0c0c2387a4 at ghcr.io/nvidia/cluster-readiness-engine. The version tag is mutable, so verify and install the same digest:

CHART=ghcr.io/nvidia/cluster-readiness-engine
DIGEST=sha256:75458557b0ff2610c7eea01e0b315ae0355cc6b2baa816756a2dba0c0c2387a4

# 1. Verify the chart. Requires cosign; the package is public, so no login is needed.
cosign verify \
  --certificate-identity "https://github.com/NVIDIA/cluster-readiness-engine/.github/workflows/attest.yml@refs/tags/v0.8.2" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  "${CHART}@${DIGEST}"

# 2. Pull that exact digest, then install the file you just verified.
helm pull "oci://${CHART}@${DIGEST}"
helm install nvcre ./cluster-readiness-engine@${DIGEST/:/-}.tgz \
  --namespace nvcre --create-namespace

helm install --version v0.8.2 resolves the tag at install time, which is not provably the digest you verified. helm install --verify looks for a PGP .prov file; this project publishes none.

The controller image is ghcr.io/nvidia/cluster-readiness-engine/manager:v0.8.2. This release signed these exact digests; pin the index rather than the tag:

IMAGE=ghcr.io/nvidia/cluster-readiness-engine/manager
INDEX=sha256:c5de16ddb95bd923c6b04d04d7ff9754a7d7b83d18fa400baa33e7ae72fc593d

cosign verify \
  --certificate-identity "https://github.com/NVIDIA/cluster-readiness-engine/.github/workflows/attest.yml@refs/tags/v0.8.2" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  "${IMAGE}@${INDEX}"

To require that same signature and SLSA provenance at admit time, this release still ships two samples under config/samples/policy/: a Kyverno ImageValidatingPolicy and a Sigstore policy-controller ClusterImagePolicy. Both check the controller image before a Pod is admitted. From v0.4.0 onward, the attest self-test is gated to main, and attest.yml refuses allow_untagged on a release tag. Release-path tests still refuse a weakened attest.yml signing boundary. The signed THIRD_PARTY_NOTICES.md asset matches the modules compiled into nvcrectl.

Controller write access per run namespace

The controller’s permissions are split between nvcre-manager-role, still bound cluster-wide, and nvcre-run-role. Chart value rbac.scope decides where the controller may write. cluster, the default, keeps the grant earlier releases made. namespaced grants every write to a namespaced resource outside NVCRE’s API group only in namespaces with a RoleBinding nvcre-manager-run to nvcre-run-role. rbac.runNamespaces renders that binding for long-lived namespaces. The default becomes namespaced in the next minor release; to keep today’s grant then, set rbac.scope: cluster explicitly (nvcrectl setup init --rbac-scope cluster, or --set rbac.scope=cluster with Helm). Later setup init runs keep it.

A Workflow checks the controller’s access in its namespace with SelfSubjectAccessReviews before writing anything there. Without access it stays InProgress with reason NamespacePermissionsMissing (or NamespaceAccessCheckFailed when the review cannot run), emits a Warning naming the RoleBinding to create, and proceeds on its own once the binding exists. The wait is bounded by manager.namespaceAccessTimeout (default 10 minutes), after which the Workflow fails. nvcrectl certification run and workloadrun run grant the controller access in a namespace they create. In a namespace that already existed, such as default for workloadrun run without -n, they stop and print the binding unless --grant-run-access is passed. nvcrectl rbac print --namespace <run-namespace> writes the RoleBinding for runs that do not go through nvcrectl. Under rbac.scope: cluster nothing changes. ADR-092 is the design.

Setup next to an existing JobSet

nvcrectl setup init and setup reset coexist with an existing JobSet installation.

setup init detects who owns the JobSet CRD before it touches Kubeflow Trainer. A verified external JobSet controller is reused: Trainer installs without the bundled one. A retained CRD with no controller gets the bundled controller without a schema refresh. Anything ambiguous stops before Helm mutation and prints the manual Trainer path with --skip-phases=deps. A read-only JobSet consumer role — GKE’s system:clustermetrics is the documented example — is not mistaken for a leftover JobSet controller. That misclassification used to block Kubeflow Trainer installation.

setup reset keeps the shared JobSet CRD in every ownership mode, so JobSets that belong to other workloads survive. If Trainer uninstall fails, reset stops and exits non-zero instead of continuing into CRD cleanup. --skip-phases=helm is honored on both init and reset.

Breaking: both commands reject unknown --skip-phases values. init accepts deps and helm. reset accepts cr, helm, and deps. A script that passed any other name used to be ignored; it now exits with an error that names the accepted phases.

4. Certify one category

Start with communication/nccl-all-reduce. It works on any node count and finishes in minutes on a healthy cluster:

kubectl nvcre certification run \
  --category communication/nccl-all-reduce \
  --wait

The CLI creates a namespace named nvcrectl-<timestamp> and a Certification in it. Note both names from the output. With --wait, it prints status updates and then the report. While the run is in progress, kubectl describe on the Certification — or its Workflow, Job, or WorkloadRun — shows phase-transition Events without reading controller logs.

--timeout is validated only when --wait is set. A value below 1s is rejected before the run is created. Without --wait, --timeout is ignored. When --timeout is omitted with --wait, the CLI derives it from the selected categories’ catalog timeoutPerJob budgets (never less than 30 minutes). Communication entries default to 1h; training/nemotron5-8b and nemotron5-56b now render 24h, so a training category on its own derives a 36h wait. Pass --timeout in pipelines that need a tighter bound.

5. Print the report again

kubectl nvcre certification report <name> -n <namespace>

The report command defaults to the default namespace. The run created its own, so leave out -n and you find nothing. That is the common first-day mistake called out in the guide.

The report lists each failed node with its reason next to the name. JSON reports carry the full per-node reason and message under failedNodeDetails. On an execution failure, the failed-group section can include a captured log excerpt from one selected workload pod (pod, node, exit code, termination reason). JSON reports put the same data under categories[].failedGroups[].failureLog when the Job still exists. The field is absent if the failed Job was deleted before the report was generated. A category that measures bandwidth, failed, and recorded no bandwidth for a group of two or more nodes also shows a Next step line naming testScale: diagnose.

The sample report in NVIDIA’s README is example output for an aws / gb300 / 16-node run, not a cluster I operated:

╔════════════════════════════════════════════════════════════════╗
║                      Certification Report                      ║
╚════════════════════════════════════════════════════════════════╝

  Name:      nvcrectl-20260806-162730
  Platform:  aws
  GPU:       gb300
  Nodes:     16

┌────────────────────────────────────────────────────────────────┐
│  communication/nccl-all-reduce                                 │
├────────────────────────────────────────────────────────────────┤
│  Status:    Succeeded                                          │
│  Runtime:   3m 56s                                             │
│  Scale:     full-scale                                         │
│  Nodes/Job: 16                                                 │
│  Jobs:      1                                                  │
│  MNNVL:     Enabled                                            │
│                                                                │
│  Bandwidth:                                                    │
│    Size       AlgBW        BusBW        Samples                │
│    16 GB      473.44 GB/s  932.09 GB/s  9                      │
└────────────────────────────────────────────────────────────────┘

┌────────────────────────────────────────────────────────────────┐
│  Summary                                                       │
├────────────────────────────────────────────────────────────────┤
│  Categories:   1/1 passed                                      │
│  Failed Nodes: none                                            │
│  Result:       PASSED                                          │
└────────────────────────────────────────────────────────────────┘

Pass and fail are informational unless you set thresholds. No thresholds ship by default. To enforce them, use a Certification YAML with options.thresholds and run with --cert-file.

Ad-hoc WorkloadRun

For a single custom run without the catalog, write a WorkloadRun YAML. This is the WorkloadRun quick start’s NCCL all-reduce example, with Unpinned placement so it is one four-node job:

# nccl-all-reduce.yaml
apiVersion: nvcre.nvidia.com/v1alpha1
kind: WorkloadRun
metadata:
  name: nccl-all-reduce
spec:
  image: nvcr.io/nvidia/pytorch:26.01-py3
  numNodes: 4
  framework:
    mpi:
      binary: /usr/local/bin/all_reduce_perf_mpi
      mpirunPath: /usr/local/mpi/bin/mpirun
      args: ["-b", "8", "-e", "32G", "-f", "2", "-n", "100"]
  orchestration:
    placement: Unpinned
  bandwidthMeasurement:
    logProfileRef: nccl-bandwidth
    testType: all_reduce

Under the default Pinned placement, numNodes is the node count per job group, not the cluster total: on a 16-node cluster, numNodes: 4 produces four 4-node jobs. placement: Unpinned makes numNodes the total size of the one job that runs.

kubectl nvcre workloadrun run nccl-all-reduce.yaml --wait

The same NCCL WorkloadRun, plus a multi-category Certification, also live as runnable files under examples/.

WorkloadRun.spec.env values survive platform overrides that set TrainJob.trainer.env, which is the documented user-wins precedence. MPI ranks receive that merged environment through mpirun -x; without it, ranks started under sshd with a fresh environment and never saw spec.env or most of the base NCCL defaults. A Certification uses the same env field from v0.8.0, described above.

Read-only MCP server

nvcrectl mcp serve exposes four read-only tools over stdio so an agent can ask whether a certification passed and which nodes failed, without scraping CLI output. The tools are list_categories, get_certification_status, get_certification_report, and list_failed_nodes. No tool creates, mutates, or triggers a run. Every verdict is projected from the same code that backs nvcrectl certification report, so the two cannot disagree.

{
  "mcpServers": {
    "nvcre": {
      "command": "nvcrectl",
      "args": ["mcp", "serve"]
    }
  }
}

Cluster access uses the kubeconfig of whoever launches the server. The server holds no credentials of its own. Full flag and RBAC detail is in the MCP CLI reference.

Cleanup

kubectl delete namespace <namespace>
kubectl nvcre setup reset

setup reset also removes Kubeflow Trainer. Pass --skip-phases=deps to keep it. The shared JobSet CRD stays in every ownership mode. If Trainer uninstall fails, reset stops and exits non-zero instead of continuing into CRD cleanup.

Metrics and breaking changes

NCCL bandwidth metrics now include the Job’s workflow label. This changes Prometheus series identity, so queries matching an absent or empty workflow label will not match new measurements. Measurements created before this change retain the empty label until they are removed. Images and binaries are built with Go 1.27.3. v0.8.2 patched 1.27.2 for GO-2026-6809 in crypto/x509.

Patch notes from v0.8.1 (20 October 2026) and v0.8.2 (21 October 2026):

• v0.8.1 bounds and retries the MPI worker sshd install. A hung apt-get no longer waits out timeoutPerJob.
• v0.8.2 renders Certification spec.imagePullSecrets on every catalog entry, including the MPI launcher. Before that, only the two training categories got them.
• v0.8.2 rebuilds with Go 1.27.3.

Behavioral changes that landed in v0.8.0 and still apply:

• The base NCCL environment no longer sets NCCL_NVLS_ENABLE=1, and four catalog fragments stop forcing it where no measurement backs the value. An unchanged Certification on on-prem GB200/GB300, TogetherAI, OCI GB200, or Azure GB200 now lets NCCL’s probe decide.
• Each NCCL variable has one owning fragment on OCI. Duplicate mpirun -x pairs on three OCI arms disappear; the names and values the ranks receive stay the same.
• A new MPI WorkloadRun that already set spec.volumes, volumeMounts, config, or initContainers now mounts them on the workers. A volume that cannot attach holds the pods in ContainerCreating where those fields used to be ignored. Existing objects are unaffected because spec is immutable.
• A timed-out Job that actually ran now reports failed nodes. A certification whose timeout was previously silent on the node list will show a non-empty one.
• setup init keeps chart values set on the installed release instead of returning every value but rbac.* to its default. The controller image is still worked out on every run and never kept.
• The controller’s write grant is split. The default rbac.scope: cluster keeps today’s grant, except that unused list/watch verbs are dropped and get on Namespaces plus create on SelfSubjectAccessReviews are added. The default becomes namespaced in the next minor release.

Related reading

Saiyam Pathak published a hands-on lab the day after the v0.1.0 cut: NVIDIA Cluster Readiness Engine on vCluster Private Nodes. He ran NVCRE inside a tenant cluster backed by two A100 workers joined as vCluster Private Nodes. That is his lab and his report. I am not claiming his cluster, bandwidth numbers, GKE setup, or PASSED result as mine. Read it if you want an end-to-end private-node burn-in path on top of the same engine described here.

References

• NVIDIA/cluster-readiness-engine
• v0.8.2 release (21 October 2026, Latest)
• v0.8.1 release (20 October 2026)
• v0.8.0 release (19 October 2026)
• README (v0.8.2)
• CHANGELOG.md
• examples/ (runnable Certification and WorkloadRun)
• Admission-policy samples (Kyverno, Sigstore policy-controller)
• Your first certification
• Certify a cluster (incl. gang scheduling)
• Deployment (air-gapped installs, sourceRepo, image preload)
• nvcrectl mcp (read-only MCP server)
• nvcrectl certification (incl. --gpu-arch, --timeout, render Workflow names)
• WorkloadRun quick start (Unpinned placement)
• Prometheus metrics (status gauges, timed-out Jobs as failed)
• Certification API (env, dcgmHost, imagePullSecrets, workloadMetadata, taintSelectors, gangScheduler, rdmaDeviceAccess)
• WorkloadRun API (placement, MPI volumes)
• ADR-081: cordoned node selection
• ADR-082: GPU architecture from DRA ResourceSlices
• ADR-084: read-only MCP server
• ADR-085: Nscale NKS DRA
• ADR-087: mlnxPerNode zero opt-out
• ADR-088: OCI GB200 RoCE and rdmaDeviceAccess
• ADR-089: Unpinned single-job placement
• ADR-093: MPI worker sshd capability
• ADR-094: HGX B300
• ADR-092: per-namespace controller write access
• ADR-096: Mokka local environment and UAT runtime lane
• ADR-097: Certification category env
• ADR-098: setup init chart values
• ADR-099: failed-node attribution
• ADR-100: NCCL fragment ownership
• FAQ (air-gapped MPI / bounded sshd install)
• Interpret results (incl. failedNodeDetails)
• Catalog (eight categories, dcgmHost)
• Install (Helm 3.8, setup init values)
• nvcrectl setup (chart values, --rbac-scope)
• nvcrectl rbac print
• Run NVCRE locally without GPUs (Mokka)
• Platform detection (incl. Nscale, B300, OCI GB200, GCP H100 TCPXO)
• Run NCCL Benchmarks
• Nemotron 5 WorkloadRun guide
• Saiyam Pathak: NVCRE on vCluster Private Nodes