Most projects unit-test the binary. Almost nobody unit-tests the YAML that signs the binary. That asymmetry is how supply-chain regressions land: a well-meaning refactor deletes a signing step, CI stays green, and the failure shows up on release day — under time pressure, on a red release, with the blog post already drafted.

This piece is about treating release workflow policy as code you can break in a PR and have make test refuse to merge. The concrete case study is NVIDIA Cluster Readiness Engine (NVCRE) — a Go tool that validates GPU Kubernetes clusters and ships multi-platform binaries under a Sigstore / provenance contract — but the pattern applies to any repo whose trust story lives in .github/workflows more than in main.go.

The failure mode that unit tests miss

Application tests answer: does this function still compute the right digest?

Release policy answers a different question: did we still sign what we claim we signed, with the identity we published, using credentials we intended to grant?

Those answers diverge in predictable ways:

Drift What CI still sees What users get
Second signer sneaks into another workflow Green build The published certificate-identity pin silently widens
id-token: write on a job that never signs Green build Broader OIDC blast radius than the contract requires
uses: actions/checkout@v4 instead of a SHA Green build An unreviewed move of the release path
New platform in the Makefile, forgotten in the asset lists Green build An unsigned binary and unsigned SBOM next to notes that say every asset carries a Sigstore bundle

None of these break compilation. The release-time gate may catch them — but that is the last useful place to learn. Policy wants to fail in the PR that introduced the drift.

Four invariants worth pinning

NVCRE’s release-policy suite is a useful checklist for any project shipping attested binaries. The names below are the general form; the implementation detail is “reintroduce the defect and watch the test fail,” not “assert the happy path once.”

1. Sole signer, call-only

Exactly one workflow is allowed to invoke the signing primitives (cosign sign / attest, actions/attest-build-provenance, and cousins). That workflow is workflow_call-only — no push, no workflow_dispatch, no path that lets a branch forge the published identity.

Why: users pin one workflow identity. A second signer anywhere quietly expands what that pin accepts. A directly runnable attest workflow is an identity you can mint from a feature branch.

2. OIDC scope matches signing duty

Jobs that sign request id-token: write and nothing more. Jobs that do not sign do not request it at all.

Why: the OIDC token is the credential the whole provenance contract rests on. “This job might need it later” is how blast radius grows without a product decision.

3. No mutable tags on the release path

Every third-party uses: is a full 40-character commit SHA. Local ./ actions are fine; @v4 that can move is an unreviewed change to release.

Why: Dependabot PRs that bump a tag are visible. A floating tag that retargets overnight is not. Pinning forces the review onto the SHA you actually run.

4. Expected assets match what release uploads

Platform lists, attest matrices, pre-publish checks, digest maps, and post-publish gates are easy to fork into four hardcoded copies of the same truth. Assets themselves often come from a Makefile glob (NVCRECTL_PLATFORMS and friends). Adding a platform can ship unsigned artifacts while every gate stays green and the release notes still claim universal Sigstore coverage.

Why: ADR-level commitments about “every asset is attested” are only real if the lists cannot drift independently. One source of truth, or tests that treat divergence as a hard fail.

Mutation tests, not decoration

A policy assertion that only reads the current YAML is a snapshot. The useful bar is mutation:

  • Add a second cosign attest outside the sole-signer workflow → make test fails.
  • Give the attest workflow a non-workflow_call trigger → fail.
  • Grant id-token: write to a non-signing job → fail.
  • Replace any pinned action SHA with a tag → fail.
  • Extend the platform list without updating every asset list → fail.

If reintroducing the defect does not fail the suite, the test is documentation with a green checkmark. NVCRE’s test/releasepolicy package runs in ordinary CI — no cluster, no network — which is the right cost model for something you want on every PR.

Where this sits relative to SLSA and Sigstore

SLSA Build levels and Sigstore verification give consumers a verify story. Policy tests give maintainers a do not regress the producer story. They are complementary:

  • Sigstore / identity pins tell a downloader which workflow is allowed to have built the artifact.
  • Sole-signer + call-only tests keep that published identity from quietly growing.
  • SHA pins keep the actions inside that workflow from floating.
  • Asset-list tests keep the set of blobs covered by attestations aligned with what actually shipped.

Skip the producer side and you get beautiful verification docs pointing at a pipeline that a refactor can hollow out without anyone noticing until gh attestation verify fails for a customer.

A practical adoption order

If you are starting from a release workflow that “mostly works”:

  1. Write down the identity contract you already publish (which workflow, which repo, which subject). That is the sole-signer target.
  2. Inventory id-token: write. Every grant should map to a job that actually signs or attest.
  3. Pin every third-party action on the release path by SHA. Do the Dependabot theater once; stop relying on tags for the trust boundary.
  4. Collapse or cross-check asset lists against the Makefile / matrix that actually uploads. Prefer one generated list; if you cannot, mutation-test the forks.
  5. Put the four checks in make test, same job as unit tests. Release-time gates stay as belt-and-suspenders, not as the only alarm.

What this is not

This is not a substitute for code review, for a human release checklist, or for an offline key ceremony when your threat model requires one. It is also not “more YAML.” It is Go (or whatever your repo already tests in) that parses the workflows and refuses merges that violate the contract you already claim in docs.

For GPU and platform tooling — the NVCRE class of project — the binary is the product users install on clusters, but the attestation story is what lets those users trust the binary without rebuilding it. Policy tests are how that story survives the next tidy-up PR.

If your release notes say every asset carries a Sigstore bundle, your test suite should be able to prove that claim still holds after someone adds linux/arm64 on a Friday afternoon.

Anpoo Sivanadi, Staff Software Engineer