Supply chain and image provenance

Supply-chain security has a peculiar failure mode: almost everyone implements the signing half and stops. Signatures nobody verifies are a filing system, not a control.

Four separate questions

• Integrity — is this the same bytes that were built? Digests.
• Authenticity — did we build it? Signatures.
• Provenance — from which commit, by which pipeline? Attestations.
• Composition — what is inside it? An SBOM.

They are commonly bundled as one topic and they are not. You can have a signed image full of known-vulnerable packages, and an accurate SBOM for an image nobody can prove you built.

Tags are mutable. Digests are not.

# this can point somewhere else tomorrow
image: ghcr.io/you/api:v1.4.2

# this cannot
image: ghcr.io/you/api@sha256:3f1b...c90a

# what is that tag right now?
crane digest ghcr.io/you/api:v1.4.2

A tag is a mutable pointer. v1.4.2 can be moved, and imagePullPolicy: IfNotPresent means different nodes may be running different bytes under the same tag with no sign of it. Deploying by digest removes a whole class of "works on one node" mystery, and it is what GitOps tooling should write for you.

Sign in the pipeline, verify at admission

# in CI, keyless: the identity is the workflow, not a key you have to store
cosign sign --yes ghcr.io/you/api@sha256:3f1b...c90a

# attach provenance and an SBOM as attestations
syft ghcr.io/you/api@sha256:3f1b...c90a -o spdx-json > sbom.json
cosign attest --yes --predicate sbom.json \
  --type spdxjson ghcr.io/you/api@sha256:3f1b...c90a

Keyless signing is the part worth adopting deliberately. Instead of a private key you have to store and rotate, the signature is bound to an OIDC identity — the specific GitHub Actions workflow in the specific repository. There is no key to leak, and the thing you verify is "built by that workflow", which is what you actually wanted to know.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-images
spec:
  validationFailureAction: Enforce      # the whole point
  rules:
    - name: signed-by-our-workflow
      match:
        any:
          - resources: { kinds: [Pod] }
      verifyImages:
        - imageReferences: ["ghcr.io/you/*"]
          attestors:
            - entries:
                - keyless:
                    subject: "https://github.com/you/api/.github/workflows/build.yml@refs/heads/main"
                    issuer: "https://token.actions.githubusercontent.com"

SBOMs are only useful if something reads them

Generating an SBOM is one command. The value is entirely in what consumes it. A useful minimum: store the SBOM as an attestation next to the image, and run a scanner against your running images on a schedule — not just at build time. Vulnerability disclosures happen after you ship, which means the build-time scan that passed is not evidence about today.

# what is actually running, right now
kubectl get pods -A -o jsonpath=\
  '{range .items[*]}{range .status.containerStatuses[*]}{.imageID}{"\n"}{end}{end}' \
  | sort -u

That list is the only inventory that matters, and it is usually shorter and stranger than anyone expects.

What to actually do with this

• Pin one production workload to a digest and notice what else has to change.
• Add cosign sign to one pipeline this week; verification can come later.
• Switch verification on in audit mode and read what fails before enforcing.
• Scan running images on a schedule, not only at build.