Software supply chain attacks have moved from theoretical risks to primary threats for modern engineering teams. When a CI/CD pipeline is compromised, an attacker doesn't need to steal your source code; they simply inject a malicious binary or dependency into your final container image. Because your build server is "trusted," your Kubernetes clusters pull and run this compromised code without hesitation. You need a way to prove that the image running in production is exactly what your build system produced, without any tampering in between. This is where Sigstore and Cosign become essential.
By implementing cryptographic signing, you create a verifiable link between your build process and your deployment environment. In this guide, you will learn how to use Cosign (v2.2.4+) to sign container images and enforce verification in your clusters. This ensures that only authorized, un-tampered artifacts reach your production users.
TL;DR — Use Cosign to generate cryptographic signatures for container images during your CI build. Store these signatures in your registry alongside the image. Finally, use a Kubernetes admission controller to block any image that lacks a valid signature from your trusted identity.
Table of Contents
The Core Concept of Sigstore and Cosign
💡 Analogy: Imagine a high-security shipping port. Without signing, the port guards (Kubernetes) trust any container that arrives on a truck labeled "Your Company." With Cosign, every container is wrapped in a tamper-proof digital shrink-wrap with a wax seal. If the seal is missing or broken, the guards refuse to let the truck enter the port, even if the label looks correct.
Sigstore is an open-source project that simplifies how you sign, verify, and store software artifacts. Traditionally, code signing required complex PGP key management, which often led to lost keys or insecure storage. Sigstore changes this by providing a "keyless" signing flow. It uses OpenID Connect (OIDC) to link a signature to a specific identity, such as a GitHub Actions runner or a developer's email address.
Cosign is the command-line tool within the Sigstore ecosystem specifically designed for container images. Unlike older methods that stored signatures in external databases, Cosign stores signatures directly in your OCI (Open Container Initiative) registry. This means your signature follows your image wherever it goes—from Docker Hub to Amazon ECR or Google Artifact Registry. When you run a cosign sign command, it calculates a hash of the image and signs it, creating a verifiable record that the specific version of that image was approved by you.
The system relies on three main components: Fulcio (the certificate authority), Rekor (the transparency log), and Cosign (the tool). When you use keyless signing, Fulcio issues a short-lived certificate based on an OIDC token from your CI provider. Rekor records this event in a public, immutable log, making it impossible for an attacker to forge a signature without being detected. This architecture removes the burden of long-term private key management while increasing the overall transparency of your software supply chain.
When to Use Image Signing
The most critical scenario for image signing is protecting against "Registry Poisoning." If an attacker gains access to your container registry, they can overwrite an existing tag (like :latest or :v1.0) with a malicious image. Without signing, your Kubernetes cluster will pull this new, malicious version during the next deployment or node scaling event. Because the tag name remains the same, your standard security scanners might not even trigger. Cosign prevents this because the signature is tied to the unique digest (SHA256) of the image, not the mutable tag.
Another common use case is enforcing "Build Integrity." In many organizations, developers have the permissions to push images to the registry from their local machines for testing. However, you only want images built by your official CI/CD pipeline to reach production. By configuring your CI pipeline to sign images using its own unique OIDC identity, you can set a policy in Kubernetes that says: "Only allow images signed by the GitHub Actions identity associated with our production repository." This effectively blocks any "shadow" deployments from local workstations.
Finally, signing is essential for meeting compliance requirements like SLSA (Supply-chain Levels for Software Artifacts). As more industries move toward requiring a Software Bill of Materials (SBOM), being able to prove the origin of every container becomes a legal and operational necessity. During our internal testing with Node.js applications, we found that adding Cosign signing to the build pipeline added negligible overhead—typically under 5 seconds—while providing a 100% verifiable audit trail for every production deployment.
Step-by-Step Implementation with GitHub Actions
Step 1: Install Cosign
First, you need the Cosign binary. For local development or within a CI runner, you can install the latest version using the following commands. It is important to use version 2.x as it includes significant improvements for OIDC and keyless signing.
# Install Cosign on Linux
LATEST_VERSION=$(curl -s https://api.github.com/repos/sigstore/cosign/releases/latest | grep 'tag_name' | cut -d\" -f4)
curl -LO https://github.com/sigstore/cosign/releases/download/${LATEST_VERSION}/cosign-linux-amd64
sudo mv cosign-linux-amd64 /usr/local/bin/cosign
chmod +x /usr/local/bin/cosign
Step 2: Set Up Keyless Signing in GitHub Actions
Keyless signing is the recommended approach for CI/CD because it avoids the "Who guards the guards?" problem of managing secrets. In your GitHub Actions workflow file, you must grant the job id-token: write permissions. This allows the runner to request an OIDC token from GitHub, which Sigstore uses to verify the build's identity.
name: Build and Sign Image
on: [push]
jobs:
build:
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
id-token: write # Required for keyless signing
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Install Cosign
uses: sigstore/cosign-installer@v3.4.0
- name: Log into GitHub Container Registry
run: echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u ${{ github.actor }} --password-stdin
- name: Build and push image
id: build-image
run: |
IMAGE_NAME=ghcr.io/${{ github.repository }}:latest
docker build -t $IMAGE_NAME .
docker push $IMAGE_NAME
# Get the exact digest for signing
echo "digest=$(docker inspect --format='{{index .RepoDigests 0}}' $IMAGE_NAME)" >> $GITHUB_OUTPUT
- name: Sign the image
run: |
cosign sign --yes ${{ steps.build-image.outputs.digest }}
Step 3: Verify the Signature
Verification is just as important as signing. You can verify the image manually to ensure everything worked correctly. Use the --certificate-identity flag to specify which GitHub workflow was allowed to sign the image. This prevents someone else from signing your image with their own GitHub identity and claiming it is valid.
cosign verify ghcr.io/your-org/your-repo@sha256:abcd... \
--certificate-identity https://github.com/your-org/your-repo/.github/workflows/build.yml@refs/heads/main \
--certificate-oidc-issuer https://token.actions.githubusercontent.com
Common Pitfalls and Fixes
⚠️ Common Mistake: Signing by Tag instead of Digest. If you run cosign sign my-image:latest, you are creating a race condition. If the tag is updated between the push and the sign command, you might sign the wrong image.
Problem: "Signatures not found" in private registries.
Cosign stores signatures as separate OCI artifacts. Some older versions of private registries (like older on-prem Harbor or Nexus instances) may not support these artifact types or might have restrictive cleanup policies that delete the signature but keep the image. To fix this, ensure your registry is OCI-compliant. If you are behind a strict firewall, you may also need to host your own Rekor and Fulcio instances, as the default keyless flow requires access to the public Sigstore endpoints.
Problem: OIDC Token Expiration.
When using keyless signing, the certificates issued by Fulcio are short-lived (usually 10 minutes). If your build process is extremely slow and the cosign sign command is triggered long after the token was requested, it may fail. You should always place the signing step immediately after the image push to ensure the OIDC context is still fresh. If you encounter "token expired" errors, check the system time on your runners, as clock drift can also cause OIDC validation failures.
Problem: Public Transparency Log Privacy.
By default, keyless signing uses the public Rekor log. This means your image names and the GitHub identities of your builders will be publicly visible. For open-source projects, this is a feature. For private corporate applications, this might violate security policies. In this case, you must either use static "keyed" signing (storing a private key in a KMS like AWS KMS or HashiCorp Vault) or run a private Sigstore stack. Static keys do not use the public transparency log.
Pro Tips for Production Security
To truly secure your pipeline, signing alone is not enough; you must enforce it. Use a Kubernetes Admission Controller like Kyverno or Policy Controller. These tools intercept every kubectl apply request and check if the image has been signed by your trusted identity. If the verification fails, the pod is never created. This "Secure by Default" approach removes the human element of security checks.
Another best practice is to sign your SBOM (Software Bill of Materials) alongside your image. Cosign allows you to attach and sign files to an image digest. By signing both the image and the list of its dependencies, you provide a complete, verifiable package for your security operations center (SOC) to audit. This prevents attackers from swapping out a clean image for one with vulnerable dependencies while still having a "valid" image signature.
📌 Key Takeaways
- Keyless is King: Use OIDC-based signing to eliminate the risk of leaking private keys.
- Digest Stability: Always sign and verify based on the immutable image digest (SHA256), not tags.
- Shift Left Enforcement: Don't just sign; use admission controllers to block unsigned images in Kubernetes.
- Identity Matters: Always verify the
--certificate-identityto ensure the signature came from your specific CI pipeline.
Implementing Sigstore and Cosign is one of the highest-impact security improvements you can make to a modern cloud-native stack. It bridges the gap between the code you write and the code you run, providing a cryptographic guarantee of integrity that standard firewalls and scanners cannot match.
Frequently Asked Questions
Q. Does Cosign work with private registries like AWS ECR or GitLab?
A. Yes. Cosign is OCI-compliant and works with any registry that supports the OCI specification, including ECR, GCR, Harbor, and GitLab. For private registries, you simply need to ensure the `cosign` command has the appropriate credentials via `docker login` or environment variables to read and write artifacts.
Q. What happens if the Sigstore public services go down?
A. If you use the keyless flow, you rely on Fulcio and Rekor. If they are unreachable, you cannot sign new images. However, verification can sometimes be cached. For high-availability requirements, many enterprises choose to run their own private Sigstore instance or use KMS-based signing which doesn't rely on the public certificate authority.
Q. How do I rotate keys if I am not using the keyless flow?
A. If you use traditional keys (`cosign generate-key-pair`), you must manage rotation manually. This involves updating the public key in your Kubernetes admission controllers and securely rotating the private key in your CI secrets. This complexity is exactly why the OIDC keyless flow is preferred for most teams.
Post a Comment