Just like supply chains of physical goods, the security of software supply chains is extremely important. Malicious attacks on the software supply chain are an ever-present threat and can cause extreme damage, which is evident from the Colonial Pipeline security attack which took down the largest fuel pipeline in the US due to a compromised password.
The software supply chain is anything that affects your software, including everything that goes into writing and distributing your software: code, package manager, dependencies, who wrote it, who reviewed it, distribution mechanism, how it runs, where it runs, licensing … pretty much everything that touches it at any point.
Organizations need to be securing their software, so it's imperative to secure the software supply chain at every possible opportunity.
Simply put, signing an image provides cryptographic evidence that the author is who they say they are; based on them having access to the trusted private key and the content has not been changed since. Auth0 provides a great explanation on asymmetric public key cryptography, and also cover signatures.
Cryptographic signing has been around long before container images were a thing. I've personally taken pride in signing my git commits for over a decade (see how I manage my keys), but generating and protecting keys is not for the faint of heart.
We live in a world with high-profile attacks on supply chains and an increasing requirement to demonstrate an auditable record of the provenance of where our assets have come from.
By simply authenticating to the registry to push while a strong first defense introduces further requirements for protecting those credentials and particularly when your CI system is creating the images those credentials may be visible to multiple team members.
This is then exasperated if you’re running your own registry since the administrators of that are likely administrators of other things, which lowers the barrier for how much collusion is required (too much privilege - trust is extended a long way).
Let's play a game of 'spot the difference': One of these images has something malicious in it that’ll do bad things when it runs (not really, it’ll just print “I am the bad guy” and then exit, but you get the point). The table shows the output of the Continuous Integration pipeline in GitHub and the corresponding images it built and pushed, storing it in Github Container Registry.
Signed Image: ghcr.io/chrisns/cosign-demo-spotthedifference:signed-1417835412.1
Unsiged Image: ghcr.io/chrisns/cosign-demo-spotthedifference:unsigne-1417835412.1
Signed Image: ghcr.io/chrisns/cosign-demo-spotthedifference:signed-1417836769.1
Unsigned Image: ghcr.io/chrisns/cosign-demo-spotthedifference:unsigne-1417836769.1
Signed Image: ghcr.io/chrisns/cosign-demo-spotthedifference:signed-1417837856.1
Unsigned Image: ghcr.io/chrisns/cosign-demo-spotthedifference:unsigne-1417837856.1
Signed Image: ghcr.io/chrisns/cosign-demo-spotthedifference:signed-1417838926.1
Unsigned Image: ghcr.io/chrisns/cosign-demo-spotthedifference:unsigne-1417838926.1
Spot anything by just looking at the unsigned images?
By looking at the metadata of the build outputs it is hard to tell if any of these images have been changed since they were built from the CI pipeline. Spoiler alert: It's Build #7.
Okay, maybe I made this one easy since the time it took me to write the bad code, push and sign it under my own signature makes it stand out in the registry.
One way of checking if the image has been tampered with is to use a utility called cosign. If we take the signed image name from Build #8 and run the following command:
$ COSIGN_EXPERIMENTAL=1 cosign verify ghcr.io/chrisns/cosign-demo-spotthedifference:signed-1417836769.1 | jq
The following output is shown:
"Issuer": "https://token.actions.githubusercontent.com",
"Subject": "https://github.com/chrisns/cosign-demo-spotthedifference/.github/workflows/ci.yml@refs/heads/main"
If we run the same command but this time with the image from Build #7, you can see that the Issuer and subject are different.
$COSIGN_EXPERIMENTAL=1 cosign verify ghcr.io/chrisns/cosign-demo-spotthedifference:signed-1417835412.1 | jq
"Issuer": "https://github.com/login/oauth",
"Subject": "chris@cns.me.uk"
You can verify this yourself by installing cosign and running against any of the images in our demonstration repo. In the two outputs above, it can be seen image in Build #8 was created by a pipeline and the image from Build #7 was created by a bad actor aka chris@cns.me.uk.
Let’s not forget that if a bad actor has write access to your registry (or the ability to impersonate it) then they don’t need to leave their bad image there for long. They can always replace it with the good one later after they’ve reached their goal of it running in your cluster; hence while signing is important, constantly verifying that is how you unlock the value.
When creating and pushing signed images to a container registry, two things are required: registry credentials and a private key. You’ve now got two problems: Securing registry credentials and securing a private key. Inevitably you’ll keep these in a similar way, a compromised device such as a developer laptop or CI system quickly negates any of the benefits of signing in the first place.
It’s possible to use hardware tokens such as a Yubikey for humans which can be configured to require a PIN and physically touch the token to sign anything. It is not possible to extract the private key material from the device. However, this is inherently not practical to implement for your non-human fingerless builders.
Enter: Cosign
Cosign from sigstore makes the traditional signing of container images vastly easier. However, this can be taken a step further in combination with the wider sigstore products. Using fulcio and rekor, it's possible to sign images (and other things too) without keys and later verify them against an immutable transparency log.
Okay, we don’t really sign without keys. But behind the scenes, cosign creates the keypair ephemerally (they last 20 minutes) and gets them signed by Fulcio using your authenticated OIDC identity.
OIDC
OpenID Connect 1.0 is a simple identity layer on top of the OAuth 2.0 protocol. It allows Clients to verify the identity of the End-User based on the authentication performed by an Authorisation Server, as well as to obtain basic profile information about the End-User in an interoperable and REST-like manner.
OIDC with human interaction
https://www.youtube.com/watch?v=a0KC3DPLD2s
$ COSIGN_EXPERIMENTAL=1 cosign sign chrisns/cosign-keyless-demo:latest
$ COSIGN_EXPERIMENTAL=1 cosign verify chrisns/cosign-keyless-demo:latest
Note the issuer and subject, this is from the identity Fulcio received and stored in the Rekor transparency log.
"Issuer": "https://accounts.google.com",
"Subject": "chris.nesbitt-smith@appvia.io"
Now lets do it without interaction, using the following workflow:
name: Build Push Sign
on: { push: { branches: ['main'] } }
jobs:
build:
runs-on: ubuntu-latest
permissions:
packages: write
id-token: write
steps:
- uses: actions/checkout@v2.3.5
- name: Login to GitHub
uses: docker/login-action@v1.9.0
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: build+push
uses: docker/build-push-action@v2.7.0
with:
push: true
tags: ghcr.io/chrisns/cosign-keyless-demo:latest
- uses: sigstore/cosign-installer@main
- name: Sign the images
run: |
cosign sign ghcr.io/chrisns/cosign-keyless-demo:latest
env:
COSIGN_EXPERIMENTAL: 1
Now your images will have something like the following when you run:
$ COSIGN_EXPERIMENTAL=1 cosign verify ghcr.io/chrisns/cosign-keyless-demo:latest
…
"Issuer": "https://token.actions.githubusercontent.com",
"Subject": "https://github.com/chrisns/cosign-keyless-demo/.github/workflows/ci.yml@refs/heads/main"
What this allows us to do is cryptographically demonstrate that the image we can pull was built by the CI pipeline of that repository. Neat!
As we’ve seen, signing your container images can be as trivial as a couple of lines added to your CI pipeline.
If you’re a software vendor that ships their product as a container, signing is a great way to reassure your customers that you take security seriously and allow them to verify the authenticity of your product.
If you’re building software for your own business use, it’s also a great way to validate the integrity of your own supply chain internally.
So now that we can verify the image manually, the next logical step is to let our Kubernetes cluster do that for us.
Kubernetes admission controller
In Kubernetes, an Admissions Controller is a piece of code that intercepts requests to the API server and does something with it. We have created an admissions controller that can either approve or deny the image from running in the cluster based on its cryptographic signature.
For the next step, you'll need a Kubernetes cluster. If you don’t have one to hand, you can use KiND or minikube.
Installation
# if you don't already have cert-manager
kubectl apply -f https://github.com/jetstack/cert-manager/releases/latest/download/cert-manager.yaml
kubectl apply -k https://github.com/appvia/cosign-keyless-admission-webhook
Usage
In your pod spec you set an annotation(s) of subject.cosign.sigstore.dev/CONTAINERNAME* to the subject of the certificate and also set the issuer.cosign.sigstore.dev/CONTAINERNAME* to the Issuer.
*CONTAINER_NAME is the name of the container from your pod specification.
Full example:
apiVersion: v1
kind: Pod
metadata:
annotations:
subject.cosign.sigstore.dev/demo: https://github.com/chrisns/cosign-keyless-demo/.github/workflows/ci.yml@refs/heads/main
issuer.cosign.sigstore.dev/demo: https://token.actions.githubusercontent.com
name: cosign-keyless-demo
spec:
containers:
- image: ghcr.io/chrisns/cosign-keyless-demo:latest
name: demo
Save the contents above in a file called pod.yaml and run:
$ kubectl apply -f pod.yaml
Since the image ghcr.io/chrisns/cosign-keyless-demo:latest is signed as we expect the Admission controller to accept the request and the pod will be accepted.
And if you change the pod.yaml to:
apiVersion: v1
kind: Pod
metadata:
annotations:
subject.cosign.sigstore.dev/demo: chris@cns.me.uk
issuer.cosign.sigstore.dev/demo: "https://github.com/login/oauth",
name: cosign-keyless-demo-bad
spec:
containers:
- image: ghcr.io/chrisns/cosign-keyless-demo:latest
name: demo
You'll find it will be rejected by the admission controller and not permitted to run on the cluster as this image is not signed as expected.
If you’ve followed along with the above you can now sign your images, and check the signatures in Kubernetes before accepting the pod to run, assuring that the image your cluster is about to start hasn’t been tampered with. Ensuring that the images form part of a secure software supply chain from creation to runtime.
The admission controller I made is open-source on GitHub, so please do star and watch it, and as always pull requests are always very welcome.
.png)
