Github Attestation

Published: Jun 6, 2024 by Isaac Johnson

Recently Github rolled out preview of Github Attestation that let one create “signed attestations” for anything built in Github Actions.

Let’s dig into this Preview release and see how we can use it to authenticate binaries from our Github workflows. We’ll look at binaries, containers and local as well as remote verification.

Usage

To test this out, I’ll use a public project of mine, Express Uploader

$ git clone https://github.com/idjohnson/expressUploader.git

I know I need to add permissions

permissions:
  id-token: write
  attestations: write
  contents: read

But I think I’m okay as I already have ‘write-all’ enabled

jobs:
  build:
    permissions: write-all
    runs-on: ubuntu-latest

Let’s save my Docker image to an Attestation

name: Build and Push Docker Image to GHCR

on:
  push:
    branches:
      - main
  workflow_dispatch:


jobs:
  build:
    permissions: write-all
    runs-on: ubuntu-latest

    steps:
      - name: 'Checkout GitHub Action'
        uses: actions/checkout@main

      - name: 'Login to GitHub Container Registry'
        uses: docker/login-action@v1
        with:
          registry: ghcr.io
          username: $
          password: $

      - name: 'Build and Push Image'
        run: |
          docker build . --tag ghcr.io/$/expressupload:latest
          docker push ghcr.io/$/expressupload:latest

      - name: 'Save as a TGZ'
        run: |
          docker save ghcr.io/$/expressupload:latest | gzip > expressupload.tar.gz

      - name: Attest Build Provenance
        uses: actions/attest-build-provenance@897ed5eab6ed058a474202017ada7f40bfa52940 # v1.0.0
        with:
           subject-path: "expressupload.tar.gz"


      - name: 'Helm package and push'
        run: |
          helm package ./helm/
          export HELM_EXPERIMENTAL_OCI=1
          helm push expressupload-*.tgz oci://ghcr.io/$/

We can see it passed and is uploaded

/content/images/2024/06/attestation-01.png

The idea is I can now verify the binary ( expressupload.tar.gz), provided I have it somewhere, that it is indeed that which I pushed.

I pulled and recompressed and got a 404 on that new archive

$ gh attestation verify ./expressupload.tgz -o idjohnson
Loaded digest sha256:c0fb67da9c87b66aa3a62dd223f69106c7e6aee6b3f3391f6233c3adf8a4af14 for file://expressupload.tgz
✗ Loading attestations from GitHub API failed

Error: failed to fetch attestations from idjohnson: HTTP 404: Not Found (https://api.github.com/orgs/idjohnson/attestations/sha256:c0fb67da9c87b66aa3a62dd223f69106c7e6aee6b3f3391f6233c3adf8a4af14?per_page=30)

That is because this is not the same binary. We can see that from the attstation page

/content/images/2024/06/attestation-02.png

Let’s go another way, let’s test something we can ensure is the same, the tgz’ed helm chart as pushed as an OCI artifact.

I’ll change it up in the Github flow

name: Build and Push Docker Image to GHCR

on:
  push:
    branches:
      - main
  workflow_dispatch:


jobs:
  build:
    permissions: write-all
    runs-on: ubuntu-latest

    steps:
      - name: 'Checkout GitHub Action'
        uses: actions/checkout@main

      - name: 'Login to GitHub Container Registry'
        uses: docker/login-action@v1
        with:
          registry: ghcr.io
          username: $
          password: $

      - name: 'Build and Push Image'
        run: |
          docker build . --tag ghcr.io/$/expressupload:latest
          docker push ghcr.io/$/expressupload:latest

      - name: 'Helm package and push'
        run: |
          helm package ./helm/
          export HELM_EXPERIMENTAL_OCI=1
          helm push expressupload-*.tgz oci://ghcr.io/$/

      - name: Attest Build Provenance
        uses: actions/attest-build-provenance@897ed5eab6ed058a474202017ada7f40bfa52940 # v1.0.0
        with:
           subject-path: "expressupload-0.1.0.tgz"

which I can see worked and uploaded a new SHA for verification

/content/images/2024/06/attestation-03.png

I can now pull it down locally

$ helm pull oci://ghcr.io/idjohnson/expressupload --version 0.1.0
Pulled: ghcr.io/idjohnson/expressupload:0.1.0
Digest: sha256:257b70bd3d03a5316477c53ce39a399812a921cb4ad965c59ca1daa91441e59a

As expected, this verifies successfully

$ gh attestation verify expressupload-0.1.0.tgz -o idjohnson
Loaded digest sha256:ff7ba1bb25585daaa3d2ac762f1edf0efefaa5a812e63ae311fb04e84a050b12 for file://expressupload-0.1.0.tgz
Loaded 1 attestation from GitHub API
✓ Verification succeeded!

sha256:ff7ba1bb25585daaa3d2ac762f1edf0efefaa5a812e63ae311fb04e84a050b12 was attested by:
REPO                       PREDICATE_TYPE                  WORKFLOW
idjohnson/expressUploader  https://slsa.dev/provenance/v1  .github/workflows/build.yml@refs/heads/main

Validating in GH

We can now change the build YAML to pull down the artifact and validate it is indeed the right tgz using attest

name: Build and Push Docker Image to GHCR

on:
  push:
    branches:
      - main
  workflow_dispatch:


jobs:
  build:
    permissions: write-all
    runs-on: ubuntu-latest

    steps:
      - name: 'Checkout GitHub Action'
        uses: actions/checkout@main

      - name: 'Login to GitHub Container Registry'
        uses: docker/login-action@v1
        with:
          registry: ghcr.io
          username: $
          password: $

      - name: 'Build and Push Image'
        run: |
          docker build . --tag ghcr.io/$/expressupload:latest
          docker push ghcr.io/$/expressupload:latest

      - name: 'Helm package and push'
        run: |
          helm package ./helm/
          export HELM_EXPERIMENTAL_OCI=1
          helm push expressupload-*.tgz oci://ghcr.io/$/

      - name: Attest Build Provenance
        uses: actions/attest-build-provenance@897ed5eab6ed058a474202017ada7f40bfa52940 # v1.0.0
        with:
           subject-path: "expressupload-0.1.0.tgz"

  validate:
    runs-on: ubuntu-latest
    needs: build
    steps:
      - name: 'Fetch Chart'
        run: |
          export HELM_EXPERIMENTAL_OCI=1
          helm pull oci://ghcr.io/$/expressupload --version 0.1.0

      - name: 'Validate'
        run: |
          set -x
          ls -l
          gh attestation verify ./expressupload-0.1.0.tgz -o idjohnson
        env:
          GH_TOKEN: $

      - name: 'Validate2'
        run: |
          set -x
          ls -l
          gh attestation verify ./expressupload-0.1.0.tgz -R idjohnson/expressUploader

Above I am validating two ways; one against the organization and one against the specific repo (Validate2)

This should kick off a build

/content/images/2024/06/attestation-04.png

However it fails for a good reason; we didnt set any auth for the GH CLI

/content/images/2024/06/attestation-05.png

We can use the ephimeral Github Token provided to our workflows (the provided secret GITHUB_TOKEN). I’ll add to the top of the validate build job.

name: Build and Push Docker Image to GHCR

on:
  push:
    branches:
      - main
  workflow_dispatch:


jobs:
  build:
    permissions: write-all
    runs-on: ubuntu-latest

    steps:
      - name: 'Checkout GitHub Action'
        uses: actions/checkout@main

      - name: 'Login to GitHub Container Registry'
        uses: docker/login-action@v1
        with:
          registry: ghcr.io
          username: $
          password: $

      - name: 'Build and Push Image'
        run: |
          docker build . --tag ghcr.io/$/expressupload:latest
          docker push ghcr.io/$/expressupload:latest

      - name: 'Helm package and push'
        run: |
          helm package ./helm/
          export HELM_EXPERIMENTAL_OCI=1
          helm push expressupload-*.tgz oci://ghcr.io/$/

      - name: Attest Build Provenance
        uses: actions/attest-build-provenance@897ed5eab6ed058a474202017ada7f40bfa52940 # v1.0.0
        with:
           subject-path: "expressupload-0.1.0.tgz"

  validate:
    runs-on: ubuntu-latest
    needs: build
    env:
      GH_TOKEN: $
    steps:
      - name: 'Fetch Chart'
        run: |
          export HELM_EXPERIMENTAL_OCI=1
          helm pull oci://ghcr.io/$/expressupload --version 0.1.0

      - name: 'Validate'
        run: |
          set -x
          ls -l
          gh attestation verify ./expressupload-0.1.0.tgz -o idjohnson

      - name: 'Validate2'
        run: |
          set -x
          ls -l
          gh attestation verify ./expressupload-0.1.0.tgz -R idjohnson/expressUploader

We can now see it is validated without issue, both using my organization (just me) and the specific repo

/content/images/2024/06/attestation-06.png

Container images

I did show using container images with attest as just a tgz’ed file, however there is a much simpler way.

name: Build and Push Docker Image to GHCR

on:
  push:
    branches:
      - main
  workflow_dispatch:


jobs:
  build:
    permissions: write-all
    runs-on: ubuntu-latest

    steps:
      - name: 'Checkout GitHub Action'
        uses: actions/checkout@main

      - name: 'Login to GitHub Container Registry'
        uses: docker/login-action@v1
        with:
          registry: ghcr.io
          username: ${{github.actor}}
          password: ${{secrets.GITHUB_TOKEN}}

      - name: 'Build and Push Image'
        run: |
          docker build . --tag ghcr.io/${{github.actor}}/expressupload:latest
          docker push ghcr.io/${{github.actor}}/expressupload:latest

      - name: 'Build and Push Image'
        run: |
          docker build . --tag ghcr.io/${{github.actor}}/expressupload:latest
          docker push ghcr.io/${{github.actor}}/expressupload:latest
          echo "======================= TESTS ======================="
          docker images --no-trunc --quiet ghcr.io/${{github.actor}}/expressupload:latest
          docker inspect -f '{{.Id}}' ghcr.io/${{github.actor}}/expressupload:latest

      - name: 'Set Image SHA0'
        run: |
          CONTAINERSHA=`docker images --no-trunc --quiet ghcr.io/${{github.actor}}/expressupload:latest`
          echo "CONTAINERSHA=$CONTAINERSHA" >> $GITHUB_OUTPUT
        id: consha0

      - name: 'Set Image SHA'
        run: |
          CONTAINERSHA=$(docker inspect -f '{{.Id}}' ghcr.io/${{github.actor}}/expressupload:latest)
          echo "CONTAINERSHA=$CONTAINERSHA" >> $GITHUB_OUTPUT
        id: consha1

      - name: Generate artifact attestation
        uses: actions/attest-build-provenance@v1
        with:
          subject-name: ghcr.io/${{github.actor}}/expressupload
          subject-digest: ${{ steps.consha0.outputs.CONTAINERSHA }}
          push-to-registry: true

      - name: 'Helm package and push'
        run: |
          helm package ./helm/
          export HELM_EXPERIMENTAL_OCI=1
          helm push expressupload-*.tgz oci://ghcr.io/${{github.actor}}/

      - name: Attest Build Provenance
        uses: actions/attest-build-provenance@897ed5eab6ed058a474202017ada7f40bfa52940 # v1.0.0
        with:
           subject-path: "expressupload-0.1.0.tgz"

  validate:
    runs-on: ubuntu-latest
    needs: build
    env:
      GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
    steps:
      - name: 'Fetch Chart'
        run: |
          export HELM_EXPERIMENTAL_OCI=1
          helm pull oci://ghcr.io/${{github.actor}}/expressupload --version 0.1.0

      - name: 'Validate'
        run: |
          set -x
          ls -l
          gh attestation verify ./expressupload-0.1.0.tgz -o idjohnson

      - name: 'Validate2'
        run: |
          set -x
          ls -l
          gh attestation verify ./expressupload-0.1.0.tgz -R idjohnson/expressUploader

I actually tried this a few ways. It does upload, but errors on verification. I can see it is adding a “v2” to the URL

/content/images/2024/06/attestation-07.png

But I can see results were uploaded on the SHA

/content/images/2024/06/attestation-08.png

Reviewing the docs, I question if I really need the ‘push-to-registry’ step

I removed it

      ...
        id: consha1

      - name: Generate artifact attestation
        uses: actions/attest-build-provenance@v1
        with:
          subject-name: ghcr.io/$/expressupload
          subject-digest: $

      - name: 'Helm package and push'
      ...

And this time it worked

/content/images/2024/06/attestation-09.png

And we can view the attest on the repo as it’s public

/content/images/2024/06/attestation-10.png

The point

So what is the point? Why create attestation entry? As the Github Docs say:

Artifact attestations enable you to create unfalsifiable provenance and integrity guarantees for the software you build. In turn, people who consume your software can verify where and how your software was built.

Think of block chain, but for build artifacts. It is a way to guarantee the thing you downloaded was that which was created from the originating pipeline.

The highest profile attack (of which we are aware) was the Solarwindws Hack of 2019 in which threat actors attacked the Orion build pipeline with malicious code known as ‘sunburst’.

Near the end of that article, in the section “Need for software bill of materials highlighted in aftermath of attack” it is noted why SBOMs are so important:

Modern software applications no longer rely on a monolithic stack of discrete software components. Developers now build applications out of many components that can come from many sources. Any one of the components that makes up an application could potentially represent a risk if there is an unpatched vulnerability. As such, it is critical for developers, organizations they work for and end users that consume applications be aware of all the different components that make up an application. It’s an approach that is known as a software bill of materials (SBOM). An SBOM is like a “nutritional label” that is present on packaged food products, clearly showing consumers what’s inside a product.

A Software Bill of Material (SBOM) that can then be checked against a secure store, such as Attestation, is key to minimizing these kinds of vulnerabilities.

A bit of commentary

We have many ways to check and store SHA256 hashes. The idea of a Container Digest is just that - a unique generatable key that matches a container to guarantee that is the same container.

In past systems I have often used the SHA as the image tag to guarantee authenticity, even if it makes image tags harder to sort and trace (you can always have more than one tag on a container image repository).

Another way I minimize risk is to use more than on Artifact store. For a small extra price for redundant storage, I’ve long held the line about storing containers in multiple clouds (or cloud and on-prem). Containers can be identified by SHA and any outage or threat detected can be mitigated by comparing to the others. At one prior employer, the AzDO system would store containers in Azure, AWS and an on-prem Artifactory repo. I could then guarantee quorum at any point. It also let me minimize retention policies in certain clouds to save on spend.

Summary

The announcement just came out this May with promises of more deep dives in Github Universe in October. Right now, it’s just part of the Github deal - no additional pricing.

I see Attestations as a great (?free) way to add some SBOM authenticity checks to our pipelines, provided one builds in Github.

There are non-commercial options, mostly using the blockchain like Tacos and OpenAttestation which we might explore in future posts. (If interested, start here)

Github Attestation SBOM

Have something to add? Feedback? You can use the feedback form

Isaac Johnson

Isaac Johnson

Cloud Solutions Architect

Isaac is a CSA and DevOps engineer who focuses on cloud migrations and devops processes. He also is a dad to three wonderful daughters (hence the references to Princess King sprinkled throughout the blog).

Theme built by C.S. Rhymes