Pinning and caching manylinux images in CI

This page answers one question: your builds reference a manylinux image by tag, the toolchain underneath moves without warning, and cache hits collapse — so how do you pin the image by digest and keep the pull from costing minutes on every cell? It sits inside the Manylinux and Manyarm Docker Base Images section of the Modern Python Build Tooling & Wheel Configuration reference, and gives you the digest workflow, the caching options and the automated bump.

What moves under a floating tag and what a digest fixes Under a floating tag the compiler, the C library patch level, the bundled Python interpreters and the auditwheel policy files can all change between two builds with no change in the repository. Pinning by digest fixes all four, so a change to any of them arrives as a reviewed commit rather than as an unexplained difference in the artifact or a collapsed cache hit rate. floating tag — these can move the GCC version the glibc patch level the bundled CPython builds auditwheel and its policy files a different wheel from the same commit and every cached object invalidated digest — all four fixed the same bytes on every run cache keys stay meaningful a toolchain change is a commit and a reviewable diff the cost: someone has to bump it, which is what the scheduled job is for

Context & Root Cause

manylinux images are rebuilt regularly — for security updates, for new CPython releases, for policy changes. A reference like quay.io/pypa/manylinux_2_28_x86_64:latest, or even a dated tag that is later re-pushed, resolves to whatever is current at pull time. Two builds of the same commit a week apart can therefore use different compilers.

For a spatial project that has two consequences beyond the obvious one. The compiled output differs, which defeats any attempt at reproducibility and makes a regression impossible to attribute. And every cached compiler object becomes a miss, because the compiler identity is part of the hash — so a build that normally takes ninety seconds silently takes twenty minutes, with nothing in the log explaining why. Pinning by digest fixes both, at the cost of someone having to move the pin.

Solution / Fix

This targets cibuildwheel 3.0+, Docker or Podman, and the manylinux_2_28 images.

1. Resolve the tag to a digest once 

docker buildx imagetools inspect quay.io/pypa/manylinux_2_28_x86_64:latest \
  --format '{{.Manifest.Digest}}'
# sha256:9f3c1ab8...

2. Reference the digest everywhere 

# pyproject.toml
[tool.cibuildwheel]
manylinux-x86_64-image = "quay.io/pypa/manylinux_2_28_x86_64@sha256:9f3c1ab8..."
manylinux-aarch64-image = "quay.io/pypa/manylinux_2_28_aarch64@sha256:2b71ee04..."

FROM quay.io/pypa/manylinux_2_28_x86_64@sha256:9f3c1ab8...

3. Make the digest part of the cache key 

key: native-${{ matrix.arch }}-${{ hashFiles('ci/native-versions.lock') }}-${{ env.IMAGE_DIGEST }}

Including it means a toolchain change invalidates the cache deliberately rather than accidentally, and the log shows a new key rather than an unexplained slowdown.

4. Cache the image itself 

- name: Restore image layers
  uses: actions/cache@v4
  with:
    path: /tmp/img
    key: img-${{ env.IMAGE_DIGEST }}
- run: |
    [ -f /tmp/img/base.tar ] && docker load -i /tmp/img/base.tar || {
      docker pull "$IMAGE"; mkdir -p /tmp/img; docker save "$IMAGE" -o /tmp/img/base.tar; }

Verification 

# 1. Nothing references a floating tag
grep -rn 'manylinux[^@"]*:' --include='*.toml' --include='*.yml' --include='Dockerfile*' . \
  | grep -v '@sha256:' || echo "all references are pinned"

# 2. The digest in use is the one recorded
docker image inspect "$IMAGE" --format '{{index .RepoDigests 0}}'
# expected: matches ci/image-digest.txt

# 3. The toolchain inside is what the pin promised
docker run --rm "$IMAGE" bash -c 'gcc --version | head -1; ldd --version | head -1'
# expected: the same two lines as the last recorded build

The third check is worth recording as a build artifact. Two lines per release, kept over time, turn “did the compiler change?” from an investigation into a diff — and they are the fastest way to explain a wheel that suddenly behaves differently.

Choosing How to Cache the Image

Pulling a manylinux image is a gigabyte-scale download, and there are three ways to avoid paying it on every cell.

Three ways to avoid re-pulling the base image on every matrix cell Relying on the runner's own image cache is free and unreliable because ephemeral runners start empty. Saving and restoring the image through the CI cache is reliable and pays a large upload and download. Mirroring the image into a registry close to the runners is fastest and needs a registry to maintain. All three key on the digest, which is what makes a hit meaningful. the runner's own cache free, and empty on an ephemeral runner — which is most of them fine for self-hosted runners that persist between jobs save and restore via the CI cache reliable and provider-agnostic; keyed on the digest costs a large upload once and a download per job — often still a win mirror into a nearby registry fastest pull, shared across every job and branch needs a registry, credentials and a retention policy of its own whichever you choose, key it on the digest — a tag-keyed image cache can serve a different image than the pin names

The middle option is the usual answer for hosted runners, and its economics depend on the image size relative to the provider’s cache bandwidth. Measure before adopting it: for a large image on a fast registry connection, pulling directly can be quicker than restoring a cached tarball, and the only way to know is to time both on your own runners.

The third option becomes compelling once you are already building a custom image with GDAL baked in, as the hybrid in cibuildwheel vs manual Docker matrix for GDAL wheels describes — the registry is already there, and the base image is one more artifact in it.

Bumping the Pin Without Forgetting

A pin nobody moves becomes a security liability, so the bump has to be automatic enough that it happens and reviewed enough that it is deliberate.

A scheduled job that proposes a digest bump as a reviewable change A weekly job resolves the current tag to a digest, compares it with the pinned one, and if they differ opens a pull request updating the digest and recording the new compiler and C library versions. Continuous integration then builds the full matrix against the proposal, so the review sees both the toolchain change and its effect on the artifacts before anything merges. weekly job resolve the tag compare against the pin open a pull request digest + toolchain versions CI builds it the full matrix review the pull request body should carry the old and new compiler and glibc versions, because those two lines are what a reviewer actually needs to judge it and the first build after a bump is expected to miss every cache — that is the pin working, not a fault

The last line is worth stating in the pull request template. A digest bump invalidates the compiler object cache by design, so the build that validates it is slow; treating that as a red flag leads people to revert bumps that were correct. One sentence in the template prevents a recurring conversation.

Pitfalls & Alternatives

Pinning the tag but not the aarch64 image. The two are separate images with separate digests, and a project that pins one and floats the other gets reproducibility on half its matrix.

Recording the digest in only one place. If pyproject.toml, a Dockerfile and a CI file each name it, they drift. Put it in one file that all three read, exactly as the native library versions are handled.

Caching the image under a tag-based key. A key of img-manylinux_2_28-latest will happily serve last month’s image after the pin moves. Key on the digest so a hit is by definition the right bytes.

Bumping the image and the library versions in one change. Both alter the compiled output; doing them together makes any difference unattributable. Separate commits, separate builds, as pinning GDAL and PROJ versions across a wheel set also argues.

Frequently Asked Questions

How often should the digest move?

Monthly is a reasonable default, with immediate moves for a security update to the image. The scheduled proposal makes the cadence a review decision rather than something that depends on someone remembering.

Does pinning by digest stop me getting security updates?

It stops you getting them silently, which is the point. The updates still arrive, as a proposed change with a visible diff and a full matrix build behind it — which is a better position than discovering that a toolchain changed by noticing that a wheel behaves differently.

Is the digest stable across registries?

The manifest digest is content-derived, so mirroring an image preserves it. That is what makes a registry mirror safe: the same digest resolves to the same bytes wherever it is pulled from, and the pin remains meaningful.

What about multi-architecture manifests?

A multi-arch tag resolves to a manifest list whose digest covers every architecture. Pinning the list digest is usually what you want, because a single pin then serves both x86_64 and aarch64 cells; pinning per-architecture digests is also valid and makes each cell’s toolchain independently visible.

Should the Dockerfile for a custom image pin its base?

Yes, and it is the most important place to do it, because the custom image is built rarely and used everywhere. An unpinned base there means your own image’s contents change between builds that were meant to be identical.

Does any of this apply to macOS and Windows runners?

The digest concept does not, because those are runner images rather than containers — but the underlying problem does. Runner images are versioned and updated by the provider, so pinning the runner label to a specific version rather than a rolling one is the closest equivalent and is worth doing for the same reasons.

What belongs in the digest record besides the digest itself?

The compiler banner, the C library version and the date the pin was taken. Those three lines cost nothing to capture and they are what a reviewer reads to judge a proposed bump — a digest on its own says only that something changed, while “GCC 13.2 to 13.3, glibc unchanged” says what kind of change it is.

Should the custom image inherit the base pin automatically?

No — that is precisely the drift the pinning removes. The custom image’s Dockerfile should name a digest, and updating it should be the same reviewed change as updating the wheel pipeline’s. Automating the proposal is fine; automating the merge puts an unreviewed toolchain into your releases.

How does this interact with a project that supports several manylinux policies?

Each policy is a separate image with its own digest, so a project publishing both manylinux2014 and manylinux_2_28 wheels pins two. Keeping both in the same file makes the pair visible; splitting them across configuration files is how one gets bumped and the other does not.

What happens to an old digest when the registry prunes it?

It can become unpullable, which turns a historical build into one that cannot be reproduced. Mirroring the digests you have released into a registry you control removes that risk, and for a project that cares about rebuilding past releases it is the missing piece the public registry cannot provide.