Making spatial wheels byte-reproducible

This page answers one question: two builds of the same commit produce wheels with different hashes, so how do you find which inputs differ and eliminate them one at a time until the two artifacts are identical? It sits inside the Reproducible Builds and Supply-Chain Attestation section of the Modern Python Build Tooling & Wheel Configuration reference, and gives you the diff procedure, the fixes in the order they pay, and the point at which to stop.

Where two builds of the same commit usually diverge Four layers of a wheel with the divergence typical of each. The archive itself differs in entry timestamps and ordering. Compiled Python bytecode differs in embedded timestamps. The extension module differs in embedded build paths and, on some toolchains, a build identifier. Bundled native libraries differ in all of those plus whatever the library's own build recorded. The archive layer is the cheapest to fix and the native libraries the most work. the archive entry timestamps and ordering — fixed by SOURCE_DATE_EPOCH and a sort cheapest to fix, and the most common single cause compiled bytecode embedded timestamps in .pyc files — also fixed by SOURCE_DATE_EPOCH or by excluding bytecode from the wheel entirely the extension module build paths in debug info, and a build ID on some toolchains fixed by -ffile-prefix-map and by stripping bundled native libraries all of the above plus whatever GDAL's own build recorded — the most work

Context & Root Cause

A wheel is a zip archive, so it records a modification time and an order for every entry. Inside it, compiled bytecode embeds a timestamp, and compiled objects embed the absolute path they were built in — sometimes a build identifier as well. Two builds on the same machine minutes apart therefore differ, and two builds on different runners differ more.

None of that is a defect; it is metadata nobody normally looks at. It becomes a problem when you want a second party to be able to rebuild from the same inputs and confirm they get the same artifact, and it becomes a useful problem because eliminating it means eliminating hidden inputs — the floating base image, the ambient environment variable, the machine-specific path. That is the real return, as the parent reproducible builds and supply-chain attestation chapter argues.

Solution / Fix

This targets scikit-build-core 0.9+, GCC 13, auditwheel 6.x and a digest-pinned manylinux image.

1. Build twice and find out where they differ 

git clean -xdf && python -m build --wheel -o a
git clean -xdf && python -m build --wheel -o b
sha256sum a/*.whl b/*.whl

mkdir -p /tmp/a /tmp/b && unzip -q -o a/*.whl -d /tmp/a && unzip -q -o b/*.whl -d /tmp/b
diff -rq /tmp/a /tmp/b

2. Fix the archive layer first 

export SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct)
export PYTHONHASHSEED=0
export TZ=UTC LC_ALL=C.UTF-8

3. Fix the compiled layer 

export CFLAGS="-ffile-prefix-map=$(pwd)=. -g0"
export CXXFLAGS="$CFLAGS"
export LDFLAGS="-Wl,--build-id=none"

-ffile-prefix-map rewrites the build directory out of anything the compiler embeds; -g0 removes debug information entirely, which both shrinks the wheel and removes the largest remaining source of path-dependent bytes.

4. Compare the objects, not just the hashes 

for f in $(cd /tmp/a && find . -name '*.so*'); do
  cmp -s "/tmp/a/$f" "/tmp/b/$f" || echo "differs: $f"
done
# then, for one that differs:
diff <(readelf -a /tmp/a/path.so) <(readelf -a /tmp/b/path.so) | head -20

Verification 

# 1. Two clean builds agree
git clean -xdf && SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct) python -m build --wheel -o r1
git clean -xdf && SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct) python -m build --wheel -o r2
sha256sum r1/*.whl r2/*.whl | awk '{print $1}' | sort -u | wc -l
# expected: 1

# 2. Builds from two different directories agree
cp -r . /tmp/elsewhere && (cd /tmp/elsewhere && python -m build --wheel -o /tmp/r3)
sha256sum r1/*.whl /tmp/r3/*.whl | awk '{print $1}' | sort -u | wc -l
# expected: 1 — this is what -ffile-prefix-map buys

# 3. No build path survives anywhere in the artifact
unzip -o r1/*.whl -d /tmp/w >/dev/null
grep -rl "$(pwd)" /tmp/w 2>/dev/null || echo "no build paths embedded"

The second check is the interesting one, because it is the one that fails in CI even when local rebuilds agree: runners use different working directories, and a wheel that is reproducible only in one directory is not reproducible.

Working the Diff Down

Reproducibility is reached by elimination, and the order matters because each fix removes a layer of noise that would otherwise hide the next one.

The order in which to eliminate differences between two builds Five steps in order. First fix timestamps, which otherwise make every file differ and hide everything else. Then fix entry ordering. Then fix embedded build paths. Then strip debug information. Only then compare the remaining native libraries, where the causes are usually a moved toolchain or a build the library itself made non-deterministic. Each step reveals the next, which is why doing them out of order wastes time. timestamps everything differs entry order archive layout build paths in debug info strip removes the rest native libs the long tail each step removes noise that would otherwise hide the next — going straight to the native libraries wastes days and after the first four, most projects find the remaining diff is one library or none at all if a native library is still non-deterministic, the cause is usually inside its own build rather than yours

The last line is where to stop for most projects. If GDAL’s own build embeds something non-deterministic — a configure timestamp, a parallel-link ordering — fixing it means patching GDAL’s build, which is a legitimate contribution and a large one. Reaching reproducibility for everything except one vendored library still delivers the real benefit: every input to your build is declared, and a difference in the artifact now has exactly one candidate explanation.

An alternative that sidesteps the tail entirely is to make the native libraries an input rather than an output. If GDAL comes from a pinned conda package or a digest-pinned image layer, it is byte-identical by construction and never rebuilt, which removes the hardest layer from the problem.

Keeping It Reproducible

Reproducibility, once achieved, decays quietly. A check that runs on a schedule is what keeps it.

A scheduled job that rebuilds a released commit and compares hashes A weekly job checks out the last released tag, rebuilds it with the recorded environment, and compares the resulting hashes against the published artifacts. A match confirms nothing has drifted. A mismatch names the file that differs, and the usual cause is an input that moved — most often a base image or a build tool installed outside the pinned environment. check out the tag the last release rebuild recorded environment compare hashes against the published files report match, or which file a mismatch on an unchanged commit means an input moved — almost always the image or a tool outside the pin weekly is enough; the signal changes rarely and the job is one full matrix build recording the environment alongside the release is what makes the rebuild possible at all

Recording the environment is the prerequisite the last line names. A release artifact bundle containing the image digest, the resolved native versions and the compiler banner is what makes a rebuild a mechanical operation rather than an archaeology exercise, and it is the same bundle the provenance work produces anyway.

Pitfalls & Alternatives

Chasing the native libraries first. They are the hardest layer and the noise from the earlier layers hides what is actually different. Fix timestamps, ordering and paths before opening a single object file.

Using the current time as SOURCE_DATE_EPOCH. It has to be a function of the source — the commit time — or every build differs by construction while appearing to be configured correctly.

Forgetting that stripping changes the artifact. A stripped wheel and an unstripped one are different files, so the comparison must be between two builds with the same setting. That is obvious in isolation and easy to get wrong when one build was made before the flag was added.

Treating a diff in .dist-info as failure. Some backends record build-time details there. Decide whether they matter for your purposes; a wheel identical except for a generator version string is close enough for most uses and can be normalised if it is not.

Frequently Asked Questions

Is this achievable for a wheel that vendors GDAL?

The Python and extension layers, yes, and reliably. The vendored libraries depend on GDAL’s own build being deterministic, which is largely but not entirely true. Making the libraries an input — a pinned package or image layer — sidesteps the question and is what most projects end up doing.

Does reproducibility require stripping?

Not in principle, and in practice it removes the largest remaining source of path-dependent bytes, so it makes the goal much easier to reach. Since stripping is desirable for size anyway, the two goals align.

What if the difference is inside a .pyc file?

SOURCE_DATE_EPOCH covers the timestamp; a difference beyond that usually means the source differed, which points at a generated file. Excluding bytecode from the wheel entirely is also legitimate and removes the question.

Can I verify a published wheel without the build environment?

Only if the environment is recorded — the image digest, the native versions, the flags. That record is a small file and it is the difference between “you can rebuild this” and “we built this once, somehow”. Publish it with the release.

Does this conflict with a compiler cache?

No. A content-addressed cache returns objects compiled from identical inputs, so a cached build and a cold build produce the same bytes. Verifying that once on your own stack is worthwhile, and it is one of the checks described in using ccache and sccache in spatial wheel CI.

How much of the benefit comes from the last ten per cent?

Very little. Reaching “everything reproducible except one vendored library” delivers essentially the whole benefit — declared inputs, attributable differences, a rebuild that can be compared. Pursuing the final library is worthwhile only if a downstream consumer specifically requires it.

Does the wheel’s own metadata need normalising?

Sometimes. Some backends record a generator version or a build timestamp in .dist-info, which differs between builds even when everything else matches. Decide whether that counts for your purposes; if it does, a normalising pass over the archive after the build is a few lines.

What is the smallest useful first step?

Setting SOURCE_DATE_EPOCH from the commit. It costs one line, removes the difference that makes every file appear to differ, and immediately reveals whether anything else is actually wrong — which is often nothing at all for the Python layer.

How does this interact with a multi-platform matrix?

Each platform is its own comparison: an aarch64 wheel is not expected to match an x86_64 one. Reproducibility means two builds of the same cell agreeing, which is what the scheduled rebuild checks, one cell at a time.

Is a partial result worth publishing?

Yes: stating which layers are reproducible is more useful than silence.