Async Build Execution and Cache Strategies

Asynchronous build execution and deterministic caching turn a slow, serial wheel pipeline into a parallel, cache-aware execution graph. This topic sits inside Modern Python Build Tooling & Wheel Configuration, the parent reference for compiling and distributing geospatial Python packages, and focuses narrowly on execution topology and cache invalidation — not container base images or registry governance, which are covered separately. The packages that motivate it — pyproj, rasterio, shapely, and fiona — wrap heavy C/C++ toolchains (GDAL, PROJ, GEOS) where a single matrix entry routinely compiles for 15–30 minutes. Without structured parallelism and strict cache scoping, CI cost scales linearly with platform coverage and dependency resolution becomes the dominant bottleneck. This page targets cibuildwheel 2.16+, actions/cache v4, and GitHub Actions runners, and shows how to fan compilation out across a dependency graph while keeping compiler artifacts, system libraries, and packaging metadata in separate, collision-proof cache tiers.

The build fans the per-version wheel jobs out after a single shared dependency stage, then re-converges for repair:

Prerequisites & Environment #

Pin every tool that participates in the build before wiring up parallelism — cache keys are only as deterministic as the toolchain that produces the artifacts they store. Reproducibility here depends on the same provisioning discipline described in environment isolation with pixi and conda, where a committed lock file fixes the exact GDAL/PROJ/GEOS revisions a job compiles against.

Declare the floor explicitly in the workflow environment so every job in the graph inherits identical pins and the cache key remains stable across reruns:

env:
  FORCE_COLOR: "1"
  CIBW_BUILD: "cp310-* cp311-* cp312-*"
  CIBW_MANYLINUX_X86_64_IMAGE: quay.io/pypa/manylinux_2_28_x86_64
  CIBW_MANYLINUX_AARCH64_IMAGE: quay.io/pypa/manylinux_2_28_aarch64
  CIBW_BUILD_FRONTEND: "build[uv]"

The manylinux_2_28 image pin is the single most load-bearing line: it fixes the glibc symbol floor, and it must match the tag your repaired wheels claim. The trade-offs of that choice — and the musl alternative — are documented in the manylinux and manyarm Docker base images reference.

Core Configuration #

The primary configuration object is the matrix workflow. It pairs each operating system with the architecture names cibuildwheel accepts and disables fail-fast so one platform failure never aborts the rest of the graph:

# .github/workflows/build-wheels.yml
name: Build Geospatial Wheels
on: [push, pull_request]

jobs:
  build:
    runs-on: ${{ matrix.os }}
    strategy:
      fail-fast: false
      matrix:
        # Pair each OS with the arch names cibuildwheel accepts for it
        # (Linux: x86_64/aarch64, macOS: x86_64/arm64, Windows: AMD64).
        include:
          - { os: ubuntu-22.04, arch: x86_64 }
          - { os: ubuntu-22.04, arch: aarch64 }
          - { os: macos-13,     arch: x86_64 }
          - { os: macos-14,     arch: arm64 }
          - { os: windows-2022, arch: AMD64 }
    steps:
      - uses: actions/checkout@v4
      - name: Set up QEMU (needed to build Linux aarch64 via emulation)
        if: runner.os == 'Linux' && matrix.arch == 'aarch64'
        uses: docker/setup-qemu-action@v3
        with:
          platforms: arm64
      - name: Install build dependencies
        run: pip install uv build cibuildwheel
      - name: Build wheel
        run: cibuildwheel --output-dir dist
        env:
          CIBW_ARCHS: ${{ matrix.arch }}

The second configuration object is the cache. A single pip cache or actions/cache entry is insufficient for geospatial wheels because Python packaging metadata, compiled object files, and system C/C++ libraries each invalidate on a different trigger. Collapsing them into one key lets a stale .o file from a different ABI poison an otherwise clean build. Isolate caching into three deterministic tiers:

Every key must incorporate the target OS, CPU architecture, and Python ABI tag. Geospatial C-extensions are acutely sensitive to glibc versions and compiler updates: a cache hit produced on a manylinux_2_28 runner will fail ABI validation when restored onto a manylinux_2_34 target. Scope the compiler-artifact cache explicitly and provide a prefix-only restore-keys fallback so a near-miss still seeds an incremental build:

- name: Cache compiler artifacts
  uses: actions/cache@v4
  with:
    path: |
      ~/.cache/pip
      _skbuild
      build/
    key: geospatial-compile-${{ runner.os }}-${{ matrix.arch }}-${{ hashFiles('**/CMakeLists.txt', '**/pyproject.toml') }}
    restore-keys: |
      geospatial-compile-${{ runner.os }}-${{ matrix.arch }}-

The scikit-build-core cache directory layout that T2 keys on is produced by the CMake integration backend; aligning your cache paths with the directories it actually writes is what makes incremental restores hit. The build-isolation guarantees that keep T1 metadata reproducible come from your pyproject.toml configuration for spatial wheels.

Step-by-Step Implementation #

Build the graph in stages so the expensive system-library compile runs once and the cheap Python extension compile runs in parallel.

1. Split system dependencies into their own job. Compile PROJ/GDAL/GEOS once per OS+arch and publish the result as an artifact keyed on the T3 strategy:

   build-system-deps:
     runs-on: ${{ matrix.os }}
     strategy:
       matrix:
         include:
           - { os: ubuntu-22.04, arch: x86_64 }
           - { os: ubuntu-22.04, arch: aarch64 }
     steps:
       - uses: actions/checkout@v4
       - name: Restore system libs
         id: syslibs
         uses: actions/cache@v4
         with:
           path: prefix/
           key: system-deps-${{ runner.os }}-${{ matrix.arch }}-v1.2
       - name: Compile PROJ/GDAL/GEOS
         if: steps.syslibs.outputs.cache-hit != 'true'
         run: ./scripts/build_native_deps.sh --prefix "$PWD/prefix"
       - uses: actions/upload-artifact@v4
         with:
           name: syslibs-${{ runner.os }}-${{ matrix.arch }}
           path: prefix/
   

2. Gate the wheel jobs on the dependency stage. Use needs: to make the fan-out wait for the shared artifact, then download it before invoking cibuildwheel:

   build:
     needs: build-system-deps
     steps:
       - uses: actions/download-artifact@v4
         with:
           name: syslibs-${{ runner.os }}-${{ matrix.arch }}
           path: prefix/
   

3. Point the compiler at the restored prefix. Pass the vendored library location into the build environment so CMake’s find_package resolves against your pinned libs rather than whatever the runner ships:

   env:
     CIBW_ENVIRONMENT: >-
       CMAKE_PREFIX_PATH=/project/prefix
       PKG_CONFIG_PATH=/project/prefix/lib/pkgconfig
     CIBW_BEFORE_ALL_LINUX: "yum install -y pkgconfig || true"
   

4. Aggregate the parallel outputs. After the matrix completes, collect every job’s dist/ into one directory for a single repair pass:

   collect:
     needs: build
     runs-on: ubuntu-22.04
     steps:
       - uses: actions/download-artifact@v4
         with:
           pattern: cibw-wheels-*
           path: dist/
           merge-multiple: true
   

5. Repair per platform. Async execution must converge into a deterministic artifact pipeline; run the platform-appropriate repair tool to vendor and re-tag the binaries. The directory conventions for the aggregated output are covered in build artifact structuring and packaging:

   # Linux: auditwheel repair
   auditwheel repair dist/*.whl --plat manylinux_2_28_x86_64 --wheel-dir dist/repaired/
   
   # macOS: delocate
   delocate-wheel --require-archs x86_64,arm64 -w dist/repaired/ dist/*.whl
   
   # Windows: delvewheel
   delvewheel repair -w dist/repaired/ dist/*.whl
   

Verification #

Confirm each stage produced what the next stage expects before trusting a green checkmark.

Enumerate exactly which identifiers the matrix will build — this catches selector typos and unsupported OS+arch pairs before any compute is spent:

cibuildwheel --print-build-identifiers
# cp310-manylinux_x86_64
# cp311-manylinux_x86_64
# cp312-manylinux_x86_64

Inspect the repaired wheel’s platform tag and bundled libraries — the tag must name the glibc floor, not a bare linux_x86_64:

auditwheel show dist/repaired/pyproj-3.6.1-cp311-cp311-manylinux_2_28_x86_64.whl
# pyproj-...-manylinux_2_28_x86_64.whl is consistent with the
# following platform tag: "manylinux_2_28_x86_64".

Confirm the extension imports and that its shared-library lookups resolve against the vendored copies rather than the host — RPATH handling is detailed under shared library path resolution:

python -c "import pyproj; print(pyproj.proj_version_str)"
# 9.3.1
ldd $(python -c "import pyproj, os; print(os.path.dirname(pyproj.__file__))")/_proj*.so | grep proj
# libproj.so.25 => .../pyproj.libs/libproj-....so.25

To prove the cache is actually saving work, check the step log for Cache restored from key: on the second run; a Cache not found line on an unchanged tree means your key is over-specific and is silently disabling reuse.

Four Caches, Four Different Keys #

“Caching the build” is really four separate caches with different contents, different lifetimes and different consequences when they go stale. Conflating them is why some pipelines have a cache that never hits and others have one that serves the wrong objects.

The second row is the dangerous one, and it is dangerous in a way the others are not: a stale native prefix produces a wheel that builds, imports and passes an ordinary smoke test while linking a library nobody chose. That is why its key must name every input — versions, architecture, libc, compiler and the build script itself — rather than something convenient like a branch name.

The third row is the safest and the most underused. Content-addressed compiler caches cannot serve a wrong object by construction, because the key is derived from the preprocessed source and the exact flags. Enabling one costs a few lines and turns a rebuild after a one-line source change from a full GDAL compile into a few seconds of work.

What to Do When the Cache Misses Anyway #

A cache that misses on every run is worse than no cache, because it pays the storage and upload cost without the saving. Three causes account for nearly all of it, and each has a distinctive signature in the job log.

The third cause is the one worth checking first for a geospatial build, because a compiled GDAL prefix with debug symbols is large enough to run into per-entry limits on several providers — and the failure is silent by design.

Optimization & Edge Cases #

Once the graph runs, most remaining wins come from tightening the cache and pruning the matrix.

• Never share compiler caches across Python minor versions or architectures. T2 keys must carry runner.os and matrix.arch; a key that omits them is the primary cause of non-deterministic wheels, because a restored .so built for one ABI links cleanly but crashes at import on another.
• Prune impossible combinations. windows + aarch64 and similar pairs waste runner minutes. Encode the supported set in the include: list rather than excluding from a full cross-product.
• musl vs glibc. A musllinux target needs its own image, its own system-deps compile, and its own T3 key — you cannot restore glibc-built static libs into a musl build. When you add musllinux_1_2 to the matrix, branch the cache namespace; the decision criteria are laid out in manylinux2014 vs musllinux for spatial libs.
• QEMU emulation is slow. Linux aarch64 built under QEMU can run 5–10× slower than native; cache the system-deps prefix aggressively so the emulated job only ever recompiles the Python extension, never GDAL.
• ABI tags are the contract. A wheel tagged cp311-cp311-linux_x86_64 instead of cp311-cp311-manylinux_2_28_x86_64 installs but fails silently on target platforms; the underlying ABI rules are covered in C-API vs CPython ABI compatibility.

For the full breakdown of cache path scoping, invalidation triggers, and fallback chains tuned to compiled extensions, work through how to set up build caching for C-extensions.

Troubleshooting #

ERROR: Cache restore failed: another job is restoring this cache key Two matrix entries share a T2 key. Two jobs writing the same actions/cache key race, and one silently loses its artifacts. Add matrix.arch (and the Python tag if you cache per-version) to the key so each job owns a distinct namespace.

auditwheel: error: cannot repair "...-linux_x86_64.whl" to "manylinux_2_28_x86_64" ABI because of the presence of too-recent versioned symbols The wheel was compiled against a newer glibc than the target tag allows — usually a restored cache built on a manylinux_2_34 runner. Invalidate the T3 namespace (bump -v1.2 to -v1.3) and rebuild the system deps on the pinned manylinux_2_28 image.

ImportError: libgdal.so.34: cannot open shared object file: No such file or directory The repair step did not vendor GDAL, or the wheel was installed without running auditwheel repair. Confirm the repaired wheel — not the raw dist/*.whl — is what gets published, and verify auditwheel show lists libgdal under the bundled libraries.

OSError: PROJ: proj_create_from_database: Cannot find proj.db The data files were not bundled even though the shared library was. Set PROJ_DATA at runtime or include the proj.db directory in the wheel’s package data; this is a packaging gap, not an ABI fault, and it survives an otherwise-clean auditwheel repair.

Frequently Asked Questions #

Is it safe to cache a compiled native prefix at all? #

Yes, provided the key names every input that could change what is inside it: the library versions, the architecture, the libc variant, the compiler identity and the hash of the build script. The danger is not caching itself but a key that omits one of those, because the result is a wheel linked against a library nobody selected — and unlike a slow build, that failure is silent.

Should feature branches write to the shared cache? #

No. Let them read, so contributors get fast builds, and restrict writes to the default branch. A branch experimenting with a different PROJ version can otherwise publish objects that a later release build restores, which produces exactly the kind of intermittent, unreproducible failure that consumes days.

What is the difference between caching the prefix and caching compiler objects? #

The prefix cache stores the finished libraries and is keyed by you; the compiler cache stores object files and is keyed by content. The second cannot serve a wrong result by construction, so it is the safer of the two, and the two compose well: a prefix restore that misses still benefits from object-level reuse when only one dependency moved.

How do I tell whether the cache is actually helping? #

Print the statistics at the end of the job — hit rate for a compiler cache, restored key for a provider cache — and record the phase durations. A cache that silently stopped matching is invisible otherwise: the build simply takes longer, and nobody connects that to a key change made weeks earlier.

Does parallelism help a GDAL build, or is it I/O bound? #

It helps substantially, because the build is dominated by compiling many independent translation units. Setting the job count to the runner’s core count is usually right; going higher trades memory for little gain, and a GDAL build with high parallelism on a small runner can exhaust memory and fail in ways that look like compiler bugs.

Should the cache be warmed on a schedule? #

For a project whose native dependencies change rarely, yes: a nightly job that rebuilds the prefix and populates the cache means the first build after a dependency bump is the only slow one, and it happens without a person waiting for it. It also gives you an early signal when an upstream change breaks the native build, separate from any change of your own.

What is the first thing to measure in a slow spatial build? #

Whether the native stack is being compiled at all. That single question separates a build measured in minutes from one measured in tens of minutes, and no amount of parallelism or caching tuning matters until it is answered. Print the duration of each phase and look for the twelve-minute block before adjusting anything else.

Related #

• Modern Python Build Tooling & Wheel Configuration — the parent reference tying together backends, base images, and registry publishing.
• How to set up build caching for C-extensions — deep dive on cache paths, invalidation triggers, and restore-key fallbacks.
• Integrating CMake with scikit-build-core — the backend whose cache-dir layout your T2 keys depend on.
• manylinux and manyarm Docker base images — choosing the glibc/musl floor that your wheel tags and T3 cache must match.
• Build artifact structuring and packaging — how the aggregated dist/ converges into a publishable layout.

Further reading: the PEP 600 specification for the authoritative glibc-to-manylinux tag mapping, and the cibuildwheel options reference for environment overrides and per-platform compiler flags.