Async Build Execution and Cache Strategies

Asynchronous build execution and deterministic caching turn a slow, serial wheel pipeline into a parallel, cache-aware execution graph. This topic sits inside Modern Python Build Tooling & Wheel Configuration, the parent reference for compiling and distributing geospatial Python packages, and focuses narrowly on execution topology and cache invalidation — not container base images or registry governance, which are covered separately. The packages that motivate it — pyproj, rasterio, shapely, and fiona — wrap heavy C/C++ toolchains (GDAL, PROJ, GEOS) where a single matrix entry routinely compiles for 15–30 minutes. Without structured parallelism and strict cache scoping, CI cost scales linearly with platform coverage and dependency resolution becomes the dominant bottleneck. This page targets cibuildwheel 2.16+, actions/cache v4, and GitHub Actions runners, and shows how to fan compilation out across a dependency graph while keeping compiler artifacts, system libraries, and packaging metadata in separate, collision-proof cache tiers.

The build fans the per-version wheel jobs out after a single shared dependency stage, then re-converges for repair:

Prerequisites & Environment #

Pin every tool that participates in the build before wiring up parallelism — cache keys are only as deterministic as the toolchain that produces the artifacts they store. Reproducibility here depends on the same provisioning discipline described in environment isolation with pixi and conda, where a committed lock file fixes the exact GDAL/PROJ/GEOS revisions a job compiles against.

Declare the floor explicitly in the workflow environment so every job in the graph inherits identical pins and the cache key remains stable across reruns:

env:
  FORCE_COLOR: "1"
  CIBW_BUILD: "cp310-* cp311-* cp312-*"
  CIBW_MANYLINUX_X86_64_IMAGE: quay.io/pypa/manylinux_2_28_x86_64
  CIBW_MANYLINUX_AARCH64_IMAGE: quay.io/pypa/manylinux_2_28_aarch64
  CIBW_BUILD_FRONTEND: "build[uv]"

The manylinux_2_28 image pin is the single most load-bearing line: it fixes the glibc symbol floor, and it must match the tag your repaired wheels claim. The trade-offs of that choice — and the musl alternative — are documented in the manylinux and manyarm Docker base images reference.

Core Configuration #

The primary configuration object is the matrix workflow. It pairs each operating system with the architecture names cibuildwheel accepts and disables fail-fast so one platform failure never aborts the rest of the graph:

# .github/workflows/build-wheels.yml
name: Build Geospatial Wheels
on: [push, pull_request]

jobs:
  build:
    runs-on: ${{ matrix.os }}
    strategy:
      fail-fast: false
      matrix:
        # Pair each OS with the arch names cibuildwheel accepts for it
        # (Linux: x86_64/aarch64, macOS: x86_64/arm64, Windows: AMD64).
        include:
          - { os: ubuntu-22.04, arch: x86_64 }
          - { os: ubuntu-22.04, arch: aarch64 }
          - { os: macos-13,     arch: x86_64 }
          - { os: macos-14,     arch: arm64 }
          - { os: windows-2022, arch: AMD64 }
    steps:
      - uses: actions/checkout@v4
      - name: Set up QEMU (needed to build Linux aarch64 via emulation)
        if: runner.os == 'Linux' && matrix.arch == 'aarch64'
        uses: docker/setup-qemu-action@v3
        with:
          platforms: arm64
      - name: Install build dependencies
        run: pip install uv build cibuildwheel
      - name: Build wheel
        run: cibuildwheel --output-dir dist
        env:
          CIBW_ARCHS: ${{ matrix.arch }}

The second configuration object is the cache. A single pip cache or actions/cache entry is insufficient for geospatial wheels because Python packaging metadata, compiled object files, and system C/C++ libraries each invalidate on a different trigger. Collapsing them into one key lets a stale .o file from a different ABI poison an otherwise clean build. Isolate caching into three deterministic tiers:

Every key must incorporate the target OS, CPU architecture, and Python ABI tag. Geospatial C-extensions are acutely sensitive to glibc versions and compiler updates: a cache hit produced on a manylinux_2_28 runner will fail ABI validation when restored onto a manylinux_2_34 target. Scope the compiler-artifact cache explicitly and provide a prefix-only restore-keys fallback so a near-miss still seeds an incremental build:

- name: Cache compiler artifacts
  uses: actions/cache@v4
  with:
    path: |
      ~/.cache/pip
      _skbuild
      build/
    key: geospatial-compile-${{ runner.os }}-${{ matrix.arch }}-${{ hashFiles('**/CMakeLists.txt', '**/pyproject.toml') }}
    restore-keys: |
      geospatial-compile-${{ runner.os }}-${{ matrix.arch }}-

The scikit-build-core cache directory layout that T2 keys on is produced by the CMake integration backend; aligning your cache paths with the directories it actually writes is what makes incremental restores hit. The build-isolation guarantees that keep T1 metadata reproducible come from your pyproject.toml configuration for spatial wheels.

Step-by-Step Implementation #

Build the graph in stages so the expensive system-library compile runs once and the cheap Python extension compile runs in parallel.

1. Split system dependencies into their own job. Compile PROJ/GDAL/GEOS once per OS+arch and publish the result as an artifact keyed on the T3 strategy:

   build-system-deps:
     runs-on: ${{ matrix.os }}
     strategy:
       matrix:
         include:
           - { os: ubuntu-22.04, arch: x86_64 }
           - { os: ubuntu-22.04, arch: aarch64 }
     steps:
       - uses: actions/checkout@v4
       - name: Restore system libs
         id: syslibs
         uses: actions/cache@v4
         with:
           path: prefix/
           key: system-deps-${{ runner.os }}-${{ matrix.arch }}-v1.2
       - name: Compile PROJ/GDAL/GEOS
         if: steps.syslibs.outputs.cache-hit != 'true'
         run: ./scripts/build_native_deps.sh --prefix "$PWD/prefix"
       - uses: actions/upload-artifact@v4
         with:
           name: syslibs-${{ runner.os }}-${{ matrix.arch }}
           path: prefix/
   

2. Gate the wheel jobs on the dependency stage. Use needs: to make the fan-out wait for the shared artifact, then download it before invoking cibuildwheel:

   build:
     needs: build-system-deps
     steps:
       - uses: actions/download-artifact@v4
         with:
           name: syslibs-${{ runner.os }}-${{ matrix.arch }}
           path: prefix/
   

3. Point the compiler at the restored prefix. Pass the vendored library location into the build environment so CMake’s find_package resolves against your pinned libs rather than whatever the runner ships:

   env:
     CIBW_ENVIRONMENT: >-
       CMAKE_PREFIX_PATH=/project/prefix
       PKG_CONFIG_PATH=/project/prefix/lib/pkgconfig
     CIBW_BEFORE_ALL_LINUX: "yum install -y pkgconfig || true"
   

4. Aggregate the parallel outputs. After the matrix completes, collect every job’s dist/ into one directory for a single repair pass:

   collect:
     needs: build
     runs-on: ubuntu-22.04
     steps:
       - uses: actions/download-artifact@v4
         with:
           pattern: cibw-wheels-*
           path: dist/
           merge-multiple: true
   

5. Repair per platform. Async execution must converge into a deterministic artifact pipeline; run the platform-appropriate repair tool to vendor and re-tag the binaries. The directory conventions for the aggregated output are covered in build artifact structuring and packaging:

   # Linux: auditwheel repair
   auditwheel repair dist/*.whl --plat manylinux_2_28_x86_64 --wheel-dir dist/repaired/
   
   # macOS: delocate
   delocate-wheel --require-archs x86_64,arm64 -w dist/repaired/ dist/*.whl
   
   # Windows: delvewheel
   delvewheel repair -w dist/repaired/ dist/*.whl
   

Verification #

Confirm each stage produced what the next stage expects before trusting a green checkmark.

Enumerate exactly which identifiers the matrix will build — this catches selector typos and unsupported OS+arch pairs before any compute is spent:

cibuildwheel --print-build-identifiers
# cp310-manylinux_x86_64
# cp311-manylinux_x86_64
# cp312-manylinux_x86_64

Inspect the repaired wheel’s platform tag and bundled libraries — the tag must name the glibc floor, not a bare linux_x86_64:

auditwheel show dist/repaired/pyproj-3.6.1-cp311-cp311-manylinux_2_28_x86_64.whl
# pyproj-...-manylinux_2_28_x86_64.whl is consistent with the
# following platform tag: "manylinux_2_28_x86_64".

Confirm the extension imports and that its shared-library lookups resolve against the vendored copies rather than the host — RPATH handling is detailed under shared library path resolution:

python -c "import pyproj; print(pyproj.proj_version_str)"
# 9.3.1
ldd $(python -c "import pyproj, os; print(os.path.dirname(pyproj.__file__))")/_proj*.so | grep proj
# libproj.so.25 => .../pyproj.libs/libproj-....so.25

To prove the cache is actually saving work, check the step log for Cache restored from key: on the second run; a Cache not found line on an unchanged tree means your key is over-specific and is silently disabling reuse.

Optimization & Edge Cases #

Once the graph runs, most remaining wins come from tightening the cache and pruning the matrix.

• Never share compiler caches across Python minor versions or architectures. T2 keys must carry runner.os and matrix.arch; a key that omits them is the primary cause of non-deterministic wheels, because a restored .so built for one ABI links cleanly but crashes at import on another.
• Prune impossible combinations. windows + aarch64 and similar pairs waste runner minutes. Encode the supported set in the include: list rather than excluding from a full cross-product.
• musl vs glibc. A musllinux target needs its own image, its own system-deps compile, and its own T3 key — you cannot restore glibc-built static libs into a musl build. When you add musllinux_1_2 to the matrix, branch the cache namespace; the decision criteria are laid out in manylinux2014 vs musllinux for spatial libs.
• QEMU emulation is slow. Linux aarch64 built under QEMU can run 5–10× slower than native; cache the system-deps prefix aggressively so the emulated job only ever recompiles the Python extension, never GDAL.
• ABI tags are the contract. A wheel tagged cp311-cp311-linux_x86_64 instead of cp311-cp311-manylinux_2_28_x86_64 installs but fails silently on target platforms; the underlying ABI rules are covered in C-API vs CPython ABI compatibility.

For the full breakdown of cache path scoping, invalidation triggers, and fallback chains tuned to compiled extensions, work through how to set up build caching for C-extensions.

Troubleshooting #

ERROR: Cache restore failed: another job is restoring this cache key Two matrix entries share a T2 key. Two jobs writing the same actions/cache key race, and one silently loses its artifacts. Add matrix.arch (and the Python tag if you cache per-version) to the key so each job owns a distinct namespace.

auditwheel: error: cannot repair "...-linux_x86_64.whl" to "manylinux_2_28_x86_64" ABI because of the presence of too-recent versioned symbols The wheel was compiled against a newer glibc than the target tag allows — usually a restored cache built on a manylinux_2_34 runner. Invalidate the T3 namespace (bump -v1.2 to -v1.3) and rebuild the system deps on the pinned manylinux_2_28 image.

ImportError: libgdal.so.34: cannot open shared object file: No such file or directory The repair step did not vendor GDAL, or the wheel was installed without running auditwheel repair. Confirm the repaired wheel — not the raw dist/*.whl — is what gets published, and verify auditwheel show lists libgdal under the bundled libraries.

OSError: PROJ: proj_create_from_database: Cannot find proj.db The data files were not bundled even though the shared library was. Set PROJ_DATA at runtime or include the proj.db directory in the wheel’s package data; this is a packaging gap, not an ABI fault, and it survives an otherwise-clean auditwheel repair.

Related #

• Modern Python Build Tooling & Wheel Configuration — the parent reference tying together backends, base images, and registry publishing.
• How to set up build caching for C-extensions — deep dive on cache paths, invalidation triggers, and restore-key fallbacks.
• Integrating CMake with scikit-build-core — the backend whose cache-dir layout your T2 keys depend on.
• manylinux and manyarm Docker base images — choosing the glibc/musl floor that your wheel tags and T3 cache must match.
• Build artifact structuring and packaging — how the aggregated dist/ converges into a publishable layout.

Further reading: the PEP 600 specification for the authoritative glibc-to-manylinux tag mapping, and the cibuildwheel options reference for environment overrides and per-platform compiler flags.