Step-by-step C-extension lifecycle for Python GIS

This page traces the exact lifecycle of a geospatial Python C-extension — from source compilation to a deployed, importable wheel — and pinpoints where each phase breaks when GDAL, PROJ, or PyProj bindings fail in CI/CD. It sits under the C-API vs CPython ABI Compatibility cluster, part of the broader Geospatial C-Extension Fundamentals & ABI Architecture reference. Versions assumed throughout: CPython 3.9–3.12, setuptools 68+, Cython 3.0+, cibuildwheel 2.16+, auditwheel 6.x, PROJ 9.3.x, and GDAL 3.8.x.

The five phases form a linear pipeline, each gated by validation before the next begins:

Five-phase geospatial C-extension lifecycle with validation gates A linear pipeline of five phases — ABI contract and toolchain, dependency resolution, build isolation, RPATH hardening, and CI/CD deploy. Between each phase a gate names the check that must pass first: EXT_SUFFIX equals the ABI tag, pkg-config versions pinned, isolated toolchain, ldd reports no missing libraries. Five-phase lifecycle — each phase gated before the next begins 1 ABI contract & toolchain 2 Dependency resolution 3 Build isolation 4 RPATH hardening 5 CI/CD deploy gate: EXT_SUFFIX = ABI tag gate: pkg-config pinned gate: isolated toolchain gate: ldd — no missing

Context & Root Cause

When a geospatial wheel that built green suddenly fails at import time, it is rarely a logic error. The extension encodes two independent binary contracts that pip never fully checks. The first is the interpreter ABI — whether the compiled .so was built for one specific CPython minor version or against the Stable ABI (abi3), governed by Py_LIMITED_API. The second is the native ABI of libgdal/libproj, exported as versioned symbols that hold only within a major release.

A failure surfaces when one of these contracts drifts: a missing dynamic symbol (undefined symbol), an absolute or shadowed rpath so the loader can’t find a vendored .so (cannot open shared object file), a glibc floor set too high by the wrong base image, or a C++ name-mangling mismatch. Treating these as a single “lifecycle” rather than five isolated steps lets you bisect the failure to the exact phase that introduced it.

Solution / Fix

Phase 1 — ABI contract & toolchain initialization

Decide the interpreter ABI target before anything compiles. Targeting abi3 emits one wheel for all supported Pythons and shrinks the build matrix, but it restricts you to the limited C-API subset; full version-specific builds give you the whole C-API at the cost of one wheel per minor release. The trade-off and symbol rules are dissected in the parent C-API vs CPython ABI Compatibility cluster. The classic trap is mixing abi3 with GDAL/PROJ C++ wrappers and hitting undefined symbol: PyUnicode_AsUTF8 at import, because that symbol entered the limited API only in 3.10.

[build-system]
requires = ["setuptools>=68.0", "wheel", "Cython>=3.0", "setuptools_scm[toml]>=8.0"]
build-backend = "setuptools.build_meta"

[project]
name = "my-geospatial-ext"
requires-python = ">=3.9"
classifiers = [
    "Programming Language :: Python :: 3 :: Only",
    "Programming Language :: Python :: 3.9",
    "Programming Language :: Python :: 3.10",
    "Programming Language :: Python :: 3.11",
    "Programming Language :: Python :: 3.12",
]

[tool.distutils.bdist_wheel]
py-limited-api = "cp39"  # Emit a single abi3 wheel (Python 3.9+)

The full project-table layout for spatial packages is covered in mastering pyproject.toml for spatial wheels.

Phase 2 — geospatial dependency resolution & linkage

GDAL and PROJ drag in transitive dependencies (libcurl, libsqlite3, libtiff, libproj, libgeos). The decision to bundle these or link a pinned system baseline is made deliberately — see vendoring PROJ and GDAL vs system libraries. Inside CI the build runs in a manylinux base image, but pkg-config frequently resolves stale headers or a mismatched libproj minor version, so pin and verify the versions before compiling.

env:
  PROJ_VERSION: "9.3.1"
  GDAL_VERSION: "3.8.4"
  CFLAGS: "-O2 -fPIC -Wno-unused-variable -Wno-implicit-function-declaration"
  LDFLAGS: "-Wl,-rpath,'$ORIGIN/.libs' -Wl,--no-undefined"

steps:
  - name: Install PROJ & GDAL (manylinux compatible)
    run: |
      yum install -y epel-release
      yum install -y gcc gcc-c++ make cmake pkgconfig sqlite-devel curl-devel \
        proj-devel proj proj-data gdal-devel
  - name: Verify pkg-config resolution
    run: |
      pkg-config --modversion proj
      pkg-config --modversion gdal
      # Must match env vars exactly. Mismatch = immediate runtime segfault.

Phase 3 — build isolation & toolchain setup

Deterministic builds need strict environment isolation; never link against a host toolchain. Drive cross-platform compilation with cibuildwheel across manylinux_2_28, manylinux_2_34, and musllinux_1_2 targets, and pin the compiler set as described in cross-compiler toolchain setup. If the native build is CMake-based, route it through the scikit-build-core backend so pyproject.toml translates cleanly into CMake invocations; reproducible toolchain pins can be sourced from a pixi or conda environment.

[tool.cibuildwheel]
build-frontend = "build"
skip = ["cp38-*", "pp*"]
manylinux-x86_64-image = "quay.io/pypa/manylinux_2_28_x86_64:latest"
environment = { PROJ_LIB="/usr/share/proj", GDAL_DATA="/usr/share/gdal" }
before-all = "yum install -y proj-devel gdal-devel"
test-command = "python -c \"import my_ext; assert hasattr(my_ext, 'transform_coords')\""

Phase 4 — shared library path resolution & rpath hardening

Dynamic linking fails when LD_LIBRARY_PATH is unset or when rpath points at an absolute build directory that does not exist on the deployment host. Wheels must use $ORIGIN-relative paths so the loader finds the bundled .so files regardless of install location — the mechanics of how the loader chooses between vendored and system copies are detailed in managing shared library paths in manylinux. The auditwheel repair step automates $ORIGIN injection and dependency bundling.

Absolute RPATH failure vs auditwheel $ORIGIN-relative repair Before: the compiled extension carries DT_RPATH set to an absolute build path /build/proj/lib; the ELF loader follows it on the deploy host, the directory is absent, and the import fails with OSError cannot open libgdal.so.32. auditwheel repair rewrites the RPATH to $ORIGIN/.libs and copies libgdal.so.32 and libproj.so.25 into a .libs directory beside the extension. After: the loader resolves the bundled libraries relative to the .so location and the import succeeds. Before · build-tree absolute RPATH — breaks off the build host my_ext.cpython-311.so DT_RPATH = /build/proj/lib ld.so follows RPATH /build/proj/lib — absent on deploy host ✗ OSError: cannot open libgdal.so.32 auditwheel repair --plat … After · auditwheel repair — $ORIGIN-relative RPATH, deps bundled site-packages/my_ext/ my_ext.cpython-311.so DT_RPATH = $ORIGIN/.libs resolves relative to the .so $ORIGIN/.libs ✓ .libs/ libgdal.so.32 libproj.so.25 
# 1. Inspect current rpath/runpath (on the .so in the build tree, before wheel packaging)
readelf -d dist/my_ext/*.so | grep -E "RPATH|RUNPATH"

# 2. Patch with auditwheel (automates $ORIGIN injection and dependency bundling)
auditwheel repair dist/*.whl --plat manylinux_2_28_x86_64 -w dist/repaired/

# 3. Verify post-repair linkage (extract the wheel first; ldd reads ELF, not zip)
unzip -o -q dist/repaired/*.whl -d /tmp/wheel_check
ldd /tmp/wheel_check/my_ext/*.so | grep "not found"
# Output must be empty. Any "not found" indicates missing transitive deps.

Phase 5 — CI/CD pipeline integration & deterministic deployment

Close the lifecycle by enforcing deterministic artifacts, verifying tags, and gating PyPI promotion. The internal wheel layout these checks assume is described in build artifact structuring and packaging.

# 1. Validate wheel structure against PEP 427
unzip -l dist/repaired/*.whl | grep -E "\.pyc|\.so|\.libs"

# 2. Verify ABI compatibility (packaging >= 23 exposes tags directly)
python -c "from packaging.tags import sys_tags; print([str(t) for t in list(sys_tags())[:5]])"

# 3. Upload to test index
twine upload --repository-url https://test.pypi.org/legacy/ dist/repaired/*.whl

Verification

Run these confirmations as gates between phases; each prints a value you can assert on.

# Phase 1 — confirm the emitted ABI tag matches intent
python -c "import sysconfig; print(sysconfig.get_config_var('EXT_SUFFIX'))"
# Version-specific build -> .cpython-311-x86_64-linux-gnu.so
# Stable-ABI (abi3) build -> .abi3.so

# Phase 3 — confirm toolchain architecture alignment
file dist/*.so | grep -E "ELF 64-bit LSB|ARM|AArch64"
# Must match target platform. Mismatch = ImportError: wrong ELF class

# Phase 4 — confirm the injected runpath
readelf -d /tmp/wheel_check/my_ext/*.so | grep RUNPATH
# Expected: Library runpath: [$ORIGIN/.libs]

When a phase fails, map the verbatim error signature to its phase and fix:

Error Signature Root Cause Exact Fix Validation Step
ImportError: ... undefined symbol: PyUnicode_AsUTF8 abi3 wheel built for Python < 3.10 using a symbol added to the limited API only in 3.10 Raise the floor to -D Py_LIMITED_API=0x030A0000 (3.10) or drop py-limited-api and build version-specific wheels python -c "import sysconfig; print(sysconfig.get_config_var('EXT_SUFFIX'))" must match the wheel tag
OSError: libgdal.so.32: cannot open shared object file Missing rpath or LD_LIBRARY_PATH not propagated Run auditwheel repair with --plat matching the target readelf -d *.so | grep RUNPATH must contain $ORIGIN/.libs
ImportError: ... undefined symbol: _ZTVN4proj11Coordinate... C++ name mangling mismatch or missing -lstdc++ in linkage Append -lstdc++ to LDFLAGS and ensure extern "C" wrappers nm -D *.so | grep _ZTVN should return 0 results
ValueError: PROJ: proj_create_from_database: Cannot find proj.db PROJ_LIB not embedded or runtime path missing Set PROJ_LIB in wheel metadata or bundle proj.db in .libs python -c "import pyproj; print(pyproj.datadir.get_data_dir())" returns a valid path
Segmentation fault (core dumped) ABI mismatch between GDAL C++ API and the Python extension Rebuild against the exact GDAL minor version; verify the GDAL_VERSION pin gdb --args python -c "import my_ext" → bt shows the crash in libgdal

The full GDAL-specific triage for the first two rows lives in how to fix ABI version mismatch in GDAL wheels.

Pitfalls & Alternatives

  • Building on a host that already has system GDAL/PROJ. The compiler silently links the host copy, the wheel passes its own machine, then breaks everywhere else with undefined symbol. Compile only inside a pinned manylinux base image that has no system spatial stack.
  • Reaching for LD_LIBRARY_PATH to make an import succeed. Widening the loader search path masks the missing rpath and bakes in the very mismatch you were chasing — it works on your box and fails on the next. Fix rpath with auditwheel repair so resolution is $ORIGIN-relative.
  • Bumping to the newest manylinux tag for “compatibility.” A higher manylinux_2_NN raises the glibc floor and triggers GLIBC_2.NN not found on older hosts. Pick the oldest base image your fleet still runs, and treat ABI drift as a critical pipeline failure rather than a warning — these extensions are infrastructure components.