Geospatial C-Extension Fundamentals & ABI Architecture

Shipping a Python geospatial package means shipping compiled C and C++ across machines you will never see. Bindings for GDAL, PROJ, GEOS, and raster engines are not pure Python — they are platform-specific shared objects that must agree, byte for byte, with both the CPython interpreter that imports them and the native libraries they link against. When that agreement breaks, the failure surfaces on a user’s laptop or a production container as an ImportError or a segmentation fault, not in your CI logs. This reference is the entry point for the rest of the geospatial CI/CD library; it defines the binary-interface contracts that govern extension compilation and gives maintainers and platform teams the exact flags, decision tables, and validation commands needed for deterministic wheels across Linux, macOS, and Windows. The companion discipline — turning a manifest into a tagged artifact — lives in Modern Python Build Tooling & Wheel Configuration, and the two most load-bearing chapters here are C-API vs CPython ABI compatibility and vendoring PROJ and GDAL vs system libraries.

The pipeline below traces a geospatial C-extension from source to a portable, importable wheel:

The ABI Contract: The Non-Negotiable Rule #

A Python C-extension is a shared object (.so on Linux, .dylib repackaged as .so on macOS, .pyd on Windows) that the interpreter loads with dlopen and binds at import time. Two distinct contracts must hold simultaneously, and conflating them is the root of most distribution bugs:

1. The Python ABI — the layout of PyObject, the reference-counting macros, the signatures of every Py_* symbol the extension calls. This contract is owned by CPython and changes between minor versions (3.11 → 3.12) and, for the full API, even between patch releases of the internals.
2. The native-library ABI — the SONAME and symbol versions of GDAL, PROJ, GEOS, libsqlite3, libtiff, and friends. PROJ bumping libproj.so.25 to libproj.so.26 is an ABI break even if your Python code never changes.

The default, version-specific Python ABI ties a wheel to one interpreter minor version. A wheel tagged cp312 is meaningless to CPython 3.13. To escape the combinatorial explosion of building one wheel per interpreter, modern geospatial packaging targets the Stable ABI (the abi3 tag), a deliberately narrowed subset of the C-API that CPython guarantees not to break across the 3.x series. You opt in with the Py_LIMITED_API macro, and the rule it enforces is absolute: if you touch a symbol outside the limited set, the extension either fails to link or — worse — links and then crashes on an interpreter it was never tested against. The full version-pinning matrix and the macro-level guards that keep you inside the limited set are detailed in C-API vs CPython ABI compatibility.

Declare the contract in setup.py (or the equivalent build-backend config) so the toolchain emits an abi3 wheel:

# setup.py — opt the extension into the Stable ABI
from setuptools import setup, Extension

setup(
    ext_modules=[
        Extension(
            "_geospatial_ext",
            sources=["src/_geospatial_ext.c"],
            # Target 3.9 as the floor; the wheel imports on 3.9+ unchanged.
            define_macros=[("Py_LIMITED_API", "0x03090000")],
            extra_compile_args=["-fPIC", "-O2", "-fvisibility=hidden"],
            py_limited_api=True,  # emits the abi3 wheel tag
        )
    ]
)

In CI, the same intent is expressed as compiler flags so that direct cc invocations, CMake builds, and cibuildwheel runs all agree. The 0x03090000 value pins Python 3.9 as the floor; anything compiled with it imports cleanly on 3.9 and every later 3.x:

# Enforce position-independent code and the ABI floor for every compiler call
env:
  CFLAGS: "-fPIC -O2 -fvisibility=hidden -DPy_LIMITED_API=0x03090000"
  LDFLAGS: "-shared"

Two flags here are not optional for a shared library. -fPIC (position-independent code) is mandatory because the extension is loaded at an address chosen at runtime; omit it and the linker rejects the relocation. -fvisibility=hidden keeps your private symbols out of the dynamic symbol table, which prevents your build of GEOS from clobbering another extension’s GEOS when both are loaded into the same interpreter — the single most insidious class of geospatial import crash.

Dependency Topology: System vs. Vendored Binaries #

Geospatial extensions never compile in isolation. GDAL alone pulls in PROJ, GEOS, libtiff, libcurl, libsqlite3 (for proj.db), and often libjpeg, libpng, and libwebp. The defining architectural decision across every topic in this reference is whether those transitive libraries are vendored — compiled and bundled inside the wheel — or borrowed from the host system at runtime.

Vendoring guarantees that the exact PROJ and GDAL you tested are the ones that load, regardless of whether the target is Ubuntu 22.04, Alpine, or RHEL 9. The cost is wheel size, build time, and a license-compliance obligation for every bundled artifact. Relying on system packages produces small wheels quickly but makes the package a hostage to whatever apt, apk, or yum happened to install — and Alpine’s musl libc fractures reproducibility further because a glibc-built wheel will not even load there. The mechanics, including why a single vendored PROJ can balloon a wheel past 100 MB, are dissected in vendoring PROJ and GDAL vs system libraries and the size deep-dive in why vendoring PROJ causes wheel bloat.

For distributable PyPI wheels there is effectively one defensible default: build PROJ and GDAL as static archives or controlled shared objects during the wheel build, link the extension against them, and let auditwheel or delocate fold the runtime libraries into the wheel. The decision table below maps the trade-offs:

The conda/pixi row is the escape hatch when you control the runtime environment as well as the build: a solved pixi environment pins GDAL, PROJ, and the interpreter together in a lock file, sidestepping wheel repair entirely at the cost of requiring that same solver on the consuming side. For anything published to PyPI, however, the wheel must be self-contained, which forces the vendored path and the validation steps further down this page.

CI/CD Integration #

The ABI contract is only as good as the matrix that enforces it on every push. A geospatial wheel matrix has more dimensions than a pure-Python one because both the interpreter ABI and the native-library toolchain vary across the grid. The Stable ABI collapses the interpreter dimension — one abi3 wheel per platform instead of one per Python — but the platform and architecture axes remain. The workflow below builds repaired wheels with cibuildwheel, keying the native-build cache so that recompiling PROJ and GDAL does not dominate every run:

# .github/workflows/wheels.yml — abi3 geospatial wheel matrix
name: build-wheels
on: [push, workflow_dispatch]

jobs:
  wheels:
    runs-on: ${{ matrix.os }}
    strategy:
      fail-fast: false
      matrix:
        include:
          - os: ubuntu-latest
            arch: x86_64
            cibw_archs: x86_64
          - os: ubuntu-latest
            arch: aarch64        # emulated via QEMU
            cibw_archs: aarch64
          - os: macos-14
            arch: arm64
            cibw_archs: arm64
          - os: windows-latest
            arch: amd64
            cibw_archs: AMD64
    env:
      # One abi3 wheel covers every supported interpreter.
      CIBW_BUILD: "cp39-*"
      CIBW_BUILD_FRONTEND: "build"
      CIBW_ENVIRONMENT: 'CFLAGS="-fPIC -O2 -fvisibility=hidden -DPy_LIMITED_API=0x03090000"'
      CIBW_MANYLINUX_X86_64_IMAGE: "manylinux_2_28"
      CIBW_MUSLLINUX_X86_64_IMAGE: "musllinux_1_2"
    steps:
      - uses: actions/checkout@v4
        with: { submodules: recursive }

      - name: Set up QEMU (aarch64 emulation)
        if: matrix.arch == 'aarch64'
        uses: docker/setup-qemu-action@v3

      - name: Cache compiled PROJ/GDAL archives
        uses: actions/cache@v4
        with:
          path: ~/.cache/native-deps
          # Cache key includes OS, arch, and the pinned native versions
          key: native-${{ matrix.os }}-${{ matrix.arch }}-gdal3.9-proj9.4-${{ hashFiles('build/native_versions.lock') }}

      - name: Build & repair wheels
        uses: pypa/cibuildwheel@v2.21
        with:
          output-dir: wheelhouse

      - uses: actions/upload-artifact@v4
        with:
          name: wheels-${{ matrix.os }}-${{ matrix.arch }}
          path: wheelhouse/*.whl

Three details make this matrix specifically geospatial rather than generic. First, CIBW_MANYLINUX_X86_64_IMAGE pins the manylinux_2_28 Docker base image that fixes the glibc symbol floor every Linux wheel inherits; choosing manylinux2014 versus musllinux changes which target libcs your wheel can even load, a trade-off worked through in manylinux2014 vs musllinux for spatial libs. Second, the cache key embeds the native versions (gdal3.9-proj9.4) and a lock-file hash, not the Python version — because the expensive thing to rebuild is the C library, not the extension. The full caching strategy for these archives is covered in build caching for C-extensions. Third, aarch64 is built under QEMU emulation here for simplicity; once emulation time becomes the bottleneck, the move is native cross-compilation, which is its own discipline in cross-compiler toolchain setup.

Validation & Repair #

A wheel that builds is not a wheel that works. The repair step rewrites the binary so it carries its native dependencies and resolves them by a relative path rather than a fragile environment variable. On Linux, auditwheel inspects the extension’s external links, copies each required .so into a .libs/ directory inside the wheel, and patches the RPATH to $ORIGIN so the loader looks next to the extension. On macOS, delocate performs the equivalent for .dylibs. The full set of loader behaviors — RPATH versus RUNPATH, --disable-new-dtags, and the $ORIGIN token — is the subject of shared library path resolution and the manylinux-specific walkthrough in managing shared library paths in manylinux.

Run these checks in CI and treat any deviation as a build failure, not a warning:

# 1. What platform tag did the repair produce? Must be manylinux/musllinux, never "linux".
auditwheel show wheelhouse/_geospatial_ext-1.0-cp39-abi3-linux_x86_64.whl

# 2. Repair: bundle native libs and stamp the platform tag.
auditwheel repair --plat manylinux_2_28_x86_64 -w final/ wheelhouse/*.whl

# 3. Confirm the extension's external links are now all bundled or allowed.
unzip -o final/*.whl -d /tmp/wcheck >/dev/null
ldd /tmp/wcheck/*.libs/../_geospatial_ext*.so

# 4. The real acceptance test: install into a clean container and import.
python -c "import _geospatial_ext; print(_geospatial_ext.__file__)"

The pass/fail criteria are concrete. auditwheel show must report a versioned platform tag such as manylinux_2_28_x86_64; if it still says linux_x86_64, the wheel is not portable and PyPI will reject it. After repair, ldd against the bundled extension should show every geospatial library resolving to a path inside the wheel’s .libs/ directory, with only the permitted base-system libraries (libc, libm, libpthread, libdl) pointing at the host. The final import in a fresh manylinux-free container — a plain python:3.12-slim is ideal — is the only test that proves the contract held end to end. On macOS the equivalents are delocate-listdeps --all wheel.whl and delocate-wheel -w final/ -v wheel.whl.

Failure Modes & Diagnostics #

Five error signatures account for the overwhelming majority of geospatial extension failures. Each is reproduced verbatim so it matches what you will paste into a search bar at 2 a.m.

1. Stable-ABI violation at build time.

ImportError: ... undefined symbol: _Py_Dealloc

Root cause: the extension was compiled with Py_LIMITED_API but calls a symbol outside the limited set (here, a reference-counting internal). The macro silences the C-API header but does not police your call sites. Fix: replace the internal call with its limited-API equivalent and rebuild; the full audit of which symbols are safe is in C-API vs CPython ABI compatibility.

2. Native ABI drift — the classic GDAL/PROJ break.

ImportError: libgdal.so.34: cannot open shared object file: No such file or directory

Root cause: the wheel was linked against one GDAL SONAME and the runtime has another (or none). This is the symptom of an unrepaired wheel or of a cp-tagged version mismatch between build and run images. Fix: re-run auditwheel repair so the exact libgdal.so.34 is bundled, and verify with the ldd step above; the version-pinning resolution is detailed in how to fix ABI version mismatch in GDAL wheels.

3. Missing PROJ datum database at runtime.

pyproj.exceptions.DataDirError: Valid PROJ data directory not found. ... PROJ_LIB

Root cause: PROJ’s shared object loaded fine, but proj.db — the SQLite datum-grid database it reads at runtime — was not bundled or PROJ_LIB points nowhere. This is a data-path bug, not a linker bug, and auditwheel does not bundle data files. Fix: package proj.db as wheel data and set the search path from Python at import, or ship it inside the .libs/ payload; the vendoring trade-offs are in vendoring PROJ and GDAL vs system libraries.

4. RPATH pollution / wrong library wins.

OSError: /usr/lib/libproj.so.25: version `PROJ_9.4' not found (required by _geospatial_ext.so)

Root cause: the loader found the host PROJ before the bundled one because RUNPATH (set by the default --enable-new-dtags) is overridden by LD_LIBRARY_PATH, or because the repair left an absolute RPATH. Fix: link with -Wl,--disable-new-dtags so the bundled $ORIGIN/.libs RPATH takes precedence, and strip LD_LIBRARY_PATH from the runtime; the loader-precedence rules are mapped in shared library path resolution.

5. Symbol collision between two extensions.

Segmentation fault (core dumped)

Root cause: two extensions in the same interpreter each vendor GEOS, both export the GEOS symbols globally, and the second dlopen binds the first one’s symbols against the second one’s structs. Fix: compile with -fvisibility=hidden and link with a version script so only the module init symbol is exported, isolating allocations as covered in memory management in geospatial extensions. When the input itself is hostile — a crafted GeoTIFF or WKT string driving the crash — treat it as a boundary problem and apply the hardening in security boundaries and sandboxing.

Memory Safety and Thread Behaviour #

Geospatial workloads move large coordinate arrays, raster tiles, and topology graphs across the boundary between Python’s garbage-collected heap and manually managed C/C++ memory. Two contracts govern correctness here. The reference-counting contract requires that every PyObject you return owns exactly the references it should — an off-by-one Py_INCREF leaks raster buffers, an off-by-one Py_DECREF frees memory the interpreter still holds. The threading contract requires releasing the GIL around long-running native work with Py_BEGIN_ALLOW_THREADS / Py_END_ALLOW_THREADS, so a multi-second coordinate transform does not freeze the whole interpreter — while never touching a PyObject inside that block. Large allocations should go through PyMem_RawMalloc, which is safe to call without the GIL, rather than the system allocator. The full allocation and thread-safety playbook for spatial buffers is in memory management in geospatial extensions.

Security at the FFI Boundary #

A C-extension runs at the privilege of the interpreter process, outside every Python-level safeguard. Malformed GeoJSON, a corrupted shapefile, or a hostile PROJ pipeline string is parsed by native code that will happily overflow a buffer or follow a path-traversal if you let it. Hardening starts at compile time — -D_FORTIFY_SOURCE=2, -fstack-protector-strong, and -Wl,-z,relro,-z,now turn classes of memory bug into clean aborts — and continues with strict validation at the FFI boundary and, for untrusted input, isolation of the heavy parse in a subprocess or sandbox. The complete threat model and the hardened-build recipe live in security boundaries and sandboxing and the compile-time half in securely compiling spatial C-extensions.

Deep-Dive Guides in This Series #

The chapters below break each contract on this page into runnable detail. Start with the ABI compatibility guide if you are debugging an ImportError, or the vendoring guide if you are deciding how to ship native libraries:

• C-API vs CPython ABI compatibility — version pinning, Py_LIMITED_API guards, and the abi3 wheel strategy that collapses your interpreter matrix.
• Vendoring PROJ and GDAL vs system libraries — static vs dynamic linking, license obligations, and reproducibility across distros.
• Shared library path resolution — RPATH, RUNPATH, $ORIGIN, and the loader-precedence rules behind “cannot open shared object file”.
• Memory management in geospatial extensions — reference counting, GIL release, and allocator choice for large spatial buffers.
• Security boundaries and sandboxing — threat modeling and process isolation for untrusted spatial input.
• Cross-compiler toolchain setup — sysroots, CMAKE_TOOLCHAIN_FILE, and building aarch64 wheels without QEMU.

Conclusion #

A portable geospatial wheel is the product of two binary contracts held at once: the Stable ABI that decouples the extension from any single interpreter, and the bundled native-library payload that decouples it from any single host. Enforce Py_LIMITED_API at compile time, vendor and repair your PROJ and GDAL dependencies, and gate every build on the auditwheel show and clean-container import checks above, and your wheels will import identically on a developer laptop, a manylinux runner, and a serverless container. The most actionable next step depends on your symptom — take the C-API vs CPython ABI compatibility guide for an ImportError, or move to Modern Python Build Tooling & Wheel Configuration to wire these contracts into a declarative pyproject.toml build.

Related #

• Modern Python Build Tooling & Wheel Configuration — the companion reference that turns a declarative manifest into a tagged, repaired wheel.
• Integrating CMake with scikit-build-core — the build backend that translates pyproject.toml into the CMake invocations that compile these extensions.
• Mastering pyproject.toml for spatial wheels — declarative configuration of the ABI flags and metadata described above.
• Environment isolation with pixi and conda — the lock-file approach for reproducible build and runtime environments.

Further reading: the Stable ABI is specified in PEP 384, and the wheel platform tags this page validates against are defined by the PyPA manylinux/musllinux specifications.