Manylinux2014 vs musllinux for Spatial Libs: CI/CD Decision Matrix & Build Recovery Guide

Packaging Python geospatial stacks (GDAL, PROJ, PyProj, rasterio, shapely) requires strict ABI alignment between build hosts and target runtimes. The choice between manylinux2014 and musllinux is not a stylistic preference; it dictates glibc versus musl libc behavior under heavy C/C++17 spatial workloads. Misalignment triggers silent topology corruption, failed dynamic driver discovery, and unrepairable wheels. This guide maps exact error signatures to deterministic fixes, enforces PyPA-compliant wheel tagging, and provides pipeline recovery steps for maintainers operating within Modern Python Build Tooling & Wheel Configuration environments.

ABI Realities for Geospatial C/C++17 Stacks

manylinux2014 (defined in PEP 599) targets glibc 2.17+ (CentOS 7 baseline). It guarantees broad Linux compatibility, stable dlopen semantics, and mature auditwheel repair workflows. Spatial libraries compiled against glibc inherit predictable locale handling, thread-local destructor support, and seamless PROJ data path resolution.

musllinux (defined in PEP 656) targets musl libc 1.1+ (Alpine baseline). While it produces smaller wheels and faster cold-start containers, musl intentionally omits several glibc extensions that GDAL/PROJ implicitly depend on:

  • Missing __cxa_thread_atexit_impl and __cxa_finalize thread-local destructor semantics
  • Incomplete iconv fallback chains used by PROJ coordinate transformation pipelines
  • libm precision differences that trigger silent rasterio/GEOS topology failures
  • Strict symbol resolution that breaks gdal.so dynamic driver discovery

For production geospatial platforms, manylinux2014 remains the default. musllinux is viable only when you explicitly static-link PROJ/GDAL, disable runtime driver discovery, and validate against musl-specific test suites. Base image selection directly impacts downstream wheel repair; consult Manylinux and Manyarm Docker Base Images for verified CI runner configurations.

CI/CD Decision Matrix

Pick a baseline from the target runtime first, then refine with the table below:

flowchart TD Q1{"Target runtime libc?"} Q1 -->|"glibc: servers and cloud"| ML["manylinux2014 or manylinux_2_28"] Q1 -->|"musl: Alpine, edge/IoT"| Q2{"Can you static-link PROJ/GDAL and disable plugins?"} Q2 -->|Yes| MU["musllinux_1_1"] Q2 -->|No| ML ML --> OK["Stable dlopen, full PROJ grids"] MU --> WARN["Validate libm precision and threading"]
Workload Profile Recommended ABI Rationale
Enterprise GIS, PostGIS connectors, heavy raster processing manylinux2014_x86_64 Stable dlopen, full PROJ grid support, glibc thread safety
Edge/IoT deployments, Alpine containers, size-constrained runners musllinux_1_1_x86_64 Requires static PROJ/GDAL, disabled plugin loading, explicit PROJ_LIB
ARM64/Graviton spatial inference manylinux2014_aarch64 Cross-compiled via QEMU or native runners; musl ARM spatial support remains fragmented
Pure-Python fallback or WASM targets N/A Bypass C extensions entirely; use pyproj pure-Python coordinate transforms

Exact Error Signatures & Root Cause Mapping

Error Signature Root Cause Immediate Fix
ImportError: .../pyproj/_proj.cpython-310-x86_64-linux-gnu.so: undefined symbol: __cxa_thread_atexit_impl musl libc lacks glibc thread-local destructor ABI. Rebuild targeting manylinux2014 or compile PROJ with -DCMAKE_THREAD_LOCAL_STORAGE=OFF.
OSError: /lib/x86_64-linux-gnu/libc.so.6: version 'GLIBC_2.32' not found Wheel built on newer glibc host, deployed to older runtime. Pin CIBW_BUILD=cp310-manylinux2014_x86_64 and enforce manylinux2014 base in CI.
auditwheel: ERROR: cannot repair wheel to "musllinux_1_1_x86_64" because it contains libraries with incompatible ABI tags auditwheel cannot patch musl-compiled C++ stdlib or PROJ grid dependencies. Switch to manylinux2014 or use patchelf --replace-needed with static PROJ builds.
CRITICAL: proj_create_operations: Cannot find proj.db musl Alpine containers strip /usr/share/proj or use non-standard paths. Set PROJ_LIB=/usr/local/share/proj in container env and bundle grids in wheel data directory.
GEOSException: IllegalArgumentException: TopologyException: side location conflict musl libm rounding differences alter GEOS intersection tolerances. Recompile GEOS with -ffloat-store or switch to glibc baseline for deterministic geometry ops.
ImportError: libgdal.so.33: cannot open shared object file: No such file or directory Dynamic GDAL driver loading fails under musl strict RTLD_LOCAL defaults. Set LD_PRELOAD=/usr/local/lib/libgdal.so or rebuild with -DGDAL_ENABLE_DRIVER_PLUGIN=OFF.

Build Recovery & Pipeline Validation

When a spatial wheel fails in staging, execute the following recovery sequence before triggering a full rebuild:

  1. Verify Wheel ABI Tag
  python -c "from packaging.utils import parse_wheel_filename as p; print(p('your_wheel.whl'))"
  # Expected: ('your_wheel', <Version>, (), frozenset({cp310-cp310-manylinux2014_x86_64}))
  1. Inspect Dynamic Dependencies
  auditwheel show your_wheel.whl
  python -m wheel unpack your_wheel.whl -d /tmp/wh
  find /tmp/wh -name '*.so' -exec readelf -d {} + | grep NEEDED
  # Confirm no glibc 2.32+ symbols leak into manylinux2014 wheels
  1. Validate PROJ Data Resolution
  python -c "import pyproj; print(pyproj.datadir.get_data_dir())"
  # Must return absolute path inside site-packages, not system /usr/share
  1. Enforce Deterministic Build Config Add to pyproject.toml:
  [tool.cibuildwheel]
  # The build selector uses the generic platform name; the 2014 policy is
  # selected via manylinux-x86_64-image, not inside the selector string.
  build = "cp3{9,10,11,12}-manylinux_x86_64"
  manylinux-x86_64-image = "manylinux2014"
  environment = { PROJ_LIB="/project/share/proj", GEOS_CAPI_VERSION="3.11.0" }
  # CentOS 7-based manylinux2014 has no usable proj-devel/gdal-devel packages,
  # so build PROJ/GDAL from source before the wheel build.
  before-all = "yum install -y gcc-c++ sqlite-devel libtiff-devel libcurl-devel zlib-devel && bash scripts/build_geospatial_deps.sh"
  test-command = "python -c \"import rasterio, shapely, pyproj; print('ABI OK')\""
  1. Run Spatial Validation Suite Execute against a known-good dataset:
  pytest tests/ --spatial-fixtures=tests/data/valid_geotiff.tif \
    -k "test_transform_and_rasterize" --tb=short

PyPA & Spatial Standards Compliance Checklist

Operational Takeaways

  1. Default to manylinux2014 for all production spatial workloads. The ABI stability and mature repair tooling outweigh container size benefits.
  2. Reserve musllinux only for constrained edge deployments where you control the full runtime stack and can validate against musl-specific precision drift.
  3. Never trust pip install success as ABI validation. Always run auditwheel show, verify PROJ_LIB resolution, and execute geometry/raster regression tests in CI.
  4. Pin C/C++ dependency versions in pyproject.toml build-system requirements. Floating spatial library versions cause silent ABI breaks across minor releases.

Maintaining deterministic spatial wheels requires strict adherence to glibc baselines, explicit data path management, and automated ABI verification. Implement the validation steps above to eliminate runtime import failures and topology corruption in production pipelines.