Memory Management in Geospatial Extensions

Memory management in Python geospatial extensions requires strict boundary enforcement between the interpreter’s garbage collector and the native heap allocations managed by C/C++ libraries like GDAL, PROJ, and GEOS. This guide sits under the Geospatial C-Extension Fundamentals & ABI Architecture reference, which establishes interface contracts, symbol visibility, and cross-platform linking; here the focus narrows to allocation lifecycles, pointer ownership transfer, and CI-driven leak validation. Where C-API vs CPython ABI Compatibility governs which symbols a wheel may import and Vendoring PROJ and GDAL vs System Libraries governs which allocator arena those symbols resolve into, this page covers deterministic teardown, native heap tracking, and wheel-build pipeline validation. It targets CPython 3.9–3.12, GDAL 3.6–3.9, PROJ 9.2+, GEOS 3.11+, cibuildwheel 2.16+, auditwheel 6.x, delocate 0.11+, and valgrind 3.19+.

Prerequisites & Environment #

Pin every native and interpreter version before you write a single destructor. Memory behaviour is sensitive to allocator builds, and a leak that reproduces under one GDAL minor can vanish under another, so reproducibility starts with a locked toolchain. The same pins should match the pixi environment you use elsewhere in the build so the leak you debug locally is the leak CI sees.

Set the interpreter to system malloc whenever you measure. With pymalloc active, the interpreter pools small objects in 256 KiB arenas that mask native allocations from tracemalloc, Valgrind, and AddressSanitizer:

# Local debug shell — disable pymalloc and arm the fault handler
export PYTHONMALLOC=malloc
export PYTHONFAULTHANDLER=1
export PYTHONDEVMODE=1   # surfaces ResourceWarning on un-closed handles
python -X dev -c "import my_geospatial_ext"

Core Configuration #

Native geospatial libraries do not use Python’s memory subsystem. GDALOpenEx, OGRGeometryFactory::createGeometry, and proj_create_context reserve contiguous native heap blocks that persist independently of any PyObject lifetime, and they must be released through GDALClose, OGR_G_DestroyGeometry, and proj_context_destroy. Python’s reference counting and cyclic garbage collector operate exclusively on PyObject structures; they cannot introspect or reclaim memory allocated via malloc, posix_memalign, or a library-specific allocator. The bridge between those two worlds is PyCapsule: a capsule binds a native pointer to a destructor callback that runs exactly once when the reference count drops to zero.

/* capsule.c — deterministic ownership container for a GDAL dataset */
static void capsule_destructor(PyObject *capsule) {
    void *ptr = PyCapsule_GetPointer(capsule, "gdal_dataset");
    if (ptr) {
        GDALClose((GDALDatasetH)ptr);
    }
    /* The capsule object itself is being destroyed; its slot need not be
       cleared. PyCapsule_SetPointer rejects NULL and would re-enter the C-API. */
}

/* Creation — name string must match the destructor's PyCapsule_GetPointer key */
PyObject *capsule = PyCapsule_New(dataset, "gdal_dataset", capsule_destructor);

For large internal buffers — coordinate arrays, raster tiles, projection grids — bypass the pymalloc arena with PyMem_RawMalloc and PyMem_RawFree. This keeps allocations visible to system-level profilers and avoids fragmenting the small-object cache. When you implement tp_dealloc slots or Cython __dealloc__ methods, three rules are non-negotiable:

1. Nullify immediately after destruction. Set internal pointers to NULL right after calling the native destroy function; cyclic GC may invoke tp_dealloc more than once during complex teardown.
2. Never call the Python C-API in a destructor. PyErr_SetString, Py_DECREF, or importing a module inside tp_dealloc can trigger GC reentrancy or a segfault during interpreter finalization. Use fprintf(stderr, ...) or Py_FatalError for diagnostics.
3. Embed allocator metadata. Use PyCapsule_SetContext to carry the library version and context handle so cross-version ABI shifts do not force a hardcoded destroy function — the same versioned-symbol discipline enforced under C-API vs CPython ABI Compatibility.

The capsule lifecycle below shows where native memory is actually released:

Step-by-Step Implementation #

Each step below is runnable as written; treat them as the order in which a handle moves from native allocation to deterministic release.

1. Allocate the native object and immediately hand ownership to a capsule. Never expose a bare pointer to Python code where an exception could leak it:

GDALDatasetH ds = GDALOpenEx(path, GDAL_OF_RASTER, NULL, NULL, NULL);
if (!ds) {
    PyErr_Format(PyExc_OSError, "GDALOpenEx failed for %s", path);
    return NULL;
}
PyObject *cap = PyCapsule_New(ds, "gdal_dataset", capsule_destructor);
if (!cap) {              /* capsule creation failed: free now, do not leak */
    GDALClose(ds);
    return NULL;
}
return cap;              /* ownership now lives in the capsule's refcount */

2. Allocate large buffers off the pymalloc arena. Coordinate batches and raster tiles belong on the raw allocator so a profiler attributes them correctly:

size_t nbytes = (size_t)width * height * sizeof(double);
double *buf = PyMem_RawMalloc(nbytes);
if (!buf) return PyErr_NoMemory();
/* ... fill from CPLReadBlock / proj_trans_array ... */
PyMem_RawFree(buf);      /* pair every PyMem_RawMalloc with PyMem_RawFree */

3. Define a PROJ context per object and destroy it before the dataset. PROJ caches grids in a thread-local context; closing the dataset first can dangle those grid pointers:

PJ_CONTEXT *ctx = proj_context_create();
PJ *P = proj_create_crs_to_crs(ctx, "EPSG:4326", "EPSG:3857", NULL);
/* ... use P ... */
proj_destroy(P);
proj_context_destroy(ctx);   /* destroy context AFTER the PJ it created */

4. Wire the CI environment to disable pymalloc and arm diagnostics. Apply the same env to both the test job and the cibuildwheel test step so the build-time check matches local debugging:

# .github/workflows/build-and-test.yml
env:
  PYTHONMALLOC: malloc
  PYTHONFAULTHANDLER: "1"
  CIBW_ENVIRONMENT: "PYTHONMALLOC=malloc PYTHONFAULTHANDLER=1"

5. Configure the wheel build to bundle and verify native libraries. The allocators you track only behave deterministically if the same libgdal/libproj they were compiled against are the ones loaded at import; declare the matrix and test command in pyproject.toml:

# pyproject.toml
[tool.cibuildwheel]
build = "cp39-* cp310-* cp311-* cp312-*"
skip = "*-musllinux_*"
environment = { PYTHONMALLOC = "malloc" }
test-command = "python -c \"import my_geospatial_ext; print('ABI & import OK')\""

6. Repair the wheel so RPATH points at the bundled allocator. Geospatial wheels must carry their native shared libraries with correct RPATH/RUNPATH entries; the same repair tooling covered in Managing Shared Library Paths in manylinux applies here:

#!/usr/bin/env bash
# repair.sh — bundle native libs and patch RPATH post-build
set -euo pipefail
WHEEL_DIR="dist/"; OUT_DIR="wheelhouse/"; mkdir -p "$OUT_DIR"
for wheel in "$WHEEL_DIR"/*.whl; do
  echo "Validating: $wheel"
  if [[ "$OSTYPE" == linux* ]]; then
    auditwheel show "$wheel"
    auditwheel repair "$wheel" --plat manylinux_2_28_x86_64 --wheel-dir "$OUT_DIR"
  elif [[ "$OSTYPE" == darwin* ]]; then
    delocate-listdeps "$wheel"
    delocate-wheel -w "$OUT_DIR" "$wheel"
  fi
done

Verification #

Prove each layer in isolation: that the capsule destructor fires, that no native bytes survive teardown, and that the repaired wheel imports cleanly.

Confirm the destructor runs and the handle releases. Drop the last reference and assert the native side is gone:

PYTHONMALLOC=malloc python -X dev -c "
import gc, my_geospatial_ext as m
ds = m.open('sample.tif')
del ds; gc.collect()
print('capsule released, no ResourceWarning above')
"

Track native growth across a load loop with tracemalloc plus RSS. A flat RSS over many open/close cycles is the pass condition:

# leak_probe.py — RSS must not climb monotonically
import os, resource, my_geospatial_ext as m
for i in range(2000):
    ds = m.open("sample.tif"); del ds
    if i % 500 == 0:
        rss = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
        print(f"iter {i}: maxrss={rss} KiB")

Run Valgrind against the suite with the CPython suppression file. definitely lost: 0 bytes after the suppressions is the gate:

PYTHONMALLOC=malloc PYTHONFAULTHANDLER=1 \
valgrind --leak-check=full --suppressions=valgrind-python.supp \
  python -m pytest tests/ -x --tb=short
# Expected tail: "definitely lost: 0 bytes in 0 blocks"

Confirm the repaired wheel resolves its bundled allocator. auditwheel show should report a single manylinux tag and no external libgdal:

auditwheel show wheelhouse/*.whl
ldd $(python -c "import my_geospatial_ext, os; print(os.path.dirname(my_geospatial_ext.__file__))")/*.so
# Expected: libgdal/libproj resolve to paths inside the wheel's .libs directory

Optimization & Edge Cases #

Process-global caches survive object teardown. GDAL’s block cache (GDAL_CACHEMAX) and PROJ’s grid cache are process-scoped, not per-dataset. Releasing every capsule will not return that memory; call GDALDestroyDriverManager() and proj_cleanup() once at interpreter shutdown via an atexit-registered C hook, never inside tp_dealloc.

fork() without exec duplicates native arenas. Under multiprocessing with the default fork start method, child processes inherit the parent’s open GDAL handles and cached grids, and a double-GDALClose across parent and child corrupts the shared arena. Use the spawn start method for raster workers, or open datasets only after the fork.

musl vs glibc allocator behaviour differs. musl’s malloc returns freed pages to the OS far more eagerly than glibc, so a leak that shows as flat RSS on manylinux may show as steady growth on musllinux. If you ship both, run the leak probe under each base image; the same musl/glibc trade-off matters when vendoring PROJ and GDAL because the static C++ runtime carries its own operator-new arena.

Cross-compiled builds cannot run Valgrind natively. When you produce aarch64 wheels on an x86_64 host via the toolchain in Cross-Compiler Toolchain Setup, leak checks must run on a native aarch64 runner or under QEMU user emulation — AddressSanitizer instrumentation does not survive a cross-link cleanly, so gate ASan jobs to native arches only.

Troubleshooting #

Fatal Python error: deallocating None — a tp_dealloc path called Py_DECREF on a borrowed reference, or the cyclic GC entered the destructor twice. Audit for missing pointer nullification after the native destroy call and remove any C-API call from the destructor body.

free(): double free or corruption (out) at interpreter exit — two owners freed the same native pointer, typically a capsule destructor plus a manual GDALClose elsewhere, or a fork-duplicated handle. Enforce single ownership through the capsule and never close a handle you handed to PyCapsule_New.

==12345== definitely lost: 4,096 bytes in 1 blocks under Valgrind — a PyMem_RawMalloc buffer or a proj_create_* object was never paired with its free. Cross-check every allocation site for an exit path (early return on error) that skips the matching PyMem_RawFree/proj_destroy.

OSError: ... cannot allocate memory in static TLS block at import — a vendored library’s thread-local context store exhausted the static TLS budget, common when several geospatial extensions load side by side. Build the offending library with -ftls-model=global-dynamic or preload it via LD_PRELOAD, as covered in Shared Library Path Resolution.

Production Checklist #

• Wrap all native handles in PyCapsule with explicit destructor callbacks.
• Nullify internal pointers immediately after native teardown to prevent double-free.
• Avoid Python C-API calls inside tp_dealloc or __dealloc__.
• Use PyMem_RawMalloc for large, long-lived native buffers.
• Set PYTHONMALLOC=malloc in CI to disable pymalloc and enable accurate leak tracking.
• Validate wheel RPATH/install_name with auditwheel/delocate before publishing.
• Enforce strict teardown order: destroy contexts → close datasets → drop Python refs.
• Compile debug wheels with -fsanitize=address for nightly leak regression tests.

Memory safety in geospatial extensions is not an interpreter concern; it is a build and architecture responsibility. By enforcing deterministic teardown, isolating native heap boundaries, and embedding validation into the wheel pipeline, maintainers ship stable, production-grade distributions that scale across data platforms and cloud environments.

Related #

• Geospatial C-Extension Fundamentals & ABI Architecture — the parent reference covering compile, link, repair, and import for native geospatial extensions.
• C-API vs CPython ABI Compatibility — why PyCapsule and PyMem_RawMalloc are Stable-ABI safe and how versioned symbols interact with destructors.
• Vendoring PROJ and GDAL vs System Libraries — how a statically vendored C++ runtime gives the native heap its own allocator arena.
• Shared Library Path Resolution — RPATH/TLS issues that surface as allocation and import failures at load time.
• Cross-Compiler Toolchain Setup — why AddressSanitizer and Valgrind leak gates must run on native arches.

Further Reading #

• Python C-API Memory Management (docs.python.org/3/c-api/memory.html).
• PROJ context and threading reference (proj.org/development/reference/functions.html).