Tracking native allocations with Valgrind and ASan

This page answers one question: your process grows and Python’s own profilers show nothing, so how do you find which native allocation in GDAL, PROJ or your own extension is never released — using Valgrind, AddressSanitizer and a suppression file that makes CPython’s ordinary behaviour stop drowning the signal? It sits inside the Memory Management in Geospatial Extensions section of the Geospatial C-Extension Fundamentals & ABI Architecture reference, and gives you the invocation, the suppressions and the way to read the output.

Which tool sees which kind of native memory problem Three tools compared by what they detect. Resident set size sampling detects unbounded growth over many iterations but names nothing. Valgrind's memcheck detects definitely lost and still reachable allocations and names the exact C call site, at roughly twenty times slower execution. AddressSanitizer detects leaks plus out-of-bounds and use-after-free errors at roughly two times slower, but requires rebuilding the extension and ideally its dependencies with the sanitizer. tool detects names the site? cost RSS sampling no build change unbounded growth over iterations plateau versus straight line no free valgrind memcheck no build change definitely lost, indirectly lost, still reachable at exit yes, full C stack ~20× slower AddressSanitizer needs a rebuild leaks plus out-of-bounds and use-after-free yes, if built with it ~2× slower

Context & Root Cause

Native leaks in geospatial code are invisible to Python tooling because they are invisible to Python: the allocation was made by CPLMalloc, by GEOS, or by a driver, and the interpreter never knew about it. Resident set size shows that something grows; it cannot say what, and a leak of a few kilobytes per call takes hundreds of thousands of iterations to become visible that way at all.

Valgrind and AddressSanitizer both instrument the allocator itself, so they can name the exact call site of every block that was never freed. The obstacle is noise. CPython deliberately does not free a great deal at shutdown — interned strings, type objects, the small-object arena — and GDAL, PROJ and GEOS all keep caches alive for the life of the process. Run either tool naively on an interpreter and you get thousands of “still reachable” blocks, none of which are bugs. Making the output usable is mostly a matter of configuring the tool to ignore what is normal, which is what this page is about.

Solution / Fix

This targets Valgrind 3.19+, GCC 13 or Clang 16 with AddressSanitizer, and CPython 3.9–3.13.

1. Build CPython’s allocator out of the way

Python’s pooled allocator hides individual allocations from both tools. Setting one environment variable routes everything through malloc:

export PYTHONMALLOC=malloc      # required — otherwise pools mask the call sites
export PYTHONDEVMODE=1          # extra checks, and disables some caching

2. Run Valgrind with the right filters 

valgrind \
  --leak-check=full \
  --show-leak-kinds=definite,indirect \
  --errors-for-leak-kinds=definite \
  --num-callers=25 \
  --suppressions=ci/valgrind-python.supp \
  --log-file=valgrind.log \
  python ci/leak_workload.py

--show-leak-kinds=definite,indirect is the setting that makes the report readable: it excludes “still reachable”, which is where every cache in the process lands and where essentially no bugs are.

3. Write a suppression file for the known-normal noise 

# ci/valgrind-python.supp
{
   cpython-interned-strings
   Memcheck:Leak
   match-leak-kinds: reachable
   ...
   fun:PyUnicode_InternInPlace
}
{
   gdal-driver-registry
   Memcheck:Leak
   match-leak-kinds: reachable
   ...
   fun:GDALAllRegister
}
{
   proj-context-cache
   Memcheck:Leak
   match-leak-kinds: reachable
   ...
   fun:proj_context_create
}

4. Or build with AddressSanitizer for a faster loop 

export CFLAGS="-fsanitize=address -fno-omit-frame-pointer -g -O1"
export LDFLAGS="-fsanitize=address"
pip install --no-binary :all: -e .
ASAN_OPTIONS=detect_leaks=1:fast_unwind_on_malloc=0 \
LSAN_OPTIONS=suppressions=ci/lsan.supp \
  python ci/leak_workload.py

The workload script matters as much as the tool: it should do the suspect operation a few hundred times and then exit cleanly, so that anything still held at exit is either a cache or a leak.

Verification 

# 1. No definite losses attributable to your own code
grep -A6 'definitely lost' valgrind.log | grep -c 'geo_core'
# expected: 0

# 2. The total definitely-lost figure is stable across workload sizes
for n in 50 500; do
  ITERATIONS=$n valgrind --leak-check=summary --log-file=/tmp/v$n.log \
    python ci/leak_workload.py >/dev/null 2>&1
  grep 'definitely lost' /tmp/v$n.log
done
# expected: the same byte count for both — a leak would scale with n

# 3. Under ASan, a clean exit with no leak report
ASAN_OPTIONS=detect_leaks=1 python ci/leak_workload.py && echo "no leaks detected"

The second check is the decisive one and the one people skip. A fixed number of bytes lost regardless of iteration count is a one-time allocation — a cache, a registry, a singleton — and not a leak. Bytes that scale with the iteration count are a leak, and the multiplier tells you how much is lost per call.

Reading a Leak Report Without Drowning

Valgrind’s four leak categories mean different things, and knowing which to act on removes most of the volume from the report.

The four leak categories and which ones are bugs Definitely lost means no pointer to the block remains and it is almost always a real bug. Indirectly lost means the block was reachable only through a definitely lost block, so fixing the parent fixes it. Possibly lost means an interior pointer remains and is worth investigating but often benign. Still reachable means a pointer remains at exit, which describes every cache and registry in the process and is almost never a bug. definitely lost no pointer to the block remains anywhere act on every one of these — this is the category that means "bug" indirectly lost reachable only through a definitely lost block fixing the parent fixes these — do not chase them separately possibly lost only an interior pointer remains worth a look; frequently benign in C++ containers still reachable a pointer remains at exit every cache and registry lands here — suppress it and move on

A practical reading order: filter to definitely lost, sort by total bytes, and start at the top. The largest single entry is usually one allocation site called many times, and fixing it removes most of the report. Entries measured in tens of bytes are rarely worth the effort unless they scale — a 40-byte leak per raster block adds up in a pipeline processing millions of blocks, and the iteration-scaling test is what distinguishes it from a one-time cost.

The stack trace is the other half of the value. With --num-callers=25 the trace reaches from the allocation site up through the binding into the Python call, which usually identifies not just what leaked but which of your API functions failed to release it.

Making It Affordable in CI

Neither tool is fast enough for every commit, and both are cheap enough for a schedule. The arrangement that works is a small, deliberately-designed workload run nightly.

Where each memory check belongs in the pipeline Three tiers by frequency. On every commit, a fast resident-set-size assertion over a few hundred iterations runs in seconds and catches gross regressions. Nightly, an AddressSanitizer build runs the same workload and catches leaks plus memory errors in about twice the normal time. Weekly, a full Valgrind run over a longer workload catches small per-call leaks that the other two miss. Each tier is cheap at its own frequency. every commit RSS assertion over ~500 iterations — seconds, no build change catches gross regressions: a dataset that stopped being closed nightly AddressSanitizer build running the same workload — about 2× slower catches leaks with call sites, plus out-of-bounds and use-after-free weekly full Valgrind run over a longer workload — about 20× slower catches small per-call leaks that never move the resident set visibly the suppression file is shared by all three tiers and is reviewed like code — an over-broad entry hides real bugs

The suppression file deserves the review discipline the diagram mentions. It is tempting, when a report is noisy, to add a broad entry matching anything in libgdal; that silences the noise and every future bug in the same region. Keep entries narrow — matched on a specific function — and add a comment saying why each one is legitimate, so the next person can tell a genuine cache from a suppressed defect.

Pitfalls & Alternatives

Forgetting PYTHONMALLOC=malloc. Without it, Python’s pooled allocator satisfies most small requests from arenas the tools see as single large blocks, so individual call sites disappear and the report becomes useless.

Treating “still reachable” as a leak. Every driver registry, coordinate-operation cache and interned string lands there. Suppressing the category wholesale is correct for this domain, and doing so is what makes the remaining output short enough to act on.

Running against a release build. Without -g there are no symbols, and the stack traces name addresses. Build the extension with debug information for the leak-hunting run, even if the shipped wheel is stripped.

Concluding from one workload size. A fixed number of lost bytes is a one-time allocation; only bytes that scale with the iteration count are a leak. Running two sizes takes twice as long and answers the question definitively.

Frequently Asked Questions

Can I run these tools against a released wheel?

Valgrind yes, AddressSanitizer no. Valgrind instruments at run time and needs no rebuild, so it works on any installed package — though without debug information the stack traces name addresses. AddressSanitizer requires the extension to be compiled with it, so it applies to your own builds only.

Why does the report change between runs?

Allocation order and cache state vary, so the set of still-reachable blocks moves even when nothing leaks. That is another reason to filter to definite losses, which are stable, and to compare totals across workload sizes rather than reading one run in isolation.

Do these tools work on macOS and Windows?

AddressSanitizer works on both with the appropriate compiler; Valgrind’s support on recent macOS is limited and on Windows absent. For those platforms the practical substitutes are the sanitizer and the resident-set-size approach, which is portable.

Is a leak in a cache still a leak?

Not usually. A cache that grows to a bound and stops is doing its job; one that grows without limit is a leak whatever it is called. The iteration-scaling test distinguishes them without any argument about terminology.

How long should the leak workload run?

Long enough for a per-call leak to be distinguishable from a fixed cost, which usually means a few hundred iterations rather than a few. Running two sizes and comparing totals is more informative than running one size for longer, because the comparison is what identifies scaling.

Can these tools find leaks in GDAL itself?

They can, and the result usually is not actionable by you. Before reporting one upstream, confirm it scales with iterations and that your own teardown is correct — most apparent library leaks turn out to be a handle the caller never closed.

Does running under a sanitizer change program behaviour?

It changes timing and memory layout, which occasionally hides a race or exposes one that ordinarily does not occur. For leak hunting that is unimportant; for concurrency work it is worth remembering that a workload passing under a sanitizer has not been proved correct at full speed, only at instrumented speed. The practical consequence is to use the sanitizer for leaks and correctness, and to confirm timing-sensitive behaviour with an uninstrumented run at full speed afterwards.