Diagnosing segfaults from shared GDAL datasets

This page answers one question: your raster pipeline crashes under load with a segmentation fault whose stack trace names a function that has never been changed, so how do you establish whether a shared GDALDataset handle is the cause and fix it without rewriting the pipeline? It sits inside the Thread Safety and Concurrency in GDAL Bindings section of the Geospatial C-Extension Fundamentals & ABI Architecture reference, and gives you the reproduction, the confirmation and the two fixes.

Why two threads reading one dataset corrupt its block cache Two worker threads call read on the same dataset handle. Both reach the driver's block cache, which maintains a list of loaded blocks and an eviction order. One thread evicts a block while the other is reading from it, so the second thread dereferences memory that has been reused. The crash appears in whichever function next touches the freed block, which is rarely the function that caused the problem. thread A ReadAsArray(window 1) thread B ReadAsArray(window 2) one dataset handle · one block cache block list and eviction order, unprotected B evicts a block A is reading A dereferences reused memory the crash lands wherever the freed block is next touched — often in a completely different band which is why the stack trace is not the cause

Context & Root Cause

A GDALDataset is a stateful object. It owns a block cache, a set of open band objects, driver-specific state and, for many formats, a file handle with a position. None of that is protected by a lock in the general case, so two threads calling into the same handle concurrently can interleave inside data structures that assume a single caller.

The characteristic symptom is a crash whose stack trace implicates innocent code. One thread evicts a cached block while another is reading from it; the reading thread then dereferences memory that has been reused for something else, and the fault surfaces in whichever function next touches it. Because the outcome depends on timing, the crash is intermittent, load-dependent, and impossible to reproduce with a single worker — which is exactly the profile that makes teams suspect their own recent changes rather than the concurrency model.

Solution / Fix

This targets GDAL 3.6–3.9 with the osgeo bindings, CPython 3.9–3.13, and a thread-pool pipeline.

1. Establish that concurrency is the variable 

# Same workload, one worker versus many. If one worker never crashes, it is a race.
WORKERS=1  python pipeline.py --tiles 500   # run several times
WORKERS=16 python pipeline.py --tiles 500   # run several times

2. Find the shared handle 

# Any dataset opened outside the worker function is a candidate
grep -n "gdal.Open\|gdal.OpenEx" -r src/ | grep -v "def worker"

3. Fix it by opening per worker 

def worker(job):
    ds = gdal.Open(job.path)             # this thread's own handle
    try:
        return job.index, ds.ReadAsArray(*job.window)
    finally:
        ds.Close()

Opening is cheap relative to reading — a header parse rather than a data read — so the per-task cost is small and the shared state disappears entirely.

4. Or serialise access if opening is genuinely expensive 

import threading
_lock = threading.Lock()

def worker(job):
    with _lock:                          # correctness at the cost of parallelism
        return job.index, _shared_ds.ReadAsArray(*job.window)

The lock is a legitimate stopgap and a poor destination: it serialises exactly the work you parallelised. Use it to confirm the diagnosis, then move to per-worker handles.

Verification 

# 1. Under oversubscription, a long run completes without a crash
python pipeline.py --tiles 2000 --workers 32
echo "exit: $?"        # expected: 0, repeatedly

# 2. No dataset handle is reachable from more than one thread
python - <<'PY'
import threading, mypkg
seen = {}
orig = mypkg.open_dataset
def tracked(path):
    ds = orig(path)
    seen.setdefault(id(ds), set()).add(threading.get_ident())
    return ds
mypkg.open_dataset = tracked
mypkg.run_pool(workers=8, tasks=200)
shared = {k: v for k, v in seen.items() if len(v) > 1}
assert not shared, f"{len(shared)} dataset(s) used from multiple threads"
print("no shared handles")
PY

# 3. Output is byte-identical to a single-threaded run
python pipeline.py --tiles 200 --workers 1  --out /tmp/serial.tif
python pipeline.py --tiles 200 --workers 16 --out /tmp/parallel.tif
cmp /tmp/serial.tif /tmp/parallel.tif && echo "identical"

The second check is the one worth keeping. It instruments the open call rather than the crash, so it reports the defect deterministically instead of waiting for an interleaving that triggers it — which turns a probabilistic bug hunt into an assertion.

Telling This Apart From Its Neighbours

Three distinct problems produce crashes in threaded spatial code, and they need different fixes. Matching the symptom saves rewriting the wrong layer.

Three concurrency faults in spatial code and how to tell them apart A shared dataset handle crashes inside driver code during reads and is fixed by opening per worker. A shared PROJ context crashes inside transformation code or returns wrong coordinates and is fixed with thread-local transformers. A released interpreter lock with Python objects still being touched crashes inside CPython itself and is fixed by restoring the copy discipline. Each has a distinctive location in the stack trace. crash location cause fix inside a GDAL driver, during a read or a cache operation one dataset handle used from two threads open per worker or serialise with a lock inside proj_trans, or no crash at all and wrong coordinates one PROJ context or PJ shared between threads thread-local transformers or proj_clone per worker inside CPython itself — refcount or GC functions a PyObject touched while the interpreter lock was released restore the copy discipline around the released window

The middle row is the one that does not always crash, and that property makes it the most damaging of the three. A pipeline producing coordinates that are wrong by metres in a small fraction of cases can run for months before anyone notices, and the noticing usually happens downstream of your code. The fixed-coordinate canary described in per-thread PROJ contexts in parallel transforms is the cheapest defence.

The bottom row is the one to suspect when the trace names CPython functions rather than GDAL ones. That is not a GDAL problem at all; it is the extension touching interpreter state during a window when it was not allowed to, and the fix is in the extension’s own code.

Restructuring So the Handle Cannot Be Shared

Fixing the immediate crash is a small change. Making it impossible to reintroduce is a structural one, and it is worth doing because this defect returns whenever someone adds a “small optimisation” to avoid reopening a file.

A job description that cannot carry a dataset handle On the left, the fragile shape: the job object holds an open dataset, so scheduling it onto a worker shares the handle. On the right, the durable shape: the job holds only a path, a window and an index, and the worker opens its own handle from that description. The right-hand shape also moves unchanged from threads to processes, because nothing in the job is a native handle. fragile Job(dataset=ds, window=w) the handle travels with the job every worker gets the same one and moving to processes fails, because a handle cannot be pickled durable Job(path=p, window=w, index=i) a plain description, nothing native the worker opens its own handle and the same code runs unchanged in a process pool making the job description picklable is the cheapest way to guarantee no native state is shared — the constraint enforces the design, and it is checkable in a unit test

The pickling constraint is a neat enforcement mechanism precisely because it is mechanical. A test that asserts every job description round-trips through pickle will fail the moment someone puts a dataset, a band or a transformer into it, and the failure message points straight at the offending field.

Pitfalls & Alternatives

Concluding it is a GDAL bug. It usually is not; sharing a handle is outside the contract. Establish the one-worker-versus-many result before reporting anything upstream, because that single experiment is what distinguishes a usage error from a genuine defect.

Adding a lock and considering it fixed. The lock is correct and it serialises the work, so the pipeline is now slower than the single-threaded version it replaced. Use it to confirm the diagnosis and then restructure.

Relying on the thread-safe dataset mode. Newer GDAL releases offer one, and its availability depends on the build and the driver. Code that must work against arbitrary builds should not assume it; code that controls its own build can use it deliberately and should test that the mode is actually active.

Sharing the handle “read-only, so it is fine”. The block cache is mutated by reads. Read-only refers to the file, not to the object, and the object is where the shared state lives.

Frequently Asked Questions

How expensive is opening a dataset per worker really?

For local files it is a header parse — typically single-digit milliseconds — against a read that is orders of magnitude larger. For remote sources over a network protocol it is more significant, and the answer there is fewer, larger tasks per worker rather than a shared handle.

Can I keep one handle per worker instead of one per task?

Yes, and it is the better shape for many pipelines: open in a thread-local on the worker’s first task and reuse it. That keeps the handle single-threaded while amortising the open, exactly as the per-thread transformer pattern does for PROJ.

Does the crash prove my code is wrong rather than GDAL’s?

The one-worker-versus-many experiment answers that. If the workload is stable single-threaded and crashes under load, the concurrency model is the variable, and sharing a handle is outside what the library promises. Take that result to an upstream report if you still believe it is a library defect.

Will a process pool avoid this entirely?

Yes, because each process has its own handles, caches and registries. The trade is serialising arrays between processes, which for file-in, file-out pipelines is usually cheap and for array-passing pipelines can dominate.

Does the crash always land inside GDAL?

Not always, and that is what makes the diagnosis feel unreliable. A corrupted cache entry can be dereferenced by any later caller, so the trace sometimes names the interpreter or an unrelated library. The one-worker experiment is more reliable than the stack, which is why it comes first.

What about a dataset opened read-only from several processes?

That is fine — each process has its own handle, its own cache and its own file descriptor, and the file is not being modified. The rule about single-threaded use is about the in-memory object, not about the file on disk.

Can a crash like this corrupt the output file?

It can, and that is the reason to treat a crashed run’s output as suspect rather than resumable. A thread interrupted mid-write leaves a partially updated block, and the file may be structurally valid while containing wrong pixels. Rerun from the beginning after fixing the sharing, and compare against a single-threaded run before trusting the result. Treat any output produced by a run that crashed as evidence about the bug rather than as a deliverable, and regenerate it once the sharing is fixed and the comparison against a single-threaded run passes.