Thread Safety and Concurrency in GDAL Bindings

Geospatial workloads are embarrassingly parallel and geospatial libraries are not: GDAL, PROJ and GEOS each maintain process-global or per-context state that decides whether a thread pool speeds your pipeline up or corrupts it. This guide sits under the Geospatial C-Extension Fundamentals & ABI Architecture reference and sets out which objects may be shared between threads, which must be created per thread, and how to test the difference before a production pipeline finds it for you. It targets GDAL 3.6–3.9, PROJ 9.2+, GEOS 3.11+, CPython 3.9–3.13, and extensions built to the memory discipline described in memory management in geospatial extensions.

Which spatial objects may be shared across threads and which may not Three tiers. Process-global state — the GDAL driver registry, the error handler and the configuration options — is shared by every thread and must only be mutated during startup. Per-context state — a PROJ context and the transformations created from it — must belong to exactly one thread. Per-object state — a GDAL dataset handle and a GEOS geometry — is safe to use from one thread at a time and is not safe to share without external locking. process-global — one per interpreter GDAL driver registry · error handler · CPLSetConfigOption · PROJ search paths safe to read from any thread; mutate only during startup, before workers exist per-context — one per thread, never shared PJ_CONTEXT and every PJ created from it · GDAL thread-local error state sharing one across threads corrupts internal caches and crashes without warning per-object — one thread at a time GDALDataset handles · GEOS geometries · OGR layers and features safe sequentially from different threads; unsafe concurrently without a lock

Prerequisites & Environment

  • GDAL 3.6+ built with thread support, and knowledge of whether your build enables the thread-safe dataset mode introduced in later releases.
  • PROJ 9.2+ with the reentrant API (proj_context_create, proj_create_crs_to_crs bound to a context).
  • A workload large enough to be worth parallelising: coordinate transforms of hundreds of thousands of points, or tile-level raster operations, rather than per-feature calls.
  • A way to reproduce load: a thread pool with more workers than cores, run for long enough that races have a chance to occur.
# What was this GDAL built with? Thread behaviour depends on the answer
python -c "from osgeo import gdal; print(gdal.VersionInfo('RELEASE_NAME'))"
python -c "import pyproj; print(pyproj.proj_version_str)"

Core Configuration

The single most useful rule is that PROJ contexts are per-thread and everything created from a context inherits that restriction. A PJ object obtained from a context belongs to the thread that created it; using it from another thread is undefined behaviour even if the calls never overlap in time.

import threading
import pyproj

_local = threading.local()

def transformer():
    """One transformer per thread, created lazily and reused."""
    t = getattr(_local, "t", None)
    if t is None:
        t = pyproj.Transformer.from_crs(4326, 3857, always_xy=True)
        _local.t = t
    return t

def reproject(points):
    return transformer().transform(*zip(*points))

The equivalent in C is proj_context_create() per worker thread, with every proj_create_crs_to_crs call passing that context. Cloning is available for the case where a transformation is expensive to construct and you want one per thread without re-parsing the CRS definitions.

Object Shared across threads? How to get one per thread
PJ_CONTEXT never proj_context_create() per thread
PJ transformation never create from that thread’s context, or proj_clone
Transformer (pyproj) never threading.local() as above
GDALDataset one thread at a time open per thread, or serialise with a lock
GEOS geometry one thread at a time build per thread; do not pass between workers
driver registry yes, read-only register once at import

Step-by-Step Implementation

  1. Do all global configuration before starting workers. Registering drivers, installing an error handler and setting configuration options are process-wide mutations; doing them from a worker races with every other worker:

    from osgeo import gdal
gdal.UseExceptions()
gdal.SetCacheMax(256 * 1024 * 1024)
gdal.AllRegister()          # once, at import — never inside a task

  2. Give each worker its own PROJ context through thread-local storage, as above, and never pass a transformer between threads.

  3. Open datasets per worker rather than sharing a handle. Opening is cheap relative to reading, and it removes the need for a lock entirely:

    def worker(path, window):
    ds = gdal.Open(path)          # this thread's handle
    try:
        return ds.ReadAsArray(*window)
    finally:
        ds.Close()

  4. Release the interpreter lock around the native work in your own extension, following the buffer discipline in releasing the GIL during coordinate transforms — otherwise the threads exist but never overlap.

Where Concurrency Actually Pays

Parallelising a spatial pipeline has a shape: some stages scale nearly linearly, some are bounded by shared state, and some are bounded by the disk. Knowing which is which prevents a great deal of wasted effort.

How four spatial workloads scale with worker count Four workloads compared. Bulk coordinate transformation scales nearly linearly with worker count because each worker owns its own context and buffer. Tile-level raster processing scales well until disk bandwidth saturates. Vector feature iteration over one layer scales poorly because the layer handle serialises access. Writing to a single output dataset does not scale at all and must be funnelled through one writer. speedup workers 1 4 8 bulk transforms own context per worker tiled raster reads until disk saturates one shared layer handle serialises access single-writer output — flat the top curve is the only one worth threading; the bottom two are worth restructuring instead and every curve assumes the interpreter lock is released around the native call

Two structural conclusions follow. First, the unit of parallelism should be a chunk of data, not a feature or a point: a worker that transforms a million coordinates in one call amortises everything, while one that transforms a point per call spends all its time on overhead. Second, output is usually the serialisation point. Writing tiles from several workers into one dataset requires either a lock around every write or, better, a single writer thread consuming finished tiles from a queue.

Processes remain a legitimate alternative and often the better one. A worker process has its own driver registry, its own PROJ contexts and its own address space, so every rule above becomes trivially satisfied; the cost is serialising arrays between processes and a higher memory floor. For pipelines that read and write files rather than passing large arrays around, process-level parallelism is usually both faster to implement and harder to get wrong.

Diagnosing a Concurrency Bug

Races in native spatial code produce a characteristic set of symptoms, and matching the symptom to the cause narrows the search quickly.

Concurrency symptoms mapped to their usual cause in spatial code Four symptoms with causes. A crash that only appears under load usually means a PROJ context or transformation shared between threads. Wrong coordinates in a small fraction of results usually means a transformer reused across threads without cloning. A hang usually means the interpreter lock was reacquired inside a released window, or a lock held across a native call. Corrupted output files usually mean several writers on one dataset. symptom usual cause first thing to try crashes only under load never reproducible with one worker a PROJ context or PJ shared between two threads thread-local transformers then re-run at 4× workers a few results are wrong the rest are correct a transformer reused across threads without cloning assert a fixed coordinate in every worker throughput does not improve CPU stays near one core the interpreter lock is never released around the native call time one worker vs four before changing anything output file is corrupt several writers on one dataset funnel writes through one thread

The most valuable diagnostic habit is to make the failure deterministic before trying to fix it. Running the workload with four times as many workers as cores, on a machine under other load, converts a race that appears weekly into one that appears in seconds. Adding a fixed-coordinate assertion inside each worker turns “some results look wrong” into a failing test.

Sanitisers help for extensions you control. A build with ThreadSanitizer reports the exact pair of accesses that race, which is far more actionable than a stack trace from a crash. It is too slow for routine use and entirely appropriate for a scheduled job that runs the concurrency tests once a night.

Verification 

# 1. The transform result is identical single-threaded and multi-threaded
python - <<'PY'
from concurrent.futures import ThreadPoolExecutor
import mypkg
pts = [(5.0 + i * 1e-6, 52.0) for i in range(20000)]
serial = mypkg.reproject(pts)
with ThreadPoolExecutor(8) as ex:
    chunks = [pts[i::8] for i in range(8)]
    parallel = [r for c in ex.map(mypkg.reproject, chunks) for r in zip(*c)]
assert len(parallel) == len(pts)
print("consistent")
PY

# 2. Threading actually overlaps — wall clock must not scale with workers
python - <<'PY'
import time, threading, mypkg
def run(n):
    t0 = time.perf_counter()
    ts = [threading.Thread(target=mypkg.bulk_transform) for _ in range(n)]
    [t.start() for t in ts]; [t.join() for t in ts]
    return time.perf_counter() - t0
one, four = run(1), run(4)
print(f"1 thread {one:.2f}s   4 threads {four:.2f}s   ratio {four/one:.2f}")
PY
# expected: ratio well under 4 — near 1 means the lock is released correctly

# 3. No dataset handle is shared — each worker opens its own
grep -n "gdal.Open" mypkg/*.py | grep -v "def worker" || echo "opens are worker-local"

Optimization & Edge Cases

  • Create transformers once per thread, not per call. Constructing a Transformer parses CRS definitions and consults proj.db; doing it per call can cost more than the transform itself.
  • The block cache is shared and bounded. Several threads reading different windows of the same raster compete for one cache. Raising GDAL_CACHEMAX helps up to a point; beyond it, restructuring so each worker reads a disjoint region helps more.
  • Free-threaded CPython changes the calculus, not the rules. Without the interpreter lock, more Python-level code runs in parallel — and every rule about per-thread contexts and per-object exclusivity still applies, because those are properties of the C libraries.
  • fork after threads is unsafe. A process that has started threads and then forks inherits locks in an undefined state. Use spawn for worker processes in any program that also uses threads.
  • Error handlers are global. An error handler installed by one thread receives errors raised by all of them; if it stores state, that state must be thread-safe or the handler becomes the race.

Troubleshooting

Segmentation fault in proj_trans under load. A context or transformation crossed a thread boundary. Move to thread-local transformers and re-run at high worker counts to confirm.

ERROR 1: … not recognised as a supported file format only in workers. The driver registry was mutated after workers started, or a worker ran before registration completed. Register at import.

Results differ by a few centimetres between runs. Two threads sharing a transformer can interleave inside PROJ’s internal caches. The numbers are plausible, which is what makes this the most dangerous form of the bug.

A pool of eight workers is slower than one. Either the lock is never released — so the workers serialise and pay scheduling overhead — or every task is too small. Measure the single-worker time for one task before assuming the former.

Frequently Asked Questions

Is GDAL thread-safe?

Parts of it, with conditions, and the honest answer for binding authors is to treat dataset handles as single-threaded objects. Recent GDAL versions offer a thread-safe dataset mode, but its availability depends on how the library was built and which driver is involved — so code that must work against arbitrary builds should open per thread rather than rely on it.

Should I use threads or processes for spatial work?

Processes when the work is file-in, file-out, because every shared-state rule disappears and the serialisation cost is low. Threads when large arrays must move between stages and copying them between processes would dominate. Many production pipelines use both: processes for isolation, threads inside each for overlapping I/O with computation.

Does releasing the interpreter lock make my code thread-safe?

No — it makes concurrency possible, which means bugs that were previously masked by serialisation can now occur. Releasing the lock and sharing a PROJ context is worse than not releasing it at all, because the code now genuinely runs in parallel over shared state.

How many workers should a spatial pipeline use?

Start at the core count and measure. Raster work is frequently bounded by memory bandwidth or disk rather than by CPU, so more workers than cores usually makes things slower while increasing peak memory — which for a pipeline holding raster blocks is the constraint that bites first.

Designing a Pipeline That Is Safe by Construction

The most reliable way to get concurrency right in spatial code is to arrange the pipeline so that the dangerous sharing is impossible rather than merely avoided. Three structural patterns do most of the work, and each corresponds to a class of workload.

Partition by data, own everything else. The strongest pattern is a worker that receives a description of work — a file path and a window, a chunk of coordinates, a tile index — and owns every native object it touches for the duration. It opens its own dataset, creates its own transformer, allocates its own buffers, and returns plain arrays or plain bytes. Nothing crosses a thread boundary except immutable descriptions on the way in and owned data on the way out. This pattern scales linearly until an external resource saturates, and it survives being moved from threads to processes without a code change, which is the property that makes it worth preferring even when threads would do.

Funnel writes through one owner. Output is where the partition breaks down, because a single output dataset cannot be written concurrently. The shape that works is a queue of finished tiles consumed by exactly one writer thread. It costs a little latency at the end of the run and removes an entire category of corruption; the alternative — a lock around every write — serialises the same work while looking parallel, and tends to be held across long native calls, which negates the point of releasing the interpreter lock at all.

Make the expensive setup thread-local and lazy. Transformers, database connections and driver-specific handles are expensive to build and unsafe to share. Building them lazily in thread-local storage gives each worker one instance created on its first task and reused for the rest, which is both correct and considerably faster than per-call construction. The subtlety is that thread-local objects outlive the task, so a pool that recycles threads keeps them alive — which is usually what you want, and is worth knowing when memory is being accounted for.

# The three patterns together, in outline
import queue, threading
from concurrent.futures import ThreadPoolExecutor

_local = threading.local()

def _tools():
    t = getattr(_local, "tools", None)
    if t is None:
        t = _local.tools = build_transformer()      # per-thread, lazy, reused
    return t

def process(job):                                    # job is a plain description
    ds = open_dataset(job.path)                      # this worker owns it
    try:
        return job.index, transform_tile(ds, job.window, _tools())
    finally:
        ds.Close()

def run(jobs, out_path):
    results = queue.Queue(maxsize=32)
    writer = threading.Thread(target=single_writer, args=(out_path, results))
    writer.start()                                   # exactly one writer
    with ThreadPoolExecutor(max_workers=8) as ex:
        for item in ex.map(process, jobs):
            results.put(item)
    results.put(None)
    writer.join()

Two habits reinforce the structure. Keep the job description free of native handles — a path and a window, never an open dataset — so that moving to processes later is a scheduler change rather than a rewrite. And make the worker function importable and callable on its own, so a failing tile can be reproduced in a single-threaded interpreter without reconstructing the pool.

Testing Concurrency Before Production Does

Concurrency bugs in native spatial code are probabilistic, which means an ordinary test suite will not find them and a production pipeline eventually will. A small amount of deliberate stress testing changes those odds substantially.

The first ingredient is oversubscription. Running with two to four times as many workers as the machine has cores forces context switches at arbitrary points, which is exactly the condition under which a shared context corrupts. A test that passes at four workers on an idle eight-core machine proves very little; the same test at thirty-two workers on a loaded machine is a real signal.

The second is repetition with variety. Races depend on interleaving, so a single run is one sample. Running the same workload a few hundred times, with the work items shuffled between runs, explores far more of the space than a longer single run does. This is cheap enough to schedule nightly even for pipelines that take minutes.

The third is an invariant that must hold. “It did not crash” is a weak assertion; “every worker produced the same answer for the same fixed input” is a strong one. Including a known coordinate in every worker’s batch and asserting the transformed result to a fixed tolerance converts silent corruption — the most dangerous failure mode, because the numbers stay plausible — into a test failure.

# A stress test that has actually caught shared-context bugs
import itertools, random
from concurrent.futures import ThreadPoolExecutor

FIXED = (5.0, 52.0)
EXPECTED = (556597.4539663672, 6800125.454397307)

def batch(seed):
    rng = random.Random(seed)
    pts = [(rng.uniform(-180, 180), rng.uniform(-85, 85)) for _ in range(500)]
    pts.insert(rng.randrange(len(pts)), FIXED)       # the canary, at a random index
    out = reproject(pts)
    got = out[pts.index(FIXED)]
    assert abs(got[0] - EXPECTED[0]) < 1e-3 and abs(got[1] - EXPECTED[1]) < 1e-3, got
    return len(out)

def test_stress():
    with ThreadPoolExecutor(max_workers=32) as ex:   # deliberate oversubscription
        assert all(n == 501 for n in ex.map(batch, range(400)))

Finally, treat a sanitiser run as a scheduled job rather than a per-commit gate. ThreadSanitizer on an extension you control reports the exact pair of racing accesses, which turns a week of bisecting into a single stack trace. It is far too slow for interactive use and entirely reasonable to run once a night against the same stress test — and unlike the probabilistic tests, it finds races that did not happen to occur.

Further Reading

  • The PROJ documentation on thread contexts, and GDAL’s notes on multi-threading and the block cache.