Per-thread PROJ contexts in parallel transforms

This page answers one question: you want to reproject millions of coordinates across a thread pool, and PROJ contexts are not safe to share — so how do you give each worker its own context and transformation without paying the construction cost on every call? It sits inside the Thread Safety and Concurrency in GDAL Bindings section of the Geospatial C-Extension Fundamentals & ABI Architecture reference, and gives you the Python and C patterns plus the cost model that says when it is worth it.

One context per worker, created once and reused Four worker threads each own a PROJ context and a transformation created from it, stored in thread-local storage and built on the worker's first task. Every subsequent task on that worker reuses the same objects. Nothing crosses between workers. Below, the anti-pattern: a single shared transformer used by all four workers, which corrupts PROJ's internal caches. correct — nothing is shared worker 1 own context + PJ worker 2 own context + PJ worker 3 own context + PJ worker 4 own context + PJ incorrect — one transformer, four callers worker 1 worker 2 worker 3 worker 4 one shared Transformer

Context & Root Cause

PROJ organises its state around a context object. A PJ_CONTEXT holds the search paths, the error state, a cache of coordinate operations and a connection to proj.db; every transformation object is created from a context and carries a pointer back to it. None of that state is protected by a lock, because PROJ’s design assumption is one context per thread — a reasonable choice that gives excellent single-thread performance and puts the burden on the caller.

Sharing a transformer across threads therefore produces two failure modes. The visible one is a crash inside PROJ under load, when two threads mutate the operation cache concurrently. The dangerous one is a wrong answer: the cache returns an operation selected for a different coordinate, the transform succeeds, and the result is plausible but incorrect by metres. Because both failures depend on timing, a test suite that runs one thread will never see either.

Solution / Fix

This targets PROJ 9.2+ and pyproj 3.6+, with CPython 3.9–3.13.

1. Thread-local transformers in Python 

import threading
import pyproj

_local = threading.local()

def _transformer(src, dst):
    key = (src, dst)
    cache = getattr(_local, "cache", None)
    if cache is None:
        cache = _local.cache = {}
    t = cache.get(key)
    if t is None:
        t = cache[key] = pyproj.Transformer.from_crs(src, dst, always_xy=True)
    return t

def reproject(xs, ys, src=4326, dst=3857):
    return _transformer(src, dst).transform(xs, ys)

The dictionary matters as much as the thread-local: a pipeline transforming between several coordinate reference systems otherwise rebuilds a transformer every time the pair changes, which is the cost this pattern exists to remove.

2. Per-thread contexts in C 

static pthread_key_t ctx_key;

static PJ_CONTEXT *thread_ctx(void) {
    PJ_CONTEXT *ctx = pthread_getspecific(ctx_key);
    if (!ctx) {
        ctx = proj_context_create();
        proj_context_set_search_paths(ctx, 1, (const char *[]){bundled_proj_data});
        pthread_setspecific(ctx_key, ctx);
    }
    return ctx;
}

static PJ *thread_pj(const char *src, const char *dst) {
    /* create once per thread from that thread's context */
    return proj_create_crs_to_crs(thread_ctx(), src, dst, NULL);
}

3. Clone when construction is expensive 

/* Build once on the main thread, clone onto each worker's context */
PJ *worker_pj = proj_clone(thread_ctx(), template_pj);

proj_clone copies the transformation onto the target context without re-parsing the CRS definitions or re-querying the database, which is the fast path when many workers need the same transformation.

4. Release the interpreter lock around the transform

Without this the workers exist and never overlap; the discipline is set out in releasing the GIL during coordinate transforms.

Verification 

# 1. Parallel results match serial results exactly
python - <<'PY'
from concurrent.futures import ThreadPoolExecutor
import mypkg
xs = [5.0 + i * 1e-7 for i in range(40000)]
ys = [52.0] * len(xs)
serial = mypkg.reproject(xs, ys)
with ThreadPoolExecutor(8) as ex:
    parts = list(ex.map(lambda i: mypkg.reproject(xs[i::8], ys[i::8]), range(8)))
assert sum(len(p[0]) for p in parts) == len(xs)
print("consistent")
PY

# 2. Each worker built exactly one transformer, not one per call
PYTHONWARNINGS=always python - <<'PY'
import threading, mypkg
built = []
orig = mypkg._transformer
def counting(src, dst):
    t = orig(src, dst); built.append(threading.get_ident()); return t
mypkg._transformer = counting
mypkg.run_pool(workers=4, tasks=200)
print("constructions:", len(set(built)), "distinct threads:", len(set(built)))
PY
# expected: constructions equal to the number of threads, not to the task count

# 3. Under oversubscription, a fixed coordinate stays fixed
python ci/stress_transform.py --workers 32 --rounds 400
# expected: every round asserts the canary coordinate to 1e-3 and passes

The third check is the one that finds sharing bugs. Thirty-two workers on an eight-core machine forces the interleavings that a comfortable four-worker run never produces, and the fixed-coordinate assertion converts a silent wrong answer into a failure.

What a Context Actually Costs

Deciding between one context per thread, one per task and one shared needs the cost model, and the numbers are lopsided enough to make the decision easy.

Relative cost of creating a context, a transformation and a transform call Creating a PROJ context costs on the order of a millisecond because it opens the database. Creating a transformation from a context costs several milliseconds because it queries coordinate operations. Cloning an existing transformation onto another context costs tens of microseconds. Transforming a batch of a million coordinate pairs costs hundreds of milliseconds. The conclusion is that construction must be amortised across many calls, which is what per-thread reuse achieves. approximate cost, log scale clone a transformation tens of microseconds create a context about a millisecond — opens proj.db create a transformation several milliseconds — queries operations transform 1M pairs hundreds of milliseconds construction is worth it once per thread and never once per call — the ratio is three orders of magnitude

Two consequences. First, per-task construction is indefensible for small tasks: a worker that builds a transformer to convert a thousand points spends more time constructing than transforming. Second, per-thread construction is essentially free once the pool is warm — four contexts and four transformations, built once, amortised over the whole run.

That also explains why the batch size matters. A pool processing a million points in one call per worker sees the construction cost disappear entirely; the same pool processing a thousand calls of a thousand points each sees it once per worker as well, provided the transformer is cached rather than rebuilt. Both are fine; what is not fine is rebuilding inside the loop.

Where Thread-Local State Lives and When It Dies

Thread-local objects have a lifetime that surprises people, and for PROJ contexts the surprise has a memory dimension worth planning for.

The lifetime of a thread-local PROJ context in a worker pool A worker thread creates its context on its first task and keeps it for the life of the thread. A pool that reuses threads therefore holds one context per worker for the duration of the program, each with its own operation cache and database connection. A pool that creates threads per batch pays construction repeatedly and releases the memory between batches. Neither is wrong, but the first trades memory for speed and the second the reverse. a long-lived pool — threads are reused one context per worker for the life of the program, each with its own operation cache fastest; costs a few megabytes per worker that never comes back a pool per batch — threads are recreated construction paid per batch; memory released when the pool is torn down appropriate for a long-running service processing occasional batches a context holds a database connection and a cache — for eight workers that is real memory, and it is not a leak if a memory audit flags it, the answer is fewer workers or a shorter-lived pool, not a shared context

The last line matters because a memory audit will flag this. Eight PROJ contexts each holding a database connection and an operation cache look exactly like a leak to a tool that counts allocations at exit — and the fix is emphatically not to share one context between threads. Suppressing the pattern in the leak-hunting configuration, as tracking native allocations with Valgrind and ASan describes, is the correct response.

Pitfalls & Alternatives

Caching the transformer at module scope. A module-level transformer is shared by every thread that imports the module, which is the exact anti-pattern. Thread-local storage is the smallest change that fixes it.

Passing a transformer into a worker as an argument. It looks like clean dependency injection and it hands one object to several threads. Pass the CRS identifiers and let each worker build or look up its own.

Assuming a process pool needs any of this. Each process has its own everything, so a ProcessPoolExecutor sidesteps the whole question — at the cost of serialising arrays between processes. For file-in, file-out work that is usually the better trade.

Forgetting that the network setting is per context. PROJ_NETWORK and the search paths are context properties, so a context created in a worker without them configured behaves differently from one created on the main thread. Configure them in the same helper that creates the context.

Frequently Asked Questions

Does pyproj already handle this internally?

It manages contexts carefully and it cannot make a Transformer safe to share, because the underlying PROJ object is not. Treat every transformer as owned by the thread that created it, regardless of which layer created it.

How many transformers will a thread accumulate?

One per coordinate-reference pair it has used, if you cache as shown. For a pipeline converting between a handful of systems that is a handful of objects per worker; for one converting to a per-feature target system, the cache should be bounded or the memory will not be.

Is proj_clone always faster than creating a transformation?

Substantially, because it skips the database queries and the operation search. The exception is a transformation so simple that construction was already trivial, where the difference is not worth the extra code path.

What about the network setting and search paths?

Both are context properties, so a worker’s context needs them configured exactly as the main thread’s was. Doing that inside the same helper that creates the context is the reliable way to keep them consistent.

Does this pattern apply to async code as well as threads?

Coroutines on one event loop share a thread, so they share a context safely — but a transform that blocks for hundreds of milliseconds blocks the loop, which is its own problem. The usual arrangement is to run the transform in an executor, at which point the per-thread rules apply again.

What is the right batch size per call?

Large enough that the per-call overhead disappears, which in practice means tens of thousands of coordinate pairs rather than hundreds. Beyond a million pairs per call the gains flatten and memory becomes the constraint, so somewhere in that range is a reasonable default.

Should the cache be bounded?

For a fixed set of coordinate systems, no — the number of entries is small and stable. For a pipeline whose target system varies per feature, yes, or the per-thread cache grows without limit over a long run.