Releasing the GIL during coordinate transforms

This page answers one question: how do you release the Global Interpreter Lock around a long-running PROJ coordinate transform so a multi-second reprojection does not freeze every other thread in the interpreter — without corrupting memory or touching a PyObject while the lock is dropped? It sits inside the Memory Management in Geospatial Extensions section of the Geospatial C-Extension Fundamentals & ABI Architecture reference, and gives you the exact Py_BEGIN_ALLOW_THREADS placement, the buffer-copy discipline it demands, and a benchmark that proves the parallel speedup.

The GIL-release window around a native coordinate transform A timeline shows a C extension function that first copies coordinate arrays out of Python objects while holding the GIL, then executes Py_BEGIN_ALLOW_THREADS to drop the lock, runs proj_trans_array on the raw C buffers with no Python access, executes Py_END_ALLOW_THREADS to reacquire the lock, and finally builds the result PyObject. A second thread runs concurrently only during the released middle window. C extension thread GIL held copy arrays → C buffer GIL released proj_trans_array() — no PyObject access Py_BEGIN_ALLOW_THREADS … Py_END_ALLOW_THREADS GIL held build result PyObject other Python thread runs concurrently — only while the lock is released blocked before and after this window

Context & Root Cause

CPython’s Global Interpreter Lock serializes execution of Python bytecode: only one thread runs interpreter code at a time. That is fine until a C-extension enters a long native call — a PROJ pipeline transforming a million coordinate pairs takes hundreds of milliseconds to seconds, and for that whole span the thread holds the GIL while doing zero Python work. Every other thread, including asyncio’s event loop and any parallel workers, is frozen behind it. The reprojection is embarrassingly parallel at the C level, yet the process behaves as if single-threaded.

The fix is to bracket the pure-native work with Py_BEGIN_ALLOW_THREADS / Py_END_ALLOW_THREADS, which drop and reacquire the lock. The constraint that makes this a memory-management problem, not just a performance tweak, is absolute: while the GIL is released you may not touch any PyObject — no reference counting, no PyList_GetItem, no buffer object access — because another thread may be mutating interpreter state concurrently. You must therefore copy the coordinate data out of Python objects into a raw C buffer before releasing, and build the result after reacquiring. Get the ordering wrong and you get a race that manifests as a Segmentation fault under load, exactly the class of crash catalogued in Memory Management in Geospatial Extensions.

Solution / Fix

This targets CPython 3.9–3.12 and PROJ 9.2+ (proj_trans_array from the proj.h C API). The pattern holds whether you write raw C or Cython.

1. Copy into a native buffer under the lock 

// Still holding the GIL: validate and extract into a C-owned buffer.
Py_buffer view;
if (PyObject_GetBuffer(coords_obj, &view, PyBUF_C_CONTIGUOUS | PyBUF_FORMAT) < 0)
    return NULL;                       // borrows the Python buffer

size_t n = view.len / (2 * sizeof(double));
double *xy = PyMem_RawMalloc(view.len); // RawMalloc: safe without the GIL
if (!xy) { PyBuffer_Release(&view); return PyErr_NoMemory(); }
memcpy(xy, view.buf, view.len);
PyBuffer_Release(&view);               // done touching Python memory

PyMem_RawMalloc is deliberate: the standard PyMem_Malloc requires the GIL, so a buffer you intend to use after releasing the lock must come from the raw allocator, as Memory Management in Geospatial Extensions details.

2. Release, transform, reacquire 

PJ *P = /* thread-safe PJ* cloned per call via proj_clone on a shared context */;
int ok;

Py_BEGIN_ALLOW_THREADS               // ← GIL dropped here
    ok = proj_trans_array(P, PJ_FWD, n, (PJ_COORD *)xy);
Py_END_ALLOW_THREADS                 // ← GIL reacquired here

// Only now is it legal to touch Python objects again.
if (ok != 0) {
    PyMem_RawFree(xy);
    PyErr_SetString(PyExc_RuntimeError, "proj_trans_array failed");
    return NULL;
}

The PJ object must be safe to use off-thread. PROJ contexts (PJ_CONTEXT) are not thread-safe to share, so clone the transform onto a per-thread context with proj_clone before entering the released window, or create one context per worker thread.

3. Build the result after reacquiring 

PyObject *out = PyByteArray_FromStringAndSize((char *)xy, n * 2 * sizeof(double));
PyMem_RawFree(xy);
return out;                            // ownership handed to the interpreter

Verification 

# 1. Prove the parallel speedup with a threaded benchmark
python - <<'PY'
import threading, time
from _geospatial_ext import transform_bulk   # the extension above
data = [b"\x00" * (2*8*1_000_000)]           # 1M coord pairs
def work(): transform_bulk(data[0])
t0 = time.perf_counter()
ts = [threading.Thread(target=work) for _ in range(4)]
[t.start() for t in ts]; [t.join() for t in ts]
print(f"4 threads: {time.perf_counter()-t0:.2f}s")
PY
# expected: close to 1x the single-call time, not 4x (threads overlap)

# 2. Prove no memory corruption under contention (ASan build)
PYTHONMALLOC=malloc python -X dev -c "import _geospatial_ext; print('ok')"
# expected: ok, with no ASan heap-use-after-free or data-race report

If the four-thread time is ~4× a single call, the GIL was never released — confirm the Py_BEGIN_ALLOW_THREADS macro actually wraps the native call and is not short-circuited by an early return.

Pitfalls & Alternatives

Touching a PyObject inside the released window. Even a stray Py_DECREF or PyList_SET_ITEM between the macros is a data race. Keep the block purely native; if you need Python data, copy it out first. This is the number-one cause of intermittent Segmentation fault (core dumped) in threaded geospatial code.

Sharing one PJ_CONTEXT across threads. PROJ contexts carry mutable error state and a grid cache; concurrent use corrupts both. Clone per thread with proj_clone, or gate context creation per worker.

Releasing the GIL for trivial work. The macros have a real cost (lock handoff). For a transform of a few points the overhead dominates; only release around work measured in milliseconds or more. For the broader allocator and teardown rules these buffers rely on, see fixing memory leaks in GDAL Python bindings.

The Window Where Python Does Not Exist

The discipline this pattern demands is easier to hold if you picture the released window as a period during which the interpreter, from your code’s point of view, has been switched off. Nothing that touches a PyObject is legal — not a reference count, not an exception set, not a buffer view, not a logging call that happens to go through Python. Everything the transform needs must have been copied into plain C memory before the window opens, and everything it produces must stay in plain C memory until the window closes.

What is legal before, during and after the released-GIL window A timeline in three phases. Before releasing, holding the lock, it is legal to parse arguments, acquire a buffer, allocate with PyMem_RawMalloc and copy coordinates out. During the released window only pure C operations are legal: the PROJ transform on the raw buffer, and raw allocation or free. After reacquiring, it is legal again to build the result object, set exceptions and free the raw buffer. Below, three operations are marked as never legal during the window: touching any PyObject, calling PyMem_Malloc, and raising an exception. holding the GIL PyArg_ParseTuple PyObject_GetBuffer PyMem_RawMalloc memcpy into the C buffer PyBuffer_Release GIL released proj_trans_array(P, …) pure C loops over doubles PyMem_RawMalloc / RawFree other Python threads run concurrently — that is the point GIL reacquired build the result object PyErr_SetString on failure PyMem_RawFree the buffer return to the caller never inside the released window any Py_INCREF / Py_DECREF — the count is not atomic against other threads PyMem_Malloc or PyObject_Malloc — both require the lock you just dropped PyErr_SetString — thread state is not yours to write; record a flag and raise after

The error-handling rule in the last line is the one that catches people out, because failure is exactly when the instinct to raise immediately is strongest. The correct shape is to record the failure in a plain C variable inside the window, close the window, and only then translate it into a Python exception. PROJ makes this straightforward: proj_trans_array returns a non-zero count of failed coordinates, and proj_errno can be read afterwards for the specific code.

Thread safety of the transform object itself is the other half of the contract. A PJ_CONTEXT must not be shared across threads, and a PJ created on one context must not be used from another. The workable patterns are one context per worker thread, created once and reused, or proj_clone onto a thread-local context at the start of each call. Sharing a single transformer across a thread pool produces intermittent wrong results and occasional crashes — a failure that reproduces only under concurrency and therefore only in production.

What Four Threads Actually Do

The payoff is easiest to see as a timeline of four worker threads transforming equal slices of the same array. With the lock held throughout, the native work serialises even though it touches no interpreter state; with it released around the transform, the four native segments overlap and only the short copy-in and build-out phases contend.

Four threads transforming coordinates, with and without releasing the lock Two timelines. In the first, four threads hold the interpreter lock for the whole call, so their native transform segments run strictly one after another and total wall-clock time is four times one transform. In the second, each thread copies data in while holding the lock, releases it for the transform so all four native segments overlap, and reacquires briefly to build its result; total wall-clock time is close to one transform plus a small serialised remainder. lock held throughout — native work serialises t1 t2 t3 t4 wall clock ≈ 4 × one transform lock released around the transform — native work overlaps t1 t2 t3 t4 grey = holding the lock (copy in, build result) teal = released window (pure C transform) wall clock ≈ one transform + a short serialised remainder

The grey segments are the reason the speedup is never exactly linear: copying a million coordinate pairs into a raw buffer is real work, and it still serialises. Making those segments as small as possible — by accepting a contiguous buffer that can be copied with a single memcpy rather than iterated element by element — is what turns a two-times speedup into something closer to the core count.

When Releasing the Lock Does Not Pay

Releasing the GIL is not free: the pair of macros costs a lock handoff and, for a short call, that cost dominates. There is a rough threshold below which the pattern makes the code slower and more fragile for no benefit, and it is worth knowing roughly where it sits.

Measured on ordinary hardware, the release-and-reacquire pair costs on the order of a microsecond when uncontended, and considerably more when several threads are competing to reacquire. A PROJ transform of a single coordinate pair takes a few microseconds; a batch of a thousand takes tens of microseconds; a batch of a million takes hundreds of milliseconds. The pattern therefore pays overwhelmingly for bulk array transforms, marginally for batches in the low thousands, and not at all for per-point calls.

That has a design implication beyond the C code: the binding’s API shape determines whether the optimisation is available at all. An interface that transforms one point per call can never benefit, because the per-call overhead is the entire cost. An interface that accepts an array — a NumPy array, a buffer, a memoryview — can copy once, release once, and transform a million points in a single window. If you are designing the extension rather than maintaining one, the array-shaped API is the decision that makes everything else possible.

/* Below a threshold the released window costs more than it saves. */
if (n < 4096) {
    ok = proj_trans_array(P, PJ_FWD, n, (PJ_COORD *)xy);   /* keep the GIL */
} else {
    Py_BEGIN_ALLOW_THREADS
        ok = proj_trans_array(P, PJ_FWD, n, (PJ_COORD *)xy);
    Py_END_ALLOW_THREADS
}

Finally, note what the pattern does not buy you. Releasing the GIL lets other Python threads run; it does not parallelise the transform itself. Using several cores for one large reprojection means splitting the array across threads, each with its own context and its own released window — at which point the copy discipline matters even more, since several windows are open at once over disjoint slices of the same buffer.

Frequently Asked Questions

Does this still matter on free-threaded CPython?

The mechanics change and the discipline does not. On a build without the GIL there is no lock to release, so the macros become no-ops, but every rule about not touching interpreter state from a thread that has not attached remains — and new hazards appear, because objects genuinely can be mutated concurrently. Extensions that already separate a pure-C computation window from their Python-facing code are the ones that port most easily.

Can I use Cython instead of writing this by hand?

Yes, and for most projects you should. Cython’s with nogil: block generates the same macro pair, and its typed memoryviews give you a checked way to access buffer data without holding references to Python objects. The rules are identical — no Python objects inside the block — but the compiler enforces a good share of them, which removes the most common source of mistakes.

How do I know the lock was really released?

Measure it rather than assume it. Run the same workload in one thread and in four; if wall-clock time is flat, the lock is still held somewhere in the hot path. A common cause is a logging or callback hook inside the loop that re-enters Python, which quietly reacquires and serialises everything again.

What about GDAL — can I release the GIL around its calls too?

For most raster I/O, yes, and the bindings already do it in several places. The caveats are that GDAL’s error handler is process-global and may call back into Python if one has been installed, and that a Dataset must not be used concurrently from multiple threads unless it was opened with the thread-safe access mode. Sharing datasets across a pool is the single most common cause of hard-to-reproduce crashes in threaded GDAL code.