Zero-copy GDAL ReadAsArray with the buffer protocol

This page answers one question: reading a large raster block copies it into a fresh NumPy array and doubles peak memory, so how do you hand the native buffer to Python without copying — and attach the ownership that stops it becoming a use-after-free? It sits inside the NumPy ABI and Array Interop in Spatial Extensions section of the Geospatial C-Extension Fundamentals & ABI Architecture reference, and gives you the two mechanisms, the lifetime rule and the test that catches the mistake.

Peak memory with a copy versus a view for one raster block With a copy, the native block and the new array both exist while the copy is made, so peak memory is twice the block size and stays at one block after the dataset closes. With a view, only the native block exists and the array points into it, so peak memory is one block — but the block cannot be released until the array is, which means the dataset stays alive longer than the caller may expect. copy — peak is two blocks native block held by the dataset the new NumPy array, allocated and filled both exist during the memcpy; after close, one remains and it is independent view — peak is one block native block, with the array pointing into it no second allocation — but the block cannot be freed while the array lives so the dataset stays alive too, whether or not the caller realises it

Context & Root Cause

ReadAsArray copies, deliberately: the array it returns is owned by NumPy and remains valid after the dataset is closed, which is the right default for a general-purpose API. The cost is a second allocation the size of the region read, and for tile-scale processing over large rasters that doubling is the difference between a pipeline that fits in memory and one that does not.

A zero-copy path is straightforward to build and has exactly one hard rule: the array must hold a reference to whatever keeps the memory alive. Without it, closing the dataset frees the buffer while an array still points at it, and the next read of that array returns whatever now occupies the memory — or segfaults. The mechanism for expressing that ownership is the array’s base object, and forgetting it is the single most common defect in hand-written spatial bindings, as NumPy ABI and array interop sets out.

Solution / Fix

This targets NumPy 1.23+, GDAL 3.8+, CPython 3.9–3.13, and an extension built to the abi3 contract.

1. Wrap the native buffer and attach an owner 

static void release_block(PyObject *capsule) {
    GDALRasterBandH band = PyCapsule_GetPointer(capsule, "gdal.block");
    GDALRasterBandUnlockBlock(band);      /* or whatever releases your buffer */
}

static PyObject *band_view(PyObject *self, PyObject *args) {
    /* ... obtain band, block, rows, cols ... */
    npy_intp dims[2] = {rows, cols};
    PyObject *arr = PyArray_SimpleNewFromData(2, dims, NPY_FLOAT32, block_data);
    if (!arr) return NULL;

    PyObject *owner = PyCapsule_New(band, "gdal.block", release_block);
    if (!owner) { Py_DECREF(arr); return NULL; }

    if (PyArray_SetBaseObject((PyArrayObject *)arr, owner) < 0) {  /* steals owner */
        Py_DECREF(arr);
        return NULL;
    }
    PyArray_CLEARFLAGS((PyArrayObject *)arr, NPY_ARRAY_WRITEABLE);
    return arr;
}

PyArray_SetBaseObject steals the reference to the capsule on success and, on failure, does not — which is why the error path decrements the array and not the capsule.

2. Or expose the buffer protocol and let NumPy wrap it

For an extension that would rather not link NumPy at all, implementing tp_as_buffer on a small holder type gives the same result through a Stable-ABI mechanism:

static int block_getbuffer(PyObject *exporter, Py_buffer *view, int flags) {
    BlockHolder *self = (BlockHolder *)exporter;
    return PyBuffer_FillInfo(view, exporter, self->data, self->nbytes,
                             1 /* readonly */, flags);
}

import numpy as np
arr = np.asarray(memoryview(holder))     # no copy; holder keeps the block alive

The exporter’s reference count is incremented by PyBuffer_FillInfo, so the holder — and through it the dataset — stays alive as long as any view exists.

3. Document which functions return which 

def read_block(path, band, window):
    """Return an independent copy. Safe after the dataset is closed."""

def view_block(path, band, window):
    """Return a read-only view over native memory.

    The returned array keeps the dataset alive until it is released.
    """

Verification 

# 1. The view has an owner — the check that distinguishes correct from lucky
python - <<'PY'
import mypkg
a = mypkg.view_block("scene.tif", 1, (0, 0, 512, 512))
assert a.base is not None, "no base object — this array will dangle"
assert not a.flags.writeable, "a view over driver memory should be read-only"
print("ownership ok")
PY

# 2. It survives the dataset going out of scope
python - <<'PY'
import gc, mypkg
def get():
    return mypkg.view_block("scene.tif", 1, (0, 0, 512, 512))
a = get(); gc.collect()
print(float(a[0, 0]))          # must not crash
PY

# 3. Peak memory is one block, not two
/usr/bin/time -v python -c "
import mypkg
a = mypkg.view_block('scene.tif', 1, (0, 0, 4096, 4096))
print(a.shape)" 2>&1 | grep 'Maximum resident'
# expected: roughly the block size, not double it

The first check belongs in the test suite permanently. It is two lines, it runs in milliseconds, and it catches the one mistake that turns a performance optimisation into an intermittent crash in someone else’s pipeline.

When a View Is Worth It

Zero-copy is not free of cost — it costs a constraint on the caller — so it is worth being clear about when the trade pays.

Four situations and whether a view or a copy is the better return Four cases. Reading a small window for metadata or a statistic favours a copy because the allocation is trivial and independence is worth more. Streaming tiles through a pipeline favours a view because peak memory is the binding constraint. Handing data to another thread favours a copy because the lifetime becomes hard to reason about. Returning data across a public API boundary favours a copy because callers cannot be relied upon to honour a lifetime rule they did not read. a small window for a statistic copy — the allocation is trivial and independence is worth more a 256 by 256 float block is a quarter of a megabyte streaming tiles through a pipeline view — peak memory is the binding constraint and the array's lifetime is short and local handing data to another thread copy — the lifetime becomes hard to reason about and the dataset is not safe to touch from two threads anyway a public API return value copy — callers will not read the lifetime note offer the view as a separate, explicitly named function

The last row is the design guidance that matters most. A library whose ordinary read function returns a view has made every caller responsible for a rule they did not ask for, and the failures land as crashes in their code rather than errors in yours. Offering read_block (copy) and view_block (view) as separate functions puts the choice where it belongs and makes the constraint visible at the call site.

The middle rows are worth reading together with the concurrency guidance in thread safety and concurrency in GDAL bindings: a view pins a dataset, and a dataset is a single-threaded object, so a view crossing a thread boundary combines two constraints into one hard-to-debug failure.

What Holds What

The ownership chain in a zero-copy read is longer than it looks, and drawing it once makes the failure modes obvious.

The reference chain that keeps a zero-copy array valid The NumPy array holds a reference to a capsule through its base attribute. The capsule holds the band or block handle and a destructor. The band handle keeps the dataset open. The dataset owns the native buffer the array points into. Breaking any link in that chain frees the buffer while the array still references it, which is the use-after-free this design exists to prevent. ndarray .base → capsule + destructor band handle keeps the dataset open dataset owns the native buffer the array points directly at the buffer; the chain above is what keeps the buffer alive omit the base object and the chain is broken at the first link — the array points at freed memory release happens in the capsule destructor when the last array referencing it goes away

Two consequences follow from the chain. First, a view can keep a whole dataset — and its block cache — alive long after the caller believes it was closed, which is exactly the memory behaviour described in fixing memory leaks in GDAL Python bindings. Second, slicing a view produces another array whose base points at the first, so the chain extends rather than breaks — which is correct, and means a small slice retained from a large read pins the entire dataset.

Pitfalls & Alternatives

Returning the array without a base object. It works in every test that uses the array immediately and fails in any code that keeps it. The two-line assertion in the verification section is the whole defence.

Leaving the view writeable. Writing into a driver’s block cache through an array bypasses every consistency mechanism the driver has. Clear the writeable flag unless you have specifically designed for write-through.

Assuming the block is contiguous. A band with a line stride, or an interleaved multi-band read, is not C-contiguous. Either request a contiguous window from the driver or construct the array with explicit strides; constructing it as contiguous when it is not produces silently wrong data.

Using a view to avoid a copy that was not the problem. Profile first. If peak memory is dominated by the block cache rather than by your arrays, raising or lowering GDAL_CACHEMAX is a simpler intervention with no lifetime consequences at all.

Frequently Asked Questions

Does a view work with masked arrays?

Indirectly: build the mask separately and construct the masked array from the view and the mask. The mask itself is a second array, so a fully zero-copy masked read means two views with two owners — workable, and a good reason to consider whether the copy was really the bottleneck.

Can the caller write through the view?

Only if you leave the array writeable, and doing so writes directly into the driver’s cache, bypassing the consistency mechanisms it relies on. Clear the writeable flag unless you have designed a write-through path deliberately and documented it.

How does this interact with with blocks that close the dataset?

The view keeps the dataset alive through the base object, so the Close() inside the block does not free the memory the array points at — but it may put the dataset into a closed state the driver does not expect to read from. Prefer returning copies from context-managed helpers and reserving views for code that manages the lifetime explicitly.

Is the buffer protocol slower than the NumPy C-API?

No; both hand over a pointer. The difference is in what you get alongside it — dtype machinery and casting from NumPy, shape and strides from the buffer protocol — not in the cost of the hand-off itself.

What happens when the array is sliced?

The slice becomes another array whose base points at the first one, so the chain extends and the underlying buffer stays alive. That is correct behaviour and worth knowing: a small slice kept from a large read pins the whole dataset, which can look like a leak in a long-running process.

Should the function name signal that it returns a view?

Yes. Callers cannot tell from the returned object, and the lifetime rule is theirs to honour, so the name is the only place the constraint is visible at the call site. A separate, explicitly named function beats a keyword argument on a general-purpose reader.