Cutting import time in geospatial extension modules

This page answers one question: import geo_core takes a third of a second and your users run a command-line tool in a loop, so what in that time is avoidable and how do you move the rest off the import path? It sits inside the Binary Size and Startup Performance Tuning section of the Modern Python Build Tooling & Wheel Configuration reference, and gives you the profile, the deferral pattern and the parts you cannot remove.

Which parts of a cold import are avoidable Mapping and relocating the bundled shared objects is unavoidable once the extension is imported. C plus plus static initialisers are unavoidable for the libraries actually loaded. Driver registration is avoidable by deferring it to first use. Opening the PROJ database and building a default transformer are both avoidable the same way. Python-level module code is small and largely avoidable by not doing work at module scope. map and relocate the .so files ~140 ms cold — unavoidable once the extension is imported reduced only by shipping fewer and smaller objects C++ static initialisers ~70 ms — unavoidable for the libraries actually loaded reduced by linking fewer libraries, not by deferring driver registration ~80 ms — avoidable: defer to first use the single largest recoverable block open proj.db, build a transformer ~30 ms — avoidable the same way and often not needed at all in a given run

Context & Root Cause

Importing a vendored geospatial package does four things: it maps a dozen shared objects and processes their relocations, it runs the static initialisers of the C++ libraries among them, it registers GDAL’s drivers, and it does whatever your own module body does. The first two are the price of loading the code; the last two are work the package chose to do at import rather than on demand.

That distinction is the whole optimisation. A command-line tool invoked once per file pays the full import on every invocation, and a test suite that runs each module in a fresh interpreter pays it per file. For a long-running service none of it matters. Knowing which of your users you have decides how much of this is worth doing — and the deferral changes are cheap enough that they are usually worth doing anyway.

Solution / Fix

This targets CPython 3.9+, GDAL 3.8.x, pyproj 3.6+ and a wheel with bundled libraries.

1. Profile before changing anything 

python -X importtime -c "import geo_core" 2>&1 | sort -t'|' -k2 -n | tail -10

The last lines are the modules with the largest cumulative time, and for a spatial package the top entry is almost always the one that triggers the native load.

2. Move the expensive work behind a call 

# geo_core/__init__.py
from . import _ext              # required: this is the native load itself

_ready = False

def _ensure_ready():
    global _ready
    if _ready:
        return
    from osgeo import gdal
    gdal.UseExceptions()
    gdal.AllRegister()          # ~80 ms, once, on first actual use
    _ready = True

def open_raster(path):
    _ensure_ready()
    from osgeo import gdal
    return gdal.Open(path)

3. Do not build objects at module scope 

# before: pays for a transformer nobody may use
DEFAULT = pyproj.Transformer.from_crs(4326, 3857, always_xy=True)

# after: built on demand, cached per thread
def default_transformer():
    return _transformer(4326, 3857)     # thread-local cache

4. Keep the imports themselves lazy where they are heavy 

def to_geodataframe(...):
    import geopandas               # heavy, and only some callers need it
    ...

Verification 

# 1. Cold import time, five runs, median
python - <<'PY'
import statistics, subprocess, sys
def once():
    out = subprocess.run([sys.executable, "-X", "importtime", "-c", "import geo_core"],
                         capture_output=True, text=True).stderr.strip().splitlines()[-1]
    return int(out.split("|")[1].strip())
runs = [once() for _ in range(5)]
print(f"median {statistics.median(runs)/1000:.0f} ms   spread {(max(runs)-min(runs))/1000:.0f} ms")
PY

# 2. Nothing expensive runs at import — the registry is still empty
python -c "
import geo_core
from osgeo import gdal
print('drivers at import:', gdal.GetDriverCount())"
# expected: 0 (or a small built-in count), not the full set

# 3. The deferred work still happens exactly once
python -c "
import geo_core, time
t0=time.perf_counter(); geo_core.open_raster('scene.tif'); t1=time.perf_counter()
geo_core.open_raster('scene.tif'); t2=time.perf_counter()
print(f'first {1000*(t1-t0):.0f} ms, second {1000*(t2-t1):.0f} ms')"
# expected: the first call pays the registration, the second does not

The second check is the one that proves the deferral rather than assuming it. A driver count of zero immediately after import means the registration genuinely moved; a full count means something in the import chain still calls it.

What You Cannot Defer

Two of the four blocks are not recoverable, and knowing that prevents a lot of wasted effort.

Why mapping and static initialisers cannot be deferred Importing the extension module causes the loader to map every library it declares as needed and to run their static initialisers before the import returns. Deferring the Python-level import of the extension only moves that cost to a later point; it does not remove it, and it makes failures appear at an arbitrary time instead of at import. The only way to reduce these blocks is to ship fewer and smaller objects. import _ext one Python statement the loader maps everything every NEEDED library, transitively static initialisers run before the import returns deferring the Python import moves this cost later; it does not remove it and it makes a load failure appear at an arbitrary call site rather than at import the only reductions are fewer libraries, smaller libraries, and --as-needed

That last line ties the import budget back to the size work. --as-needed removes NEEDED entries for libraries whose symbols are never referenced, and each removed entry is one fewer file to open, map and relocate. Hidden visibility shrinks the dynamic symbol tables the loader has to process. Both were introduced as size measures in shrinking GDAL wheels with LTO and strip and both pay a second time here.

Deferring the extension import itself is the tempting move and the wrong one. It converts a predictable cost at a predictable moment into the same cost at an unpredictable one, and it means an environment problem — a missing library, a wrong architecture — surfaces in the middle of someone’s pipeline rather than at the top of their script.

Designing an API That Starts Fast

The deferral pattern works better when the package’s structure supports it, and three structural choices make the difference.

Three structural choices that keep import cheap Keeping the top-level module free of work means it only imports the extension and defines functions. Putting optional integrations behind function-level imports means a user who never calls them never pays for the library they need. And exposing a single initialisation entry point means the expensive setup happens once, at a moment the caller controls, rather than being scattered across several functions that each check a flag. a top-level module that does nothing imports the extension, defines functions, sets a version string — no objects built, no registries touched optional integrations behind function-level imports geopandas, matplotlib, cloud SDKs — a user who never calls that function never pays for the import one initialisation entry point a single idempotent function rather than a flag checked in six places — easier to reason about and to test together these usually halve a cold import without changing what the package can do

The middle row has a secondary benefit worth mentioning: it keeps optional dependencies genuinely optional. A module-level import geopandas makes that package a hard requirement at import time even if it is declared as an extra, so a user who installed without the extra gets an ImportError from your package’s import rather than from the function they did not call.

The third row is about maintainability more than speed. Six functions each doing if not _ready: _setup() is six places to get wrong; one idempotent _ensure_ready() called from each is one place, and it can be tested directly.

Pitfalls & Alternatives

Deferring the extension import. It moves the cost rather than removing it and relocates failures to arbitrary points. Import the extension at module scope; defer the work built on top of it.

Measuring once on a busy machine. Import timing varies by tens of milliseconds run to run. Five runs and a median is two lines and turns noise into a number.

Registering drivers lazily but calling something that registers them anyway. Several GDAL entry points register implicitly. The driver-count assertion after import is what catches that.

Optimising import time for a service. A long-running process pays it once. Confirm who your users are before spending effort here; the size work benefits everyone, this benefits a subset.

Frequently Asked Questions

How much can I realistically remove?

Deferring registration and database access typically removes a third to a half of a cold import. Beyond that you are into the mapping and initialiser blocks, which respond to shipping fewer and smaller libraries rather than to restructuring Python code.

Does -X importtime show the native cost?

It attributes it to the module whose import triggered the load, so the number is there but not broken down. To see inside, compare an import of the extension alone against an import of the full package, and use LD_DEBUG=statistics if you need the loader’s own accounting.

Is a frozen or bundled interpreter faster?

For the Python-level part, somewhat. For a spatial package the dominant cost is native loading, which freezing does not change. It is worth pursuing only after the deferral work, and usually not then.

Does lazy importing break type checking?

Function-level imports are invisible to some tooling, which is why the conventional arrangement is to import them under a type-checking guard for annotations while importing them lazily at run time. That keeps both properties.

What about __getattr__ at module level for lazy submodules?

It works and is a reasonable way to defer a heavy submodule while keeping the attribute access natural. Keep it simple: a dictionary of name to module path, imported on first access, is enough and stays debuggable.

Should the package expose the initialisation function publicly?

Yes, as a documented way for a long-running service to pay the cost at startup rather than on the first request. It is one line in the public API and it removes a latency spike from someone’s production traffic.

Does this apply to the packages I depend on rather than my own?

Their import cost is yours too, since importing them happens inside your import. Where a dependency is heavy and only some of your callers need it, moving that import into the function is the same technique applied one level out, and it is frequently the largest single saving available.

How do I stop a deferral from regressing?

The driver-count assertion is the mechanism: it fails the moment something in the import chain registers again. Without a check, a well-meaning change that adds a convenience import at module scope silently restores the cost.

Is there a downside to the lazy pattern for library consumers?

One: the first call becomes slower than the rest, which can surprise someone benchmarking. Exposing the initialisation function publicly lets them pay it deliberately at startup, which turns the surprise into a documented choice.