Splitting GDAL data into a companion wheel

This page answers one question: proj.db, the datum grids and GDAL’s own data files are architecture-independent yet ship inside every platform wheel, so how do you move them into one companion package without breaking the runtime lookup that finds them? It sits inside the Build Artifact Structuring and Packaging section of the Modern Python Build Tooling & Wheel Configuration reference, and gives you the package layout, the discovery shim and the versioning rule that keeps the two in step.

The same data shipped per platform versus once in a companion package On the left, four platform wheels each carry an identical eleven-megabyte data payload, so a user downloading one wheel pays for it and a mirror storing all four pays four times. On the right, the four platform wheels carry only code and a single architecture-independent data wheel carries the payload once, downloaded once per environment and stored once per release. data in every platform wheel linuxcode+ 11 MB aarch64code+ 11 MB macoscode+ 11 MB windowscode+ 11 MB 44 MB of identical bytes per release data in a companion wheel linuxcode only aarch64code only macoscode only windowscode only geo-data (py3-none-any) 11 MB, once one download per environment; one stored copy per release the trade is a dependency edge and a runtime lookup that has to find the other package

Context & Root Cause

PROJ’s coordinate database and GDAL’s own data files are pure data: identical bytes on every platform, changing only when the upstream library’s data does. Shipping them inside each platform wheel therefore duplicates them across the matrix — four or five copies per release, downloaded per user, stored per mirror — for no benefit other than that the runtime lookup is trivially satisfied.

The obstacle is that lookup. PROJ finds its database through a compiled-in path, an environment variable, or a directory set through its API, and none of those knows about a second Python package by default. Splitting the data out therefore means taking responsibility for pointing the library at it, which is a few lines of code and one design decision — and doing it wrong produces a DataDirError at import for every user, which is why the split is worth doing deliberately rather than opportunistically.

Solution / Fix

This targets PROJ 9.3.x, GDAL 3.8.x, pyproj 3.6+ and a project already bundling data per bundling proj.db and datum grids in a wheel.

1. Create the data package 

geo-data/
  pyproject.toml
  src/geo_data/__init__.py
  src/geo_data/proj/proj.db
  src/geo_data/proj/proj.ini
  src/geo_data/gdal/            # GDAL's own data files

[project]
name = "geo-data"
version = "9.3.1.1"            # tracks the PROJ version it carries
description = "Coordinate reference data for geo-core"

[tool.setuptools.package-data]
geo_data = ["proj/*", "gdal/*"]

2. Expose the directories rather than the files 

# src/geo_data/__init__.py
from pathlib import Path

PROJ_DATA = Path(__file__).parent / "proj"
GDAL_DATA = Path(__file__).parent / "gdal"

def proj_data_dir() -> str:
    return str(PROJ_DATA)

3. Point the libraries at it from the code package 

# geo_core/__init__.py — runs before any transform
import os
import pyproj
import geo_data

pyproj.datadir.set_data_dir(geo_data.proj_data_dir())
os.environ.setdefault("PROJ_DATA", geo_data.proj_data_dir())
os.environ.setdefault("GDAL_DATA", str(geo_data.GDAL_DATA))

from . import _ext        # only now is it safe to load the extension

4. Declare the dependency with a compatible range 

# geo-core's pyproject.toml
dependencies = ["geo-data>=9.3,<9.4"]

The range mirrors the PROJ minor version the code was built against, which is the compatibility boundary that actually matters — the database schema is versioned against the library that reads it.

Verification 

# 1. The platform wheels no longer contain the data
unzip -l dist/geo_core*.whl | grep -c 'proj\.db'
# expected: 0

# 2. The data package is architecture-independent
ls dist/geo_data*.whl
# expected: geo_data-9.3.1.1-py3-none-any.whl

# 3. A clean install finds the database and transforms correctly
docker run --rm -v "$PWD/dist:/d" python:3.12-slim bash -c '
  pip install -q --no-index --find-links /d geo-core &&
  PROJ_NETWORK=OFF python -c "
import pyproj, geo_core
print(pyproj.datadir.get_data_dir())
print(pyproj.Transformer.from_crs(4326, 3857, always_xy=True).transform(5.0, 52.0))"'
# expected: a path inside site-packages/geo_data, and a coordinate pair

The third check is the one that matters, and the PROJ_NETWORK=OFF is not incidental: with network transforms enabled, a missing database can be papered over by a CDN fetch, and the test would pass while the split was broken for offline users.

Where the Lookup Can Go Wrong

The discovery shim has to run before anything touches PROJ, and there are more ways for that ordering to break than there appear to be.

Four ways the data lookup fails after a split The shim runs too late, after another module has already imported pyproj and initialised a context. The environment variable is set but the library was already initialised. A second PROJ-using package in the environment has its own idea of the data directory. Or the data package is present but a different version, so the database schema does not match the library. Each has a distinct symptom and a distinct fix. the shim runs too late another import initialised a PROJ context before your package did symptom: works alone, fails in a larger program — set the directory at package import the variable is set too late PROJ read PROJ_DATA when its first context was created symptom: os.environ shows the right value and the library ignores it — use the API too another package disagrees a second PROJ-using wheel sets its own data directory globally symptom: order-dependent behaviour — prefer the API over the environment variable version mismatch the data package carries a database from a different PROJ minor symptom: some EPSG codes resolve and others do not — tighten the dependency range

The second row explains why the shim sets both the API and the environment variable. pyproj.datadir.set_data_dir covers everything reached through pyproj; the environment variable covers code that talks to PROJ’s C API directly, including GDAL when it reprojects. Setting only one leaves a path uncovered, and which path fails depends on what the user’s program does.

The third row is the argument for preferring the API where possible: the environment variable is process-global, so setting it redirects every PROJ consumer in the interpreter, including a package with its own bundled database that now reads yours. setdefault rather than assignment is the compromise — it fills the gap when nothing else has spoken, and does not overrule a package that has.

Versioning the Pair

A split package is two artifacts that have to agree, and the version scheme is what makes the resolver enforce the agreement rather than leaving it to chance.

A versioning scheme that ties the data package to the PROJ minor version The data package version begins with the PROJ minor version it carries, followed by its own revision. The code package depends on a range covering that minor version only. A PROJ patch bump republishes the data package with a new revision within the same range. A PROJ minor bump moves both packages to a new range together, which the resolver enforces because the old code package cannot accept the new data. the data package geo-data 9.3.1.1 PROJ minor, PROJ patch, then its own revision the version says which database it carries, which is the only thing about it that varies the code package geo-data>=9.3,<9.4 accepts any revision of the matching PROJ minor a data-only fix ships without touching the platform wheels at all a PROJ minor bump both move to 9.4 released together; the old range refuses the new data which is the resolver enforcing the pairing rather than a note in the README

The middle row is the return on the whole exercise beyond size. Updating datum data — a new grid, a corrected transformation — becomes a release of one architecture-independent wheel rather than a rebuild of the entire platform matrix. For a project that ships data updates more often than code changes, that alone can justify the split.

Pitfalls & Alternatives

Making the data an optional extra. It is tempting, because most users will not think about it, and it produces an install that imports and then fails on the first transform. The data is a hard dependency of a package that cannot function without it.

Splitting the grids but not the database. The database is the mandatory part and the grids are the large optional part, so the useful split is often the reverse of the intuitive one: proj.db in the required data package, optional grid sets as extras or as a second package for users who need offline high accuracy.

Assuming a namespace package is needed. Two ordinary distributions with different top-level names are simpler and avoid a class of install-order problems. Reach for namespace packaging only if you genuinely need the two to share an import path.

Forgetting the data package in a coordinated release. It compiles nothing, so it is easy to leave out of a release checklist — and it is versioned against PROJ, so it belongs in the same batch as the platform wheels whenever PROJ moves.

Frequently Asked Questions

Does this make the first install larger?

No — it makes it the same size, split across two files, and every subsequent platform in the same environment free. A user installing on one machine downloads the same total; a team building four container images downloads the data once per image rather than once per wheel.

What if a user already has PROJ data installed system-wide?

Your shim points the library at yours, which is what you want: a wheel that silently used the system database would produce results depending on their installation rather than on what you shipped. Users who genuinely want the system data can set PROJ_DATA themselves, and setdefault respects that.

Can the data package serve more than one code package?

Yes, and that is one of its advantages: several packages in a project can depend on the same data wheel, and it is installed once. The constraint is that all of them must accept the same version range, which is the coordination the versioning scheme is designed to express.

How large should the data package be?

proj.db alone is around nine megabytes and covers every coordinate reference system. Adding the full grid set takes it past five hundred, which is beyond a typical index’s file limit and defeats the purpose. Regional grid sets as separate optional packages are the workable middle ground.

Does this help with the wheel size limit?

It removes the data from the platform wheels, which for a project that was close to the cap can be enough on its own. For a project well over it, stripping and driver pruning remove more, as handling wheel size limits on PyPI for GDAL sets out.

What happens if the data package is missing at run time?

The import fails with a ModuleNotFoundError naming it, which is a clear message pointing at a dependency that was declared — a much better failure than the DataDirError that comes from data being absent without anything having declared it.

Should the data package depend on the code package?

No — the dependency runs the other way, and reversing it creates a cycle that some resolvers handle poorly. The data package should depend on nothing at all, which is what lets several code packages share it.

Can the split be reversed later?

Yes, and it is disruptive in the same way any dependency change is: users get a package that no longer needs the data wheel while the old one is still installed. Announce it, and keep accepting the data package as a dependency for a release or two.