Projects
Making Map Data Pipelines Explicit and Reproducible
A small CLI experiment for describing how GeoParquet datasets become reusable vector tiles.
Starting Point
A map interface does not begin with pixels on a screen. Before data can be explored, choices have already been made about which features to include, which properties to preserve, how geometry is simplified, and at which zoom levels it should appear.
Those choices are often spread across shell commands, scripts, and infrastructure configuration. The resulting tiles may be reproducible in practice, while the intent behind them remains difficult to inspect.
What I Wanted to Try
How might the transformation from an analytical geospatial dataset to a map-ready artifact become explicit and repeatable?
I wanted to describe the desired result in one small plan: the source, filtering and schema, tiling behavior, output, and runtime. The plan would express what the tiles should be rather than a sequence of commands for producing them.
What I Built
I built Parqtiles, a plan-driven command-line tool that converts GeoParquet datasets into PMTiles. It can read local files or data in S3 and write the finished archive locally or back to S3.
The current pipeline uses DuckDB to read and transform GeoParquet, exports an intermediate FlatGeobuf file, and runs tippecanoe to create PMTiles. It can run locally in Docker or as an ECS batch task.
The plan defines a stable boundary between source data and downstream map interfaces. A simplified plan looks like this:
What I Found Interesting
The schema is more than a technical optimization. It defines which attributes a future map can use and which ones disappear. Zoom ranges and feature-dropping rules similarly affect what becomes visible at different scales.
This suggested that reproducibility alone is not the final problem. A pipeline can produce the same artifact every time while still hiding why its representational choices were appropriate. The plan records intent in a machine-readable form, but it does not yet translate that intent for the people who will interpret the resulting map.
The larger question is how a spatial data pipeline might carry an account of its decisions into the interface that consumes it. If a feature disappears at one zoom, an attribute was excluded, or dense areas were simplified, those transformations may matter to what a person can understand and decide. A reproducible artifact becomes more useful when its representational limits are also inspectable.
What It Does Not Do Yet
Parqtiles is an early personal prototype. It assumes a particular path through DuckDB, FlatGeobuf, and tippecanoe rather than supporting multiple tiling engines or complex processing graphs.
The plan can validate structure, schema references, and zoom constraints, but it does not yet explain the visual or statistical consequences of a choice. Reproducible output also does not guarantee that a chosen representation is appropriate for its audience or decision.
What I Would Try Next
- Record source and tool versions alongside each generated artifact
- Produce a readable summary of which properties and features were removed
- Preview how tiling choices change what is visible across zoom levels
- Compare two plans and explain how their resulting representations differ
- Connect the generated artifact to the interface and decisions that depend on it