15  Workflows

Workflows are Quarto notebooks that ingest source datasets, resolve taxonomy, merge models, compute scores and publish a release. They are the only writers of the pipeline database; everything downstream reads published artifacts.

Links:

The notebook list is deliberately not reproduced here: it is generated from the pipeline’s own metadata and changes whenever a dataset is added. What is documented here is the contract that generates it.

15.1 The pipeline is a graph, generated from the notebooks

The pipeline is a targets directed acyclic graph, and the graph is not hand-maintained. Each notebook declares its own place in it, in YAML front-matter:

msens:
  target_name: ingest_aquamaps
  workflow_type: ingest
  dependency: [build_cell_grid]
  output: data/manifests/ingest_aquamaps.json
  dataset: {ds_key: am, response_type: suitability, source_authority: AquaMaps,
            temporal_interval: static, native_format: raster}

The target list is built by parsing those blocks, so adding a dataset means adding a notebook — not editing a pipeline definition that could disagree with it. The dataset: block is also the single source of truth from which the dataset registry table is consolidated; a dataset that is ingested without one fails the build loudly rather than appearing in the data with no metadata.

15.2 Stages

Stage What it does
grid Builds the global 0.05° cell grid — the sampling unit every model is rasterized onto — with environmental covariates and study-area membership.
ingest One notebook per source dataset, writing one distribution per model keyed by this release’s model id (mdl_seq). Vector ranges are rasterized by exact area coverage; rasters are interpolated onto the grid.
merge Resolves every model to a common taxon, computes each taxon’s governing extinction risk, and merges the per-dataset surfaces into one surface per taxon (Chapter 8).
score Computes per-cell metrics, rescales within ecoregions, and aggregates to each spatial unit (Chapter 9).
schema Consolidates the registry tables — dataset, model, taxon_model, listing — that describe what the release contains.
release Publishes the release to S3 as partitioned Parquet, COGs, PMTiles and a STAC catalog, writes the version manifest, and (gated) deploys the serving tier.

15.3 Reproducibility rules

These are load-bearing, not stylistic. Each one exists because its absence caused a specific failure.

Every process is a chunk in a committed notebook, run by rendering to HTML. Including the parts that touch the outside world — the S3 sync, the server deploy, the app reload. Not an ad-hoc shell command. Rendering is what produces the tracked HTML output and the content-hash checkpoint; executing the code without rendering skips both, so it is a diagnostic, never a run.

Side effects are opt-in. Anything expensive or outward-facing sits behind an environment flag that defaults to off, so a routine render cannot publish or restart anything by accident.

The orchestration is part of the process. A notebook parameterized by version is only half reproducible if the loop over versions lives in a throwaway shell script, so the multi-version drivers are committed too. Likewise, forcing a rebuild is a flag — never moving output directories aside by hand, which leaves no record of what was rebuilt or why.

A version number is not proof. A notebook whose logic lives in the msens package is only as current as the installed package. The server once reported the right package version while running an older function, and seven release manifests were regenerated missing their zone tiles as a result — with nothing failing. So every behavior change gets a version bump and a changelog entry, and the container pins move with it.

Reproducibility beats uptime. Where a reproducible fix and a fast in-place workaround are both available, the reproducible one wins even at the cost of brief downtime. Drift between a running container and the image it was built from is a bug to fix immediately, not a reason to avoid rebuilding.

15.4 Content-addressed checkpoints

Each notebook writes a manifest recording a content hash of what it produced. A downstream stage can then tell whether its input actually changed, rather than trusting a timestamp — which is what makes a re-run cheap when nothing moved, and what makes an unexpected change visible.

15.5 Guarding the science

Scientific rules live in the msens package as documented functions, not as SQL strings inside a notebook — so a notebook calls the rule and a unit test asserts it, and the two cannot drift. Every non-trivial rule has a test with a small fixture asserting its exact expected output, and every bug that is found becomes a permanent named assertion.

This matters more than the aggregate checks. Whole-pipeline validation — comparing a release’s zone scores against the previous one — catches broad drift but hides rule-level breakage: one lost exclusion affected ~750 species while barely moving any aggregate score (Chapter 8).

See Chapter 14 for the schema these workflows produce, and Chapter 2 for what each release changed.