---
title: "Release marine-atlas — versioned Parquet → S3, view-DB serving, STAC"
msens:
target_name: release_marine_atlas
workflow_type: release
# releases the scored+registry+native tables and the serve surface — must run AFTER the
# whole chain, NOT just [auto] (grid+ingest). These 4 terminals transitively pull in
# merge -> score -> registry -> native so release never runs on a stale model_cell.
dependency: [build_registry, score_zone_metrics, publish_native, build_common_names]
output: data/manifests/release_marine-atlas.json
editor_options:
chunk_output_type: console
---
Publish the **v8 marine-atlas** and stand up its serving, reproducibly:
1. **Data → S3** `s3://oceanmetrics.io-public/marine-atlas/{ver}/`:
- `tables/` — derived scoring tables (`cell`, `taxon`, `dataset`, `model`,
`cell_metric`, `zone*`, `metric`) + the relational tables (`taxon_model`, `listing`).
- `dist_merged/` — merged per-taxon surfaces (`ms_merge|…`); `dist_merged_er/` — the per-cell
extinction risk beside it for the NMFS DPS species (what `score_cell_metrics` multiplied in).
- `serve/model_cell.parquet` — the **serving-optimized** surface: one file, globally
**sorted by `mdl_key`** (row-group zone-map pruning) so a titiler tile does an
HTTP-range point read; single file ⇒ anonymous GET (no S3 LIST/creds).
- `registry/` — `dataset`/`model`/`taxon_model`/`listing` Parquet.
- `dist/` — raw per-dataset surfaces (~74G) — opt-in (`RELEASE_RAW=1`).
2. **View DB** — a **tiny** `serve.duckdb` whose tables are *views* over the S3 Parquet
(path-style HTTPS URLs). Titiler reads this; the big data never rsyncs to the server.
3. **STAC** — `stac_build(version)` → deploy the catalog to the file host `/stac`.
4. **Deploy** (`RELEASE_DEPLOY=1`) — rsync the KB view DB + STAC to `msens1`, (re)build the
parallel `titiler-v8` service, restart caddy. v7 serving is untouched (A/B).
Granular flags for the parts: `DEPLOY_APPS=1` (Shiny apps), `DEPLOY_TABLES=1` (local
Parquet + views), `DEPLOY_API=1` (plumber image), `DEPLOY_CADDY=1` (URL routing alone),
`DEPLOY_TITILER=1` (restart the tiler so it drops GDAL's cached headers for COGs
repainted at their existing URLs — run it after any `publish_native` that rebuilt them).
## Setup
```{r}
#| label: setup
librarian::shelf(DBI, dplyr, duckdb, fs, glue, here, jsonlite, logger, quiet = T)
source(here("libs/paths.R"))
dir_atlas <- glue("{dir_big_v}/marine-atlas")
tbl_dir <- glue("{dir_atlas}/tables"); dir_create(tbl_dir)
serve_dir <- glue("{dir_atlas}/serve"); dir_create(serve_dir)
# the two served surfaces, as PATHS, defined here so a GRANULAR run (which skips the staging
# chunk) can still rsync them -- DEPLOY_TABLES=1 alone died on `serve_mc` not found (2026-08-27)
serve_mc <- glue("{serve_dir}/model_cell") # partitioned by mdl_id (titiler point reads)
cell_mc <- glue("{serve_dir}/cell_model") # partitioned by spatial tile (per-cell queries)
# the released table set, from the staged FILES, for the same reason: the repoint of the server's
# views (DEPLOY_TABLES) enumerates it, and the staging chunk that used to define it is skipped
rel_tables <- if (dir_exists(tbl_dir)) path_ext_remove(path_file(dir_ls(tbl_dir, glob = "*.parquet"))) else character(0)
manifest <- here("data/manifests/release_marine-atlas.json"); dir_create(path_dir(manifest))
s3_ver <- glue("{s3_atlas}/{ver}") # s3://…/marine-atlas/v8
s3_http <- glue("https://s3.us-east-1.amazonaws.com/{sub('^s3://', '', s3_ver)}") # path-style
do_s3 <- !nzchar(Sys.getenv("RELEASE_NO_S3"))
do_raw <- nzchar(Sys.getenv("RELEASE_RAW"))
do_deploy <- nzchar(Sys.getenv("RELEASE_DEPLOY"))
do_apps <- do_deploy || nzchar(Sys.getenv("DEPLOY_APPS")) # (re)deploy the v8 Shiny apps
# v7 apps are the DEFAULT live apps on a separate checkout/branch — deliberately NOT implied by
# RELEASE_DEPLOY, so a routine v8 release can never restart them. Opt in explicitly.
do_apps_v7 <- nzchar(Sys.getenv("DEPLOY_APPS_V7"))
# the URL surface (the 301s from every retired per-version app). Implied by a full release,
# and available on its own so a routing fix need not rebuild titiler to ship.
do_caddy <- do_deploy || nzchar(Sys.getenv("DEPLOY_CADDY"))
do_s3_tbl <- do_s3 || nzchar(Sys.getenv("RELEASE_S3_TABLES")) # push derived tables (native_asset) without the full serve cutover
# the derived tables are ALSO served from local Parquet on the server (see the tables chunk);
# DEPLOY_TABLES=1 refreshes just that, with no S3 push and no titiler/caddy restart.
do_tables <- do_deploy || nzchar(Sys.getenv("DEPLOY_TABLES"))
# the pre-release review gate (preview.marinesensitivity.org): force the docs-preview
# sidecar to pull, and PROVE the gate with curl (see the two chunks at the end)
do_docs <- do_deploy || nzchar(Sys.getenv("DEPLOY_DOCS"))
do_access <- nzchar(Sys.getenv("DEPLOY_ACCESS")) # sync Cloudflare Access apps to versions.json
do_check_preview <- nzchar(Sys.getenv("CHECK_PREVIEW"))
# A GRANULAR run converges the server on what is ALREADY published; it must not
# re-stage the release and must not push to S3.
#
# `do_s3` defaults ON, so `DEPLOY_APPS=1` on its own -- restarting two Shiny apps
# -- also re-staged every table, re-hashed the ~580M-row serving surface, and
# then SYNCED TO S3. Publishing as a side effect of a reload is not something a
# flag should be able to do by accident; it took an explicit RELEASE_NO_S3=1 on
# the command line to make an app restart safe, which is exactly backwards.
#
# So: if the run asks ONLY for granular targets, it publishes nothing. Any
# publishing intent -- RELEASE_DEPLOY, RELEASE_RAW, RELEASE_S3_TABLES,
# RELEASE_CORS, or an explicit PROMOTE_LATEST -- opts back into the full path.
.granular_flags <- c("DEPLOY_APPS", "DEPLOY_APPS_V7", "DEPLOY_TABLES", "DEPLOY_API",
"DEPLOY_CADDY", "DEPLOY_TITILER", "DEPLOY_DOCS", "DEPLOY_ACCESS",
"CHECK_PREVIEW")
.publish_flags <- c("RELEASE_DEPLOY", "RELEASE_RAW", "RELEASE_S3_TABLES",
"RELEASE_CORS", "PROMOTE_LATEST")
do_granular <- any(nzchar(Sys.getenv(.granular_flags))) &&
!any(nzchar(Sys.getenv(.publish_flags)))
# staging, the S3 push, the view DB, STAC and the manifest are all PUBLISHING
# steps; a granular run skips them wholesale (see `eval:` on those chunks).
do_stage <- !do_granular
if (do_granular) {
do_s3 <- FALSE; do_s3_tbl <- FALSE
.on <- .granular_flags[nzchar(Sys.getenv(.granular_flags))]
log_info("granular run [{paste(.on, collapse=', ')}] — no staging, no S3 push, no manifest")
}
# server targets (parallel titiler-v8)
srv_repo <- "/share/github/MarineSensitivity/server"
srv_viewdb<- glue("/share/data/big/{ver}/serve.duckdb")
# cell_model + the derived tables live as LOCAL Parquet on the server (see the viewdb chunk)
srv_cellmodel <- glue("/share/data/big/{ver}/cell_model")
srv_tables <- glue("/share/data/big/{ver}/tables")
srv_modelcell <- glue("/share/data/big/{ver}/model_cell")
srv_stac <- glue("/share/data/derived/stac/{ver}")
# ONE stock titiler serves every release's COGs (`titiler-v8`, hardcoded in the apps as the
# /cog tiler for all versions since the factory was retired). A new version therefore does NOT
# get a titiler-{ver} service; TITILER_SERVICE names the compose service to (re)build.
titiler_svc <- Sys.getenv("TITILER_SERVICE", "titiler-v8")
titiler_v8 <- glue("https://{titiler_svc}.marinesensitivity.org")
# Guards the PUBLISHING path only. It opens sdm_db -- the multi-GB source DB,
# which exists only where the pipeline runs -- so on a granular deploy it was
# both unnecessary and fatal: it is why `scripts/srv_render.sh release-marine-
# atlas.qmd DEPLOY_APPS=1` died at chunk 2 on the server, where v8's 31.9 GB
# sdm.duckdb does not live. A granular run touches no source data, so it has
# nothing to guard.
if (do_stage) stopifnot("run build_registry first" = {
con0 <- dbConnect(duckdb(sdm_db, read_only = TRUE)); on.exit(dbDisconnect(con0, shutdown = TRUE))
all(c("dataset", "model", "model_cell") %in% dbListTables(con0)) })
```
## Stage derived tables + serving surface → Parquet
```{r}
#| label: stage
#| eval: !expr do_stage
con <- dbConnect(duckdb(sdm_db, read_only = TRUE))
dbExecute(con, "PRAGMA memory_limit='12GB'"); dbExecute(con, "PRAGMA threads=6")
dbExecute(con, glue("PRAGMA temp_directory='{dir_atlas}/duckdb_tmp'"))
# small/derived scoring tables (model_cell is released as dist_merged + serve/, not here).
# native_asset (mdl_key -> COG/PMTiles input surfaces) is included when publish_native has run.
# zone_taxon: the per-zone species list the scores app reads. It MUST ship —
# the app cannot recompute it from the served model_cell (S3 Parquet partitioned
# by mdl_id for point reads); attempting to scan it fails with an S3 IO error.
# taxon_model + listing: the relational tables v3-v7 published. build_registry carries them
# from merge.duckdb into sdm.duckdb precisely so they are staged here AND seen by
# manifest_build()'s introspection — staging them straight to tables/ would publish them
# undiscoverably.
rel_tables <- c("cell", "taxon", "dataset", "model", "metric", "zone_taxon",
"cell_metric", "zone", "zone_cell", "zone_metric", "native_asset",
"taxon_model", "listing")
rel_tables <- intersect(rel_tables, dbListTables(con))
for (t in rel_tables)
msens::copy_atlas_parquet(con, glue("SELECT * FROM {t}"), glue("{tbl_dir}/{t}.parquet"))
# serving surface: Hive-PARTITIONED by the integer mdl_id (from the model registry), which REPLACES
# the 40-byte mdl_key string in the stored rows (compact: rows are just cell_id,val; mdl_id lives in
# the path). A titiler tile reads exactly ONE partition (serve/model_cell/mdl_id={id}/data_0.parquet)
# by exact path -> anonymous GET, no S3 LIST, no global ORDER BY (avoids the ~500GB sort spill). The
# STABLE public identifier stays mdl_key: titiler resolves mdl_key->mdl_id from the model registry,
# so mdl_id (which renumbers as models are added) never appears in a URL.
# skip the rebuild if it exists (model_cell rarely changes); RELEASE_REDO_SERVE=1 forces.
if (!dir_exists(serve_mc) || nzchar(Sys.getenv("RELEASE_REDO_SERVE"))) {
if (dir_exists(serve_mc)) dir_delete(serve_mc)
dbExecute(con, "SET partitioned_write_max_open_files=1024") # ~one partition per merged model
msens::copy_atlas_parquet(
con, "SELECT m.mdl_id, c.cell_id, c.val FROM model_cell c JOIN model m USING (mdl_key)",
serve_mc, partition_by = "mdl_id")
} else {
log_info("serve/model_cell/ exists — skipping repartition (RELEASE_REDO_SERVE=1 to force)")
}
# CELL-ORIENTED surface: the same model_cell data, partitioned by a 2.5-degree
# SPATIAL TILE so it can be queried BY CELL. serve/model_cell is partitioned by
# mdl_id, which serves a titiler tile as one point read but makes any per-cell or
# per-polygon question (the scores app's clicked cell, and the Report tab's
# arbitrary area) a full scan of ~580M rows — over HTTPS that fails outright.
#
# tile = 50x50 cells on the 7200-col global grid -> 422 partitions, avg 1.4M rows
# (p95 5.0M, max 10.2M), so one cell reads one small partition. Deliberately NOT
# globally sorted: an ORDER BY over 580M rows previously spilled ~500GB.
# Callers must filter on `tile` as well as `cell_id` to get partition pruning —
# msens::cell_model_tiles() computes the tile ids for a set of cells.
if (!dir_exists(cell_mc) || nzchar(Sys.getenv("RELEASE_REDO_CELL_MODEL"))) {
if (dir_exists(cell_mc)) dir_delete(cell_mc)
dbExecute(con, "SET partitioned_write_max_open_files=1024")
msens::copy_atlas_parquet(
con,
paste("SELECT", msens::cell_model_tile_sql("c.cell_id"), "AS tile,",
"m.mdl_id, c.cell_id, c.val",
"FROM model_cell c JOIN model m USING (mdl_key)"),
cell_mc, partition_by = "tile")
log_info("built serve/cell_model/ ({length(dir_ls(cell_mc, type='directory'))} tiles)")
} else {
log_info("serve/cell_model/ exists — skipping (RELEASE_REDO_CELL_MODEL=1 to force)")
}
# content fingerprint of the served surface (source table = deterministic, cheaper than scanning parts)
serve_hash <- msens::hash_query(con, "SELECT mdl_key, cell_id, val FROM model_cell")
n_parts <- length(dir_ls(serve_mc, type = "directory"))
dbDisconnect(con, shutdown = TRUE)
msens::report_table(tibble(
file = c(glue("tables/{rel_tables}.parquet"), glue("serve/model_cell/ ({n_parts} mdl_id partitions)")),
mb = round(c(file_size(glue("{tbl_dir}/{rel_tables}.parquet")),
sum(file_size(dir_ls(serve_mc, recurse = TRUE, glob = "*.parquet")))) / 1e6, 1)),
caption = "released Parquet (V2/zstd/80MB)")
```
## Sync data → S3
```{r}
#| label: s3
#| eval: !expr do_stage
sync <- function(local, prefix, del = FALSE, tries = 4) {
if (!dir_exists(local)) { log_warn("skip (missing): {local}"); return(invisible()) }
args <- c("s3", "sync", local, glue("{s3_ver}/{prefix}"), "--only-show-errors", "--no-progress")
if (del) args <- c(args, "--delete") # prune stale S3 keys not present locally
# retry transient network errors: `aws s3 sync` is resumable (re-uploads only missing/changed),
# so retrying the same command picks up where a dropped connection left off.
for (i in seq_len(tries)) {
st <- system2("aws", args, stdout = TRUE, stderr = TRUE)
if (is.null(attr(st, "status")) || attr(st, "status") == 0) {
log_info("synced {path_file(local)} -> {s3_ver}/{prefix}{if (del) ' (--delete)' else ''}")
return(invisible())
}
log_warn("sync {prefix} attempt {i}/{tries} failed (transient?); retrying …")
}
stop(paste(tail(st, 20), collapse = "\n"))
}
# derived tables (incl native_asset, which the species-app native/model toggle reads) can be
# pushed on their own via RELEASE_S3_TABLES, without the multi-GB serve/dist_merged cutover.
if (do_s3_tbl) sync(tbl_dir, "tables")
if (do_s3) {
# serve is Hive-partitioned by mdl_id, which RENUMBERS when the model set changes (e.g. the
# bird crosswalk merged 25 taxa) -> --delete prunes stale partitions + the legacy single file.
sync(serve_dir, "serve", del = TRUE)
sync(glue("{dir_atlas}/dist_merged"), "dist_merged")
# per-cell extinction risk beside the scoring surface (NMFS DPS species, v9+): what scoring multiplied in
if (dir_exists(glue("{dir_atlas}/dist_merged_er"))) sync(glue("{dir_atlas}/dist_merged_er"), "dist_merged_er")
sync(glue("{dir_atlas}/registry"), "registry")
if (do_raw) sync(glue("{dir_atlas}/dist"), "dist")
} else if (do_s3_tbl) {
log_info("RELEASE_S3_TABLES: synced tables/ only (serve/dist_merged/registry NOT pushed)")
} else log_info("RELEASE_NO_S3 set — staged locally only")
```
## Public-read CORS for browser DuckDB-WASM (`query.html`)
The workflows [`query.html`](query.html) page runs DuckDB-WASM **in the browser** and range-reads the
public `tables/*.parquet` directly. That needs the bucket's CORS to allow the `Range` request header:
a bare `AllowedMethods:[GET]` rule returns **403** on the preflight (`Access-Control-Request-Headers:
range`). One-time + idempotent; gated behind `RELEASE_CORS=1` so a normal release leaves CORS
untouched. The policy is a committed file (`data/s3/cors_public.json`) — public-read only, no ACL
change. Roll back with the 2-line `{AllowedMethods:[GET], AllowedOrigins:[*]}` prior config.
```{r}
#| label: s3-cors
#| eval: !expr do_stage
cors_file <- here("data/s3/cors_public.json")
if (nzchar(Sys.getenv("RELEASE_CORS")) && file_exists(cors_file)) {
bucket <- sub("^s3://([^/]+)/.*$", "\\1", s3_ver) # oceanmetrics.io-public
system2("aws", c("s3api", "put-bucket-cors", "--bucket", bucket,
"--cors-configuration", glue("file://{cors_file}")))
log_info("applied public-read CORS (GET/HEAD + Range header) to s3://{bucket} for browser DuckDB-WASM")
} else log_info("s3 CORS unchanged (set RELEASE_CORS=1 to (re)apply data/s3/cors_public.json)")
```
## Build the tiny view DB (views over the S3 Parquet)
```{r}
#| label: viewdb
#| eval: !expr do_stage
viewdb <- glue("{serve_dir}/serve.duckdb")
if (file_exists(viewdb)) file_delete(viewdb)
cv <- dbConnect(duckdb(viewdb))
# view sources: the released S3 Parquet (production) OR the local staged Parquet when
# RELEASE_NO_S3 (a local dry-run — nothing was pushed, so an S3 view would point at nothing).
if (do_s3 || do_s3_tbl) { # S3 views whenever the tables live on S3 (even a tables-only push)
dbExecute(cv, "INSTALL httpfs; LOAD httpfs;")
# tables: single files via explicit path-style HTTPS -> anonymous GET + HTTP range (the dotted
# bucket breaks virtual-hosted TLS, so never use s3://… for these).
tbl_src <- glue("{s3_http}/tables")
# model_cell glob needs S3 LIST -> the aws extension + credential chain (server has creds).
dbExecute(cv, "INSTALL aws; LOAD aws; SET s3_url_style='path';")
dbExecute(cv, "CREATE OR REPLACE SECRET atlas_s3 (TYPE s3, PROVIDER credential_chain, REGION 'us-east-1');")
mc_glob <- glue("s3://{sub('^s3://', '', s3_ver)}/serve/model_cell/*/*.parquet")
} else {
tbl_src <- tbl_dir # local staged tables
mc_glob <- glue("{serve_mc}/*/*.parquet") # local partitioned serve
}
# scoring tables store the metric in `val`; the Shiny apps still reference `value` (the pipeline
# value->val rename never fully propagated), so expose `val AS value` on these views as a
# back-compat alias — both column names resolve, no per-app query rewrites needed.
val_tbls <- c("cell_metric", "zone_metric", "zone")
for (t in rel_tables) {
extra <- if (t %in% val_tbls) ", val AS value" else ""
dbExecute(cv, glue("CREATE VIEW {t} AS SELECT *{extra} FROM read_parquet('{tbl_src}/{t}.parquet')"))
}
# model_cell is Hive-partitioned by the integer mdl_id (stored rows are cell_id,val). The HOT tile
# path reads one partition by EXACT path in titiler (factory mdl_key->mdl_id fast-path, anonymous
# GET, no LIST). This glob VIEW is for ad-hoc/statistics queries; it joins the `model` view back to
# the STABLE mdl_key so ad-hoc queries can select by mdl_key, not the internal id.
dbExecute(cv, glue("CREATE VIEW model_cell AS
SELECT m.mdl_key, mc.mdl_id, mc.cell_id, mc.val, mc.val AS value
FROM read_parquet('{mc_glob}', hive_partitioning = true) mc
JOIN model m USING (mdl_id)"))
# cell_model — the SAME rows partitioned by a 2.5-degree spatial tile, for the
# per-CELL and per-POLYGON questions model_cell cannot answer (clicked cell,
# Report-tab area). Deliberately a LOCAL path on the server, never S3: those
# queries touch many partitions, and over HTTPS that is exactly the pattern that
# fails. It is small enough to keep on disk (1.4 GB across 422 tiles).
# NOTE the view is created against the LOCAL staged path here. DuckDB validates
# a read_parquet() glob when the view is CREATED, so a server path cannot be
# used from the laptop — those files do not exist until the deploy chunk rsyncs
# them. The deploy chunk therefore REPLACES this view server-side, pointing at
# the server's local copy.
dbExecute(cv, glue("CREATE VIEW cell_model AS
SELECT cm.tile, cm.mdl_id, cm.cell_id, cm.val, cm.val AS value
FROM read_parquet('{cell_mc}/*/*.parquet', hive_partitioning = true) cm"))
# RECORD THE GRID this release tiles on, then prove the stored tile ids agree with it.
# msens resolved the tile width from a `cell_grid` table that no release had ever written, so
# it always fell back to global05's 7200 — right for v8 by coincidence, wrong for every usa05
# release, where it computed a different but perfectly VALID tile id and pruned away the only
# partition holding the cells. The clicked-cell species list then returned EMPTY rather than
# failing. cell_model_tile_check() cannot pass on that mismatch.
msens::cell_grid_write(cv, msens::grid_for_ver(ver))
msens::cell_model_tile_check(cv)
dbDisconnect(cv, shutdown = TRUE)
log_info("view DB {viewdb} ({round(file_size(viewdb)/1e3,1)} KB) — {if (do_s3 || do_s3_tbl) 'S3' else 'LOCAL'} sources, model_cell by mdl_id, cell_model by spatial tile (local)")
```
## Emit STAC catalog (tile links → titiler-{ver})
```{r}
#| label: stac
#| eval: !expr do_stage
suppressWarnings(try(library(msens), silent = TRUE))
source(here("../msens/R/stac.R")) # override any stale install with the current (v8) generators
stac_dir <- glue("{dir_atlas}/stac")
cfg <- stac_cfg(ver); cfg$titiler_base <- titiler_v8
con <- dbConnect(duckdb(sdm_db, read_only = TRUE))
stac_build(version = ver, dir_out = stac_dir, cfg = cfg, con = con)
dbDisconnect(con, shutdown = TRUE)
log_info("STAC catalog: {stac_dir}/{ver}/collection.json")
```
## Deploy to msens1 (parallel titiler-{ver})
```{r}
#| label: deploy
if (!do_deploy) {
log_info("RELEASE_DEPLOY unset — skipping server deploy (S3 data + local view DB/STAC ready)")
} else {
# 1. the KB view DB + the STAC version subtree (data stays on S3 — no big rsync)
system2("ssh", c("msens", glue("mkdir -p {path_dir(srv_viewdb)} {srv_stac}")))
# ... EXCEPT cell_model, which is deliberately local on the server: the per-cell /
# per-polygon queries it serves touch many partitions, which is precisely what
# fails over HTTPS. ~1.4 GB, so rsync only what changed (--delete prunes stale
# tiles if the grid or model set ever changes).
system2("rsync", c("-aq", "--delete", glue("{cell_mc}/"), glue("msens:{srv_cellmodel}/")))
log_info("synced cell_model -> msens:{srv_cellmodel}")
system2("rsync", c("-aq", viewdb, glue("msens:{srv_viewdb}")))
# repoint cell_model at the server's LOCAL copy (the staged view still names the
# laptop path). Done here, after the rsync, because DuckDB validates the glob at
# CREATE VIEW time — it must run where the files actually are.
fix_view <- glue(
# -u 1000:1000: `docker exec` runs as ROOT (the image USER, which RStudio
# Server's init needs), so a bare exec writes root-owned files into /share.
# The uids are NOT misaligned -- the container's rstudio user is already
# uid 1000 -- you just have to ask. Numeric, not `-u rstudio`: compose sets
# DEFAULT_USER, so the account NAME changes on the next container recreate
# while the uid does not. The damage is silent until git aborts a merge with
# "unable to unlink ...: Permission denied".
"docker exec -u 1000:1000 rstudio Rscript -e ",
shQuote(glue(
"con <- DBI::dbConnect(duckdb::duckdb('{srv_viewdb}')); ",
"DBI::dbExecute(con, \"CREATE OR REPLACE VIEW cell_model AS SELECT cm.tile, cm.mdl_id, ",
"cm.cell_id, cm.val, cm.val AS value FROM read_parquet('{srv_cellmodel}/*/*.parquet', ",
"hive_partitioning = true) cm\"); ",
"cat('CELLMODEL_OK', DBI::dbGetQuery(con, 'SELECT count(*) n FROM cell_model')$n, '\\n'); ",
"DBI::dbDisconnect(con, shutdown = TRUE)")))
out_cm <- system2("ssh", c("msens", shQuote(fix_view)), stdout = TRUE, stderr = TRUE)
if (!any(grepl("CELLMODEL_OK", out_cm))) stop(paste(tail(out_cm, 15), collapse = "\n"))
log_info("cell_model view repointed on server: {grep('CELLMODEL_OK', out_cm, value = TRUE)}")
system2("rsync", c("-aq", glue("{stac_dir}/{ver}/"), glue("msens:{srv_stac}/")))
# add this version as a child of the STAC root (keep other versions)
system2("ssh", c("msens", shQuote(glue(
"python3 - <<'PY'\nimport json\np='/share/data/derived/stac/catalog.json'\nc=json.load(open(p))\nh=[l['href'] for l in c['links'] if l.get('rel')=='child']\nif './{ver}/collection.json' not in h:\n c['links'].append({{'rel':'child','href':'./{ver}/collection.json'}}); json.dump(c,open(p,'w'),indent=2)\nPY"))))
# 2. update the server repo + REBUILD & FORCE-RECREATE titiler-{ver} (so a new factory always
# takes effect even if compose sees no config change), restart caddy. Explicit fetch + ff-only
# with `set -e` so a stale/failed pull FAILS LOUDLY instead of silently serving the old factory
# (a silent `git pull --quiet` once left the mdl_key factory undeployed for days).
deploy_out <- system2("ssh", c("msens", shQuote(glue(
"set -e; cd {srv_repo} && git fetch --quiet origin && git merge --ff-only origin/main && ",
"docker compose up -d --build --force-recreate {titiler_svc} && docker compose restart caddy && ",
"echo DEPLOY_OK $(git rev-parse --short HEAD)"))), stdout = TRUE, stderr = TRUE)
log_info("{titiler_svc} deploy: {paste(tail(deploy_out, 2), collapse=' | ')}")
if (!any(grepl("DEPLOY_OK", deploy_out))) stop(paste(tail(deploy_out, 15), collapse = "\n"))
# 3. smoke test — the stock /cog route on one of THIS release's published merged COGs (the
# factory is retired; every app layer reads a COG). /cog/info → expect 200.
c0 <- dbConnect(duckdb(sdm_db, read_only = TRUE))
cog_u <- if ("native_asset" %in% dbListTables(c0)) dbGetQuery(c0,
"SELECT asset_url FROM native_asset WHERE asset_type = 'cog' AND ds_key = 'ms_merge' ORDER BY mdl_key LIMIT 1")$asset_url else character(0)
dbDisconnect(c0, shutdown = TRUE)
if (length(cog_u)) {
code <- system2("curl", c("-s","-o","/dev/null","-w","%{http_code}","--retry","3",
shQuote(glue("{titiler_v8}/cog/info?url={URLencode(cog_u, reserved = TRUE)}"))), stdout = TRUE)
log_info("{titiler_svc} /cog/info on {basename(cog_u)}: HTTP {code} (expect 200)")
if (!identical(code, "200")) log_warn("{titiler_svc} cannot read this release's merged COG — check S3 + the service")
} else log_warn("no merged COG in native_asset to smoke-test (run publish_native with PUBLISH_MERGED_COG=1)")
}
```
## Serve the derived tables from LOCAL Parquet (versioned sync)
S3 stays the published artifact; **serving** reads a server-local copy. Every app interaction
re-reads each Parquet footer over HTTPS, which puts a **~140 ms floor** under every query — worst
for the small, interactive ones. Benchmarked on msens1 itself (same region as the bucket, so
best-case S3), running the scores app's real queries:
| query | rows | cold S3 | cold local | warm S3 | warm local |
|---|---|---|---|---|---|
| Program-areas panel (`app.R:1488`) | 20 | 0.364 s | 0.015 s | 0.147 s | 0.008 s |
| cell layer, full study area | 623,212 | 1.267 s | 0.244 s | 0.247 s | 0.180 s |
| species table for one Program Area | 3,232 | 0.468 s | 0.041 s | 0.054 s | 0.012 s |
| clicked-cell environment | 1 | 0.710 s | 0.175 s | 0.150 s | 0.068 s |
18–24× on the interactive queries, and *cold* is what every new app session pays. The whole set is
**445 MB** and syncs in ~14 s, so there is no reason to pay that. Same rsync-then-repoint pattern as
`cell_model`, and for the same reason: DuckDB validates a `read_parquet()` path at **CREATE VIEW**
time, so the repoint must run where the files are. Versioned by `{ver}` on both ends, so a version
bump lands in its own directory and the views follow it.
`model_cell` deliberately stays on S3 — it is the per-model point-read surface titiler uses, not
something the apps scan.
```{r}
#| label: tables-local
if (!do_tables) {
log_info("RELEASE_DEPLOY/DEPLOY_TABLES unset — leaving the server's table views as they are")
} else {
system2("ssh", c("msens", glue("mkdir -p {srv_tables} {srv_modelcell}")))
# PULL from S3 on the server rather than PUSH from here.
#
# tables/ is ~445 MB and S3 is the published artifact sitting in the SAME REGION as msens1,
# so the server fetches it in seconds; pushing it up from a laptop is a ~20-minute transfer
# over a ~400 KB/s link. That transfer failed three times in one night — the render process
# died mid-rsync each time, leaving an orphaned rsync still writing into the destination and,
# on one occasion, two concurrent `rsync --delete` runs against the same directory.
#
# Pulling is also the stronger guarantee: the server ends up holding exactly what was
# PUBLISHED, not whatever happens to be staged on the machine driving the release.
if (do_s3_tbl) {
out_t <- system2("ssh", c("msens", shQuote(glue(
"aws s3 sync {s3_ver}/tables/ {srv_tables}/ --delete --only-show-errors && echo TABLES_OK"))),
stdout = TRUE, stderr = TRUE)
if (!any(grepl("TABLES_OK", out_t))) stop(paste(tail(out_t, 15), collapse = "\n"))
log_info("server pulled tables from S3 -> {srv_tables}")
} else {
# nothing was pushed to S3 this run, so the local copy is the only source
system2("rsync", c("-aq", "--delete", glue("{tbl_dir}/"), glue("msens:{srv_tables}/")))
log_info("synced tables -> msens:{srv_tables} (rsync — tables not on S3 this run)")
}
# model_cell too — the LAST remote table. The v8 species app reads it in two
# places (mdl_bbox() for fit_bounds, and the clicked-cell value), and both are
# degrading silently today: the S3 GLOB needs LIST credentials the app
# container lacks, so the tryCatch around each returns NULL/NA and the app
# quietly falls back to the study-area extent and a blank value. 3.3 GB.
#
# Safe for tiles: titiler builds its own read_parquet path from the
# SERVE_MODEL_CELL env var (server/titiler/factory.py) and only uses this DB
# for the `model` registry — it never reads this view.
system2("rsync", c("-aq", "--delete", glue("{serve_mc}/"), glue("msens:{srv_modelcell}/")))
log_info("synced model_cell -> msens:{srv_modelcell}")
# Repoint the views — into a COPY, then swap it in atomically.
#
# Editing serve.duckdb in place needs a WRITE lock, and every running Shiny
# session holds a read-only one, so a live server fails with
# Could not set lock on file ".../serve.duckdb": Conflicting lock is held
# (it only ever succeeded when no app happened to be open). Copying first
# needs no lock — readers never mutate the file — and `mv` is atomic within
# the filesystem: in-flight sessions keep serving from the old inode, and the
# restart.txt touch rolls them onto the new one.
#
# `val AS value` is preserved (the apps still reference `value`), and each
# repoint is verified with a real count so a half-synced directory FAILS
# LOUDLY here rather than being discovered later as an empty app panel.
#
# The model_cell probe must draw its mdl_id from model_cell, NOT from `model`.
# `model` registers every model including the raw per-dataset ones, and only
# merged models have cells -- so `min(mdl_id) FROM model` is a raw model whose
# count is legitimately 0. That probe therefore returned 0 on a PERFECTLY
# HEALTHY release and would have returned 0 on a broken one too: a check that
# cannot distinguish the two states verifies nothing. It printed 0 beside
# "VIEWS_OK" for the 580,568,326-row v8 release.
repoint <- glue(
"vt <- strsplit('cell_metric,zone_metric,zone', ',')[[1]]; ",
"tt <- strsplit('{paste(rel_tables, collapse = \",\")}', ',')[[1]]; ",
"con <- DBI::dbConnect(duckdb::duckdb('{srv_viewdb}.new')); ",
"for (t in tt) DBI::dbExecute(con, sprintf(",
"\"CREATE OR REPLACE VIEW %s AS SELECT *%s FROM read_parquet('{srv_tables}/%s.parquet')\", ",
"t, if (t %in% vt) ', val AS value' else '', t)); ",
"DBI::dbExecute(con, \"CREATE OR REPLACE VIEW model_cell AS SELECT m.mdl_key, ",
"mc.mdl_id, mc.cell_id, mc.val, mc.val AS value FROM read_parquet(",
"'{srv_modelcell}/*/*.parquet', hive_partitioning = true) mc JOIN model m USING (mdl_id)\"); ",
"zm <- DBI::dbGetQuery(con, 'SELECT count(*) n FROM zone_metric')$n; ",
"mc <- DBI::dbGetQuery(con, 'SELECT count(*) n FROM model_cell WHERE mdl_id = ",
"(SELECT min(mdl_id) FROM model_cell)')$n; ",
"stopifnot(\'zone_metric view is empty\' = zm > 0, ",
"\'model_cell view returned no rows for its own first mdl_id\' = mc > 0); ",
# the repoint REPLACES the views, so it must also (re)record the grid and re-verify the
# tile ids here -- otherwise the server copy reverts to guessing the width, which is how
# the clicked-cell species list came back empty on every usa05 release.
"msens::cell_grid_write(con, msens::grid_for_ver('{ver}')); ",
"msens::cell_model_tile_check(con); ",
"cat('VIEWS_OK', length(tt), zm, mc, '\\n'); ",
"DBI::dbDisconnect(con, shutdown = TRUE)")
v8_apps <- c("species", "scores")
cmd_rp <- glue(
"set -e; ",
"cp -f {srv_viewdb} {srv_viewdb}.new; ", # no lock needed
# -u 1000:1000 for the same reason as the cell_model repoint above
"docker exec -u 1000:1000 rstudio Rscript -e {shQuote(repoint)}; ",
"mv -f {srv_viewdb}.new {srv_viewdb}; ", # atomic swap
"{paste(sprintf('touch /share/github/MarineSensitivity/apps_v8/%s/restart.txt', v8_apps), collapse='; ')}")
out_rp <- system2("ssh", c("msens", shQuote(cmd_rp)), stdout = TRUE, stderr = TRUE)
if (!any(grepl("VIEWS_OK", out_rp))) stop(paste(tail(out_rp, 20), collapse = "\n"))
log_info("views repointed to local Parquet + v8 apps reloaded: {grep('VIEWS_OK', out_rp, value = TRUE)}")
}
```
## Deploy the plumber API (`DEPLOY_API=1`)
The `/report` and `/species.csv` endpoints run in the **plumber** container, which is a *separate*
image from `rstudio` — the `DEPLOY_APPS` msens reinstall never reached it. Left unpinned, its msens
layer cached at **0.5.0** while the package moved to 0.12.0, so `/report?ver=v8` answered
`Cannot open database "…/v8/sdm.duckdb"` (no `serve.duckdb` fallback, msens 0.9.1) and drawn areas
had no `cell_model` support (0.9.0). `server/plumber/Dockerfile` now pins msens by commit and
asserts the version at build time, so this cannot recur silently.
Rebuild is required (not just a restart): msens lives in the image, not on a volume.
```{r}
#| label: deploy-api
if (!nzchar(Sys.getenv("DEPLOY_API"))) {
log_info("DEPLOY_API unset — leaving the plumber API as it is")
} else {
api_co <- "/share/github/MarineSensitivity/api"
cmd_api <- glue(
"set -e; ",
"cd {api_co} && git fetch --quiet origin && git merge --ff-only origin/main; ",
"cd {srv_repo} && git fetch --quiet origin && git merge --ff-only origin/main; ",
# --force-recreate, same reason the titiler deploy uses it: plumber.R is
# parsed ONCE at container start, and the api repo is a bind mount, so a
# source-only change leaves compose seeing no image change and the container
# keeps serving the old code indefinitely. (report.qmd is read per-render, so
# it updates without this — which made the gap easy to miss: the mdl_key fix
# took effect while the mapsp_base fix silently did not.)
"docker compose up -d --build --force-recreate plumber; ",
"echo API_OK $(git -C {api_co} rev-parse --short HEAD)")
out_api <- system2("ssh", c("msens", shQuote(cmd_api)), stdout = TRUE, stderr = TRUE)
if (!any(grepl("API_OK", out_api))) stop(paste(tail(out_api, 20), collapse = "\n"))
# assert the container actually carries the msens the endpoints need
chk <- system2("ssh", c("msens", shQuote(glue(
"docker exec plumber Rscript -e ",
shQuote("cat('MSENS', as.character(packageVersion('msens')), '\\n')")))),
stdout = TRUE, stderr = TRUE)
log_info("plumber API deployed: {grep('API_OK', out_api, value = TRUE)} | {grep('MSENS', chk, value = TRUE)}")
if (!any(grepl("MSENS 0\\.(1[2-9]|[2-9][0-9])", chk)))
stop("plumber msens < 0.12.0 — the v8 drawn-area path needs cells_in_polygon(poly, con)")
}
```
## Deploy the Caddy routing (`DEPLOY_CADDY=1`)
The **URL surface** is part of the release, not server trivia: since the cutover, every
retired per-version app URL is a 301 into `/scores` or `/species`, so a routing change decides
whether a link published in a report still resolves. It had its own bug — the redirect targets
carry their own `?ver=`, which REPLACES the incoming query, so `/mapsp/?mdl_seq=1434` arrived
as `/species/?ver=v7` and the app opened on its default taxon (MarineSensitivity/apps#6).
Caddyfile-only changes had no reproducible home: the routing rode along with `RELEASE_DEPLOY`,
which also rebuilds and force-recreates titiler. This flag is the narrow path — pull the server
repo, (re)build the caddy image if its Dockerfile changed, **validate** the Caddyfile with that
image, then restart caddy; nothing else. **`docker compose restart caddy`, not `caddy reload`**:
the Caddyfile is a single-file bind mount, so a git pull replaces the inode and the running
container keeps serving the file it opened at start. `up -d --build` alone is not enough for the
same reason — it is a no-op when image and compose config are unchanged — so it is followed by
an explicit restart. The validate step is what makes a typo in the Caddyfile a failed deploy
rather than an outage: caddy will not start on a bad config, and `restart` would leave the box
without its front door.
```{r}
#| label: deploy-caddy
if (!do_caddy) {
log_info("RELEASE_DEPLOY/DEPLOY_CADDY unset — Caddy routing left untouched")
} else {
# `set -e` + ff-only, same as the other deploys: a stale pull must fail loudly rather
# than restart caddy onto the config it was already running.
out_caddy <- system2("ssh", c("msens", shQuote(glue(
"set -e; cd {srv_repo} && git fetch --quiet origin && git merge --ff-only origin/main && ",
"docker compose build --quiet caddy && ",
"docker compose run --rm --no-deps -T caddy caddy validate --config /etc/caddy/Caddyfile --adapter caddyfile >/dev/null && ",
"docker compose up -d --no-deps caddy && docker compose restart caddy && ",
"echo CADDY_OK $(git rev-parse --short HEAD)"))),
stdout = TRUE, stderr = TRUE)
if (!any(grepl("CADDY_OK", out_caddy))) stop(paste(tail(out_caddy, 15), collapse = "\n"))
log_info("caddy validated + restarted: {grep('CADDY_OK', out_caddy, value = TRUE)}")
# PROVE THE PREVIEW ROUTES. caddy/test/run.sh starts a throwaway caddy from the
# same image importing the same routes file behind a test jwtauth, on the compose
# network, and asserts the URL scheme end to end (401 without a token, /v8/scores/
# renders v8 with ms-preview=1, ?ver= on that path is overridden, sockjs proxies,
# the pre-path spelling redirects and is never proxied). A routing regression is
# a failed deploy here, not a reviewer's bug report.
out_routes <- system2("ssh", c("msens", shQuote(glue("cd {srv_repo} && caddy/test/run.sh 2>&1"))),
stdout = TRUE, stderr = TRUE)
cat(paste(out_routes, collapse = "\n"), "\n")
if (!any(grepl("PREVIEW_ROUTES_OK", out_routes))) stop("preview routes test failed (see above)")
log_info("preview routes test: PREVIEW_ROUTES_OK")
# PROVE THE REDIRECT KEEPS THE QUERY. `curl -sf` on a 301 is not a check — it follows
# nothing and reports success on exactly the failure being fixed. Read the Location back
# and assert the model id survived it.
# the restricted-version redirect: Caddy's PREVIEW_RESTRICTED_VERSIONS (server .env) must be
# the registry's restricted set, or a version that flips public/restricted keeps (or stops)
# sending the public host's visitors to the review host. Assert BEFORE trusting the routes.
env_re <- trimws(paste(system2("ssh", c("msens", shQuote(glue(
"cd {srv_repo} && grep -E '^PREVIEW_RESTRICTED_VERSIONS=' .env | cut -d= -f2- | tr -d '\"'"))),
stdout = TRUE, stderr = TRUE), collapse = ""))
vers_now <- msens::atlas_versions(refresh = TRUE)
restr_now <- sort(vers_now$ver[!is.na(vers_now$access) & vers_now$access == "restricted"])
env_set <- sort(setdiff(trimws(strsplit(env_re, "|", fixed = TRUE)[[1]]), ""))
if (!identical(env_set, restr_now))
stop(glue("server .env PREVIEW_RESTRICTED_VERSIONS='{env_re}' != versions.json restricted ",
"'{paste(restr_now, collapse = '|')}' — edit .env on msens, then DEPLOY_CADDY again"))
log_info("PREVIEW_RESTRICTED_VERSIONS='{env_re}' matches versions.json")
for (v in restr_now) {
# shQuote the -w format too: system2() goes through a shell, and the space split it into
# `-w %{http_code}` plus a bogus URL `%{redirect_url}` (reported as "000302")
r302 <- system2("curl", c("-s", "-o", "/dev/null", "-w", shQuote("%{http_code} %{redirect_url}"),
shQuote(glue("https://app.marinesensitivity.org/{v}/scores/?mdl_key=abc"))), stdout = TRUE)
if (!identical(paste(r302, collapse = ""), as.character(glue("302 https://preview.marinesensitivity.org/{v}/scores/?mdl_key=abc"))))
stop(glue("public /{v}/scores/ should 302 to the review host with its query: got '{r302}'"))
log_info("public /{v}/scores/ -> {r302}")
}
for (u in c("mapsp/?mdl_seq=1434", "species_v8/?ver=v8&mdl_key=abc")) {
# shQuote is NOT optional: system2() runs through a shell, so a bare `&` in the URL
# ends the command there. The unquoted form asked for `?ver=v8` alone and this check
# then blamed Caddy for dropping the parameter it had never been sent.
loc <- system2("curl", c("-s", "-o", "/dev/null", "-w", "%{redirect_url}",
shQuote(glue("https://app.marinesensitivity.org/{u}"))),
stdout = TRUE)
log_info("redirect {u} -> {loc}")
id <- if (grepl("mdl_seq", u)) "mdl_seq=1434" else "mdl_key=abc"
if (!grepl(id, loc, fixed = TRUE))
stop(glue("Caddy dropped the model id: /{u} -> {loc}"))
}
log_info("redirects preserve the model id")
}
```
## Restart titiler after REPAINTING COGs at stable URLs (`DEPLOY_TITILER=1`)
`native/*` COGs live at **stable** keys — the app's `native_asset` registry points at them by
name — so republishing a model repaints bytes behind a URL titiler has already read. GDAL's
`/vsicurl` layer caches the header **per process**, and a repainted COG is usually a different
size, so the running container keeps computing offsets from the old length and reads past EOF:
z5+ served fine while z2–z4 returned HTTP 500. Restarting the container is the whole fix — the
cache is memory, not disk.
This is the narrow path, like `DEPLOY_CADDY`: pull the server repo and restart `titiler-{ver}`.
No `--build`, because the factory has not changed; use `RELEASE_DEPLOY` when it has.
The check reads the **served** bounds back and compares them to what the registry recorded for
the same model. A stale header still reports the OLD extent, so this fails on exactly the
condition it exists to catch — unlike an HTTP 200, which a stale-but-readable COG returns
happily.
```{r}
#| label: deploy-titiler
do_titiler <- nzchar(Sys.getenv("DEPLOY_TITILER"))
if (!do_titiler) {
log_info("DEPLOY_TITILER unset — {titiler_svc} left running (set it after repainting COGs)")
} else {
out_tt <- system2("ssh", c("msens", shQuote(glue(
"set -e; cd {srv_repo} && git fetch --quiet origin && git merge --ff-only origin/main && ",
"docker compose restart titiler-{ver} && echo TITILER_OK $(git rev-parse --short HEAD)"))),
stdout = TRUE, stderr = TRUE)
if (!any(grepl("TITILER_OK", out_tt))) stop(paste(tail(out_tt, 15), collapse = "\n"))
log_info("titiler-{ver} restarted: {grep('TITILER_OK', out_tt, value = TRUE)}")
# WAIT FOR READY before asserting anything. A container that has been asked to restart is not
# a container that is serving: the first version of this check queried /cog/info immediately,
# got an empty body back, and reported STALE HEADERS on COGs that were in fact correct. An
# unavailable service and a wrong answer are different failures and must not share an outcome.
tt_base <- sub("/msens$", "", titiler_v8)
ready <- FALSE
for (i in 1:30) {
code <- system2("curl", c("-s", "-o", "/dev/null", "-w", "%{http_code}", "--max-time", "10",
shQuote(glue("{tt_base}/healthz"))), stdout = TRUE)
if (identical(code, "200")) { ready <- TRUE; break }
Sys.sleep(5)
}
if (!ready) stop(glue("titiler-{ver} did not answer /healthz within 150s of restarting"))
log_info("titiler-{ver} healthy after {i} probe(s)")
# served bounds == registry bounds, on a sample of merged COGs
c0 <- dbConnect(duckdb(sdm_db, read_only = TRUE))
chk <- dbGetQuery(c0, "SELECT mdl_key, asset_url, ymin, ymax FROM native_asset
WHERE ds_key = 'ms_merge' AND asset_type = 'cog' AND ymin IS NOT NULL
ORDER BY mdl_key LIMIT 5")
dbDisconnect(c0, shutdown = TRUE)
bad <- character()
for (i in seq_len(nrow(chk))) {
# retry the READ separately: a missing answer is "ask again", only a mismatched answer is
# a finding. --retry alone does not cover a 200 with an unparseable body.
b <- NULL
for (try in 1:3) {
js <- system2("curl", c("-s", "--max-time", "60", "--retry", "2", "--retry-all-errors",
shQuote(glue("{tt_base}/cog/info?url={URLencode(chk$asset_url[i], reserved = TRUE)}"))),
stdout = TRUE)
b <- tryCatch(jsonlite::fromJSON(paste(js, collapse = ""))$bounds, error = function(e) NULL)
if (!is.null(b) && length(b) == 4) break
Sys.sleep(5)
}
if (is.null(b) || length(b) != 4) {
bad <- c(bad, glue("{chk$mdl_key[i]}: no bounds after 3 tries")); next
}
# 0.05 deg grid: agreement to a hundredth of a degree is exact, not approximate
if (abs(b[2] - chk$ymin[i]) > 0.01 || abs(b[4] - chk$ymax[i]) > 0.01)
bad <- c(bad, glue("{chk$mdl_key[i]}: served [{round(b[2],2)}, {round(b[4],2)}] != ",
"registry [{chk$ymin[i]}, {chk$ymax[i]}]"))
}
if (length(bad)) stop(paste(c("titiler is serving STALE COG headers:", bad), collapse = "\n"))
log_info("titiler-{ver} serves the republished COG extents ({nrow(chk)} models checked)")
}
```
## Deploy the v8 Shiny apps (reload only apps_v8; v7 untouched)
Reproducible app deploy — no ad-hoc ssh. Pull the **apps_v8** checkout
(`MarineSensitivity/apps@main`) and reload it via Shiny Server's per-app `restart.txt`.
Gated by `RELEASE_DEPLOY` (full release) or the granular `DEPLOY_APPS=1`.
> **Since the 2026-08-12 cutover this IS the live app.** `/share/shiny_apps/{scores,species}`
> point at `apps_v8`, which renders any published release from `?ver=`, and the 18 former
> per-version instances (`mapgl`, `mapsp`, `mapgl_v1-v6`, `mapsp_v1-v6`, `scores_v6/v8`,
> `species_v6/v8`) now live in `/share/shiny_apps_retired/` with Caddy 301ing their URLs to
> `/scores/?ver=v{n}`. So a deploy here is no longer "the v8 apps beside the live v7 ones" —
> it restarts what everyone sees. The touches target the CHECKOUT
> (`apps_v8/{scores,species}/restart.txt`), not the symlink, so retiring the aliases did not
> affect them.
```{r}
#| label: deploy-apps
if (!do_apps) {
log_info("RELEASE_DEPLOY/DEPLOY_APPS unset — skipping Shiny app deploy")
} else {
apps_co <- "/share/github/MarineSensitivity/apps_v8" # main branch; symlinked as *_v8
msens_co <- "/share/github/MarineSensitivity/msens"
v8_apps <- c("species", "scores")
# BOTH instances of the one app: the public one (the checkout Shiny Server :3838
# serves) and the PREVIEW one (:3839; the wrapper app.R per app in the server
# repo, MS_PREVIEW=1, behind the signed-in preview host). Shiny Server watches
# restart.txt in the directory it SERVES, so the wrapper's own dir is what
# reloads the preview process -- touching the checkout alone would leave it
# running the old code.
touches <- paste(c(sprintf("touch %s/%s/restart.txt", apps_co, v8_apps),
sprintf("touch %s/rstudio/shiny_apps_preview/%s/restart.txt", srv_repo, v8_apps)),
collapse = "; ")
# `set -e` + explicit ff-only fetch (a silent `git pull --quiet` once left the app + msens
# stale for hours). ALSO reinstall msens into the shiny/rstudio container: a stale container
# msens lacked `cell_tile_url(mdl_key=)` and crashed the merged surface. Reload only the v8 apps.
cmd <- glue("set -e; ",
"cd {apps_co} && git fetch --quiet origin && git merge --ff-only origin/main; ",
"cd {msens_co} && git fetch --quiet origin && git merge --ff-only origin/main && ",
"docker exec rstudio R CMD INSTALL --no-multiarch {msens_co} >/dev/null 2>&1; ",
"{touches}; echo APPS_OK $(git -C {apps_co} rev-parse --short HEAD)")
# ...and re-warm immediately. Touching restart.txt kills the workers, so the
# next visitor would pay the full cold start (~13-17 s) -- and the `warm`
# sidecar, on its 20-minute cycle, might not notice for that long. Restarting
# it re-enters its loop at once, so the deploy that invalidated the workers is
# also what rebuilds them.
cmd <- paste0(cmd, "; docker restart warm >/dev/null 2>&1 || true")
out <- system2("ssh", c("msens", shQuote(cmd)), stdout = TRUE, stderr = TRUE)
if (!any(grepl("APPS_OK", out))) stop(paste(tail(out, 15), collapse = "\n"))
log_info("deployed v8 apps [{paste(v8_apps, collapse=', ')}] on both instances (public + preview) + reinstalled msens: {tail(out,1)}")
}
```
## Pull the restricted docs (`DEPLOY_DOCS=1`)
The rendered books of **restricted** docs releases live on the docs repo's `gh-pages-preview`
branch (GitHub Pages cannot be gated), cloned at `/share/docs_preview` and served by the
signed-in preview vhost. The `docs-preview` compose sidecar polls that branch every 5 min, so
normally nothing needs doing; this flag forces the pull now (a fresh clone if the branch was
just created) and reports what is served.
```{r}
#| label: deploy-docs
if (!do_docs) {
log_info("RELEASE_DEPLOY/DEPLOY_DOCS unset — docs-preview left to its own polling")
} else {
# restarting the sidecar re-enters its loop, whose first act is clone-or-pull
out_docs <- system2("ssh", c("msens", shQuote(paste(
"set -e; docker restart docs-preview >/dev/null; sleep 20;",
"test -d /share/docs_preview/.git && echo DOCS_OK $(git -C /share/docs_preview rev-parse --short HEAD)",
"$(ls -d /share/docs_preview/v*/ 2>/dev/null | xargs -n1 basename | tr '\\n' ' ')",
"|| echo DOCS_NONE"))), stdout = TRUE, stderr = TRUE)
if (any(grepl("DOCS_NONE", out_docs)))
log_warn("docs-preview: no clone yet (gh-pages-preview branch absent?) — nothing served at /docs/ on the preview host")
else if (!any(grepl("DOCS_OK", out_docs))) stop(paste(tail(out_docs, 15), collapse = "\n"))
else log_info("docs-preview: {grep('DOCS_OK', out_docs, value = TRUE)}")
}
```
## Sync the Cloudflare Access applications (`DEPLOY_ACCESS=1`)
Who may see a restricted release is **per version**: the preview host serves each release under
its own path (`/v9/scores/`, `/docs/v9/`) and Cloudflare Access holds one application + reviewer
policy per version, because Access scopes by path and never by query string. Which versions are
restricted is `versions.json` — so the applications are *derived from the registry*, not
maintained by hand: `server/cloudflare/access.sh` reads it and converges (idempotent).
Run this **after** `build_version_manifest.qmd` has published a changed `access`, and whenever a
reviewer list changes. It runs on the server, where the API token and the reviewer lists live in
`.env` (never in a repo). New application AUDs must then go into `CF_ACCESS_AUD` — the script
prints the line — followed by `DEPLOY_CADDY=1`. Setup: `server/cloudflare/README.md`.
```{r}
#| label: deploy-access
if (!do_access) {
log_info("DEPLOY_ACCESS unset — Cloudflare Access applications left as they are")
} else {
out_acc <- system2("ssh", c("msens", shQuote(glue(
"set -e; cd {srv_repo} && git fetch --quiet origin && git merge --ff-only origin/main && ",
"set -a; . ./.env; set +a; cloudflare/access.sh"))), stdout = TRUE, stderr = TRUE)
cat(paste(out_acc, collapse = "\n"), "\n")
st <- attr(out_acc, "status")
if (!is.null(st) && st != 0) stop("access.sh failed (see above)")
# the AUD line is the hand-off to .env + DEPLOY_CADDY; surface it rather than
# leaving it buried in the log
aud <- grep("^CF_ACCESS_AUD=", out_acc, value = TRUE)
if (length(aud)) log_info("paste into the server .env, then DEPLOY_CADDY=1: {aud}")
}
```
## Prove the review gate (`CHECK_PREVIEW=1`)
Checks that **cannot pass on broken input** — every one reads a sentinel or a status code and
asserts on it, so a misrouted host or a leaked release fails the render. The apps put
`<meta name="ms-ver">` / `<meta name="ms-preview">` in `<head>` for exactly this.
- **public host, restricted version → NOT that version** (`ms-ver` = the promoted release,
`ms-preview` = 0); public versions render as themselves.
- **the preview host is actually PROXIED through Cloudflare** — asserted first, because every
other check below passes trivially when it is not: a grey-cloud DNS record sends visitors
straight to the origin, which 401s everything, so the gate reads as "closed" while Access is
not in the request path at all and no reviewer can ever get in. A Cloudflare-proxied response
carries `cf-ray`; the origin's does not.
- **preview host without a token → never 200** (Cloudflare Access 302s to its login; the
origin's `jwtauth` 401s). With a Cloudflare Access **service token**
(`CF_ACCESS_CLIENT_ID` / `CF_ACCESS_CLIENT_SECRET` in the environment) → 200,
`ms-preview` = 1, `ms-ver` = the requested (restricted) version. **The version is the PATH
there** (`/v9/scores/`, `msens::preview_app_url()`); the pre-path spelling `/scores/?ver=`
must redirect, never 200.
- **origin-direct** (`--resolve` the preview hostname to this server's IP, bypassing
Cloudflare) → 401, never 200. This is what proves proxying only that hostname is safe.
- **docs**: restricted versions 404 on GitHub Pages and (with a token) 200 on the preview host.
Reachability failures (no DNS yet, no cert yet — Phase 0/1 of the plan) are **warned**, never
counted as a pass; a 200 where the gate should hold is a hard stop.
```{r}
#| label: check-preview
if (!do_check_preview) {
log_info("CHECK_PREVIEW unset — review-gate checks skipped")
} else {
`%||%` <- function(x, y) if (is.null(x)) y else x
vers <- msens::atlas_versions(refresh = TRUE)
latest <- msens::atlas_latest(refresh = TRUE)
prev_url <- msens::atlas_preview_url()
prev_host <- sub("^https?://", "", prev_url)
restricted <- vers$ver[vers$access == "restricted"]
public <- vers$ver[vers$access == "public"]
tok_id <- Sys.getenv("CF_ACCESS_CLIENT_ID"); tok_sec <- Sys.getenv("CF_ACCESS_CLIENT_SECRET")
has_tok <- nzchar(tok_id) && nzchar(tok_sec)
tok_hdr <- if (has_tok) c("-H", shQuote(glue("CF-Access-Client-Id: {tok_id}")),
"-H", shQuote(glue("CF-Access-Client-Secret: {tok_sec}"))) else character()
log_info("versions: restricted=[{paste(restricted, collapse=',')}] public=[{paste(public, collapse=',')}] latest={latest} token={has_tok}")
# one GET -> list(code, ms_ver, ms_preview, location, ok=reachable)
probe <- function(url, extra = character()) {
f <- tempfile(); on.exit(unlink(f))
out <- suppressWarnings(system2("curl", c("-sS", "-m", "90", "-o", shQuote(f),
"-w", shQuote("%{http_code} %{redirect_url}"), extra, shQuote(url)), stdout = TRUE, stderr = TRUE))
st <- attr(out, "status"); if (!is.null(st) && st != 0) return(list(ok = FALSE, err = paste(out, collapse = " ")))
w <- strsplit(trimws(tail(out, 1)), " ")[[1]]
html <- if (file.exists(f)) paste(readLines(f, warn = FALSE), collapse = "\n") else ""
m <- function(nm) { r <- regmatches(html, regexec(sprintf('<meta name="%s" content="([^"]*)"', nm), html))[[1]]; if (length(r) > 1) r[2] else NA_character_ }
list(ok = TRUE, code = as.integer(w[1]), location = if (length(w) > 1) w[2] else "",
ms_ver = m("ms-ver"), ms_preview = m("ms-preview"), html = html)
}
fail <- character()
bad <- function(msg) { fail <<- c(fail, msg); log_error(msg) }
# 0. IS CLOUDFLARE EVEN IN THE PATH? Without this the whole check suite is
# vacuous: a DNS-only (grey-cloud) `preview` record means every request goes
# straight to the origin, which 401s -- "closed" for the wrong reason, and
# closed for the reviewers too. A proxied response carries `cf-ray`.
hdr <- suppressWarnings(system2("curl", c("-sS", "-m", "30", "-o", "/dev/null", "-D", "-",
shQuote(prev_url)), stdout = TRUE, stderr = TRUE))
if (!any(grepl("^cf-ray:", hdr, ignore.case = TRUE)))
bad(glue("{prev_host} is NOT proxied through Cloudflare (no cf-ray): set its DNS record to ",
"Proxied, or Access can never see -- or admit -- a request"))
else log_info("{prev_host} is proxied through Cloudflare (cf-ray present)")
# 1. the PUBLIC host never renders a restricted version: since 2026-08-28 it 302s the path
# form to the review host (Caddy @restricted_app, PREVIEW_RESTRICTED_VERSIONS in .env),
# carrying the query; public ones render as themselves
for (a in c("scores", "species")) {
for (v in restricted) {
r <- probe(glue("https://app.marinesensitivity.org/{v}/{a}/?probe=1"))
if (!r$ok) { bad(glue("public {a} ?ver={v}: unreachable ({r$err})")); next }
want <- glue("{prev_url}/{v}/{a}/?probe=1")
if (identical(r$code, 200L) && identical(r$ms_ver, v))
bad(glue("LEAK: public /{v}/{a}/ -> code={r$code} ms-ver={r$ms_ver} ms-preview={r$ms_preview}"))
else if (!identical(r$code, 302L) || !identical(r$location, as.character(want)))
bad(glue("public /{v}/{a}/ does not send a restricted version to the review host: code={r$code} -> {r$location} (want 302 {want}; is PREVIEW_RESTRICTED_VERSIONS in the server .env current?)"))
else log_info("public /{v}/{a}/ -> 302 {r$location} (review host, query kept; ok)")
# ...and the old spellings still get there, two hops, with their query intact
rq <- probe(glue("https://app.marinesensitivity.org/{a}/?ver={v}&probe=1"))
if (!rq$ok || !identical(rq$code, 301L) || !grepl(glue("/{v}/{a}/"), rq$location))
bad(glue("public {a} ?ver={v} no longer 301s to the path: code={rq$code %||% NA} -> {rq$location %||% NA}"))
}
for (v in public) {
# canonical since 2026-08-27: the version is the PATH on this host too
r <- probe(glue("https://app.marinesensitivity.org/{v}/{a}/"))
if (!r$ok || !identical(r$code, 200L) || !identical(r$ms_ver, v))
bad(glue("public /{v}/{a}/ does not render {v}: ok={r$ok} code={r$code %||% NA} ms-ver={r$ms_ver %||% NA}"))
# ...and every published deep link in the old spelling still lands there
rq <- probe(glue("https://app.marinesensitivity.org/{a}/?ver={v}"))
if (!rq$ok || !identical(rq$code, 301L) || !grepl(glue("/{v}/{a}/"), rq$location))
bad(glue("public {a} ?ver={v} no longer 301s to the path: code={rq$code %||% NA} -> {rq$location %||% NA}"))
}
}
# 2. the preview host: closed without a token; open + preview=1 with one
origin_ip <- trimws(system2("dig", c("+short", "app.marinesensitivity.org"), stdout = TRUE))[1]
for (v in c(restricted, head(public, 1))) {
u <- msens::preview_app_url("scores", v) # the version is the PATH on the preview host
r <- probe(u)
if (!r$ok) log_warn("preview {u}: unreachable without token ({r$err}) — DNS/cert not in place yet?")
else if (identical(r$code, 200L)) bad(glue("OPEN: preview {u} answered 200 with no token"))
else log_info("preview {u} without token -> {r$code} {r$location} (closed)")
if (nzchar(origin_ip)) {
r0 <- probe(u, c("--resolve", shQuote(glue("{prev_host}:443:{origin_ip}"))))
if (!r0$ok) log_warn("origin-direct {u}: unreachable ({r0$err}) — no cert yet?")
else if (identical(r0$code, 200L)) bad(glue("OPEN: origin-direct {u} answered 200 (jwtauth not enforcing)"))
else log_info("origin-direct {u} -> {r0$code} (closed)")
}
if (has_tok) {
rt <- probe(u, tok_hdr)
if (!rt$ok || !identical(rt$code, 200L) || !identical(rt$ms_preview, "1") || !identical(rt$ms_ver, v))
bad(glue("preview {u} with token: ok={rt$ok} code={rt$code %||% NA} ms-ver={rt$ms_ver %||% NA} ms-preview={rt$ms_preview %||% NA}"))
else log_info("preview {u} with token -> {v}, preview=1 (ok)")
# the pre-path spelling must never be served: it would sit outside every
# per-version Access policy
rq <- probe(glue("{prev_url}/scores/?ver={v}"), tok_hdr)
if (rq$ok && identical(rq$code, 200L)) bad(glue("OPEN: {prev_url}/scores/?ver={v} answered 200 (unversioned path served)"))
else if (rq$ok) log_info("preview /scores/?ver={v} -> {rq$code} {rq$location} (redirected, ok)")
}
}
# 2b. PER-VERSION isolation — the point of the whole path-based scheme. A
# credential scoped to one restricted version must open that version and be
# REFUSED elsewhere, which is what makes "reviewer of v9" different from
# "reviewer". access.sh mints a probe token per restricted version; if its
# pair is in the environment (CF_ACCESS_CLIENT_ID_PROBE_V9 / ..._SECRET_...),
# assert both halves. Skipped, loudly, when the pair is absent.
for (v in restricted) {
sfx <- toupper(gsub("[^A-Za-z0-9]", "_", v))
pid <- Sys.getenv(paste0("CF_ACCESS_CLIENT_ID_PROBE_", sfx))
psc <- Sys.getenv(paste0("CF_ACCESS_CLIENT_SECRET_PROBE_", sfx))
if (!nzchar(pid) || !nzchar(psc)) {
log_warn("no probe token for {v} in the environment — per-version isolation NOT asserted")
next
}
ph <- c("-H", shQuote(glue("CF-Access-Client-Id: {pid}")),
"-H", shQuote(glue("CF-Access-Client-Secret: {psc}")))
own <- probe(msens::preview_app_url("scores", v), ph)
if (!own$ok || !identical(own$code, 200L) || !identical(own$ms_ver, v))
bad(glue("probe token for {v} cannot open its OWN version: code={own$code %||% NA} ms-ver={own$ms_ver %||% NA}"))
else log_info("probe {v} opens /{v}/scores/ (ok)")
# ...and must NOT open another version, nor the landing page
for (other in c(setdiff(c(restricted, public), v)[1], NA)) {
u <- if (is.na(other)) prev_url else msens::preview_app_url("scores", other)
r <- probe(u, ph)
if (r$ok && identical(r$code, 200L))
bad(glue("ISOLATION BROKEN: the {v} probe token opened {u}"))
else log_info("probe {v} refused at {u} -> {r$code %||% 'unreachable'} (ok)")
}
}
# 3. docs: restricted versions are NOT on GitHub Pages; with a token they are on the preview host
for (v in restricted) {
r <- probe(glue("https://marinesensitivity.org/docs/{v}/"))
if (r$ok && identical(r$code, 200L)) bad(glue("LEAK: docs {v} is still published on GitHub Pages"))
else log_info("docs {v} on GitHub Pages -> {r$code %||% 'unreachable'} (ok)")
if (has_tok) {
rt <- probe(paste0(msens::preview_docs_url(v), "intro.html"), tok_hdr)
if (!rt$ok || !identical(rt$code, 200L) || !grepl(glue("documents release|>{v}</strong>"), rt$html))
bad(glue("preview docs {v}: ok={rt$ok} code={rt$code %||% NA} (intro sentence not found)"))
else log_info("preview docs {v} -> 200, documents {v} (ok)")
}
}
for (v in head(public, 2)) {
r <- probe(glue("https://marinesensitivity.org/docs/{v}/"))
if (!r$ok || !identical(r$code, 200L)) bad(glue("public docs {v} not served: {r$code %||% r$err}"))
}
if (length(fail)) stop("review-gate checks FAILED:\n ", paste(fail, collapse = "\n "))
log_info("review-gate checks passed ({length(restricted)} restricted, {length(public)} public; token={has_tok})")
}
```
## Deploy the v7 Shiny apps (opt-in — normally left alone)
The v7 apps are a **separate checkout** (`/share/github/MarineSensitivity/apps` on branch
`v7`, symlinked as `/share/shiny_apps/{species,scores}` → `mapsp`/`mapgl`). They are the
*default* live apps, so they stay untouched by a routine v8 release — hence a dedicated
`DEPLOY_APPS_V7=1` flag that `RELEASE_DEPLOY` deliberately does **not** imply.
Use it when a change must reach both generations — e.g. the usage-analytics instrumentation,
which is only meaningful if it covers the apps people actually use today. msens is installed
into the shared container by the v8 chunk above; v7 picks it up from the same library.
```{r}
#| label: deploy-apps-v7
if (!do_apps_v7) {
log_info("DEPLOY_APPS_V7 unset — v7 apps left untouched (the default)")
} else {
apps_v7_co <- "/share/github/MarineSensitivity/apps" # branch v7; symlinked as scores/species
v7_apps <- c("mapsp", "mapgl") # = species / scores
touches_v7 <- paste(sprintf("touch %s/%s/restart.txt", apps_v7_co, v7_apps), collapse = "; ")
# ff-only against origin/v7 (NOT main — this checkout tracks the v7 branch)
cmd_v7 <- glue("set -e; ",
"cd {apps_v7_co} && git fetch --quiet origin && git merge --ff-only origin/v7; ",
"{touches_v7}; echo APPSV7_OK $(git -C {apps_v7_co} rev-parse --short HEAD)")
out_v7 <- system2("ssh", c("msens", shQuote(cmd_v7)), stdout = TRUE, stderr = TRUE)
if (!any(grepl("APPSV7_OK", out_v7))) stop(paste(tail(out_v7, 15), collapse = "\n"))
log_info("deployed v7 apps [{paste(v7_apps, collapse=', ')}]: {tail(out_v7,1)}")
}
```
## Manifest
```{r}
#| label: manifest
#| eval: !expr do_stage
# content-addressed: fingerprint of the served model_cell surface (run-mode flags like
# pushed_s3/deployed are side effects, not content, so they're excluded).
msens::write_manifest(
manifest, target = "release_marine_atlas", content_hash = serve_hash,
stats = list(ver = ver, s3 = s3_ver, n_tables = length(rel_tables),
tables = paste(sort(rel_tables), collapse = ","),
serve = "serve/model_cell/ (partitioned by mdl_id)", n_partitions = n_parts,
titiler = titiler_v8),
force = msens::force_target("release_marine_atlas"))
```