7  Software

The Marine Sensitivity Toolkit (MST) is a stack of software components for reproducibly, interactively and hierarchically generating environmental vulnerability maps and scores. Tabular data is collated across studies evaluating sensitivity of species to oil & gas and offshore wind energy development. The best available species and benthic habitat distributions are being mosaicked across the US EEZ. Scores will be summarized according to subgroups within Species, Benthic Habitats and Primary Productivity. These scores will be averaged into an overall score and visualized as a flower plot, applicable to various levels of BOEM relevancy: regional, ecoregions, protraction diagrams, blocks and aliquots. The software components for achieving this interactively are: 1) server — a Docker configuration to spin up all the software; 2) database — a spatially enabled database (PostgreSQL with PostGIS extension); 3) workflows — scripts (as Quarto notebooks) to explore, ingest and update the database and output files; 4) APIs — application programming interfaces for rendering vector (using pg_tileserv) or raster (using TiTiler) tile services for interactive mapping as well as a custom API (using R plumber); 5) libraries — re-usable documented functions as an R package for import, analysis and visualization using the APIs; 6) applications — interactive apps using the R Shiny framework for rendering vulnerability maps; and 7) documentation — as a Quarto notebook with figures, tables, glossary and references in interactive html or static docx/pdf output formats. This toolbox is intended to primarily serve BOEM needs internally, but by being open-source and fully reproducible the hope is to enlist buy-in and even contributions from external partners, whether from other government agencies, academia, NGOs or industry.

We ascribe to the philosophy of sharing all code for the sake of reproducibility, transparency and efficiency (Maitner et al. 2024; Lowndes et al. 2017); i.e. the FAIR principles of Findability, Accessibility, Interoperability, and Reusability (Wilkinson et al. 2016).

7.0.1 Interactive Applications

We have developed a series of interactive applications to explore the data and results of the MST project. These applications allow users to visualize the data, explore the results, and interact with the data in a more intuitive way. The applications are built using the shiny package in R (Chang et al. 2024), which allows us to easily create a user interface with complex reactivity for an interactive web application easily accessed through a web browser. The applications are designed to be user-friendly and intuitive, with interactive maps, charts, and tables that allow users to explore the data in a more dynamic way.

7.0.2 Overcoming Challenges with Large Spatial Data

The MS project incorporates many large spatial datasets that are problematic to render in a typical interactive application. For instance, the most common interactive mapping R package leaflet has a 4MB limitation for displaying rasters (see “Large Raster Warning” in Raster Images • leaflet). Vectors (i.e., points, lines and polygons) get smoothed when containing many vertices, but contiguity gets lost between polygons and rendering degrades to non-usable depending on the internet speed of the user’s connection.

To work around these limitations, we have implemented “cloud native” web services and formats (see also Cloud-Optimized Geospatial Formats Guide). Our implementations effectively reduce the size of any given spatial object based on the zoom level of the user’s browser. For rasters, we use cloud-optimized GeoTIFFs (COGs) and for vectors, we use Mapbox Vector Tiles (MVT). These formats are designed to be fast and efficient for web mapping applications, and they allow us to display large spatial datasets in an interactive web application without sacrificing performance or usability. Let’s take a closer look at implementation of each.

7.0.2.1 Raster: Cloud-Optimized GeoTIFFs (COGs) and Titiler

Historically, to read a raster, such as a GeoTIFF, from the web, the client software would have to read the entire file before rendering. Cloud Optimized GeoTIFFs (COGs) take advantage of HTTP GET range requests to read only the part of the file needed for rendering. So a COG stores quadtree simplifications of the original raster at multiple zoom levels and metadata for accessing their byte ranges in the file in the metadata header. This allows the client software to request only the parts of the file needed for rendering, which can greatly reduce the amount of data transferred and speed up rendering. This is for accessing the raw data in pixel values, e.g., for a raster of species distribution then the abundance of a species in each cell. We would want to also apply a color ramp to visualize the data. The open-source (TiTiler software is a lightweight web service that serves up these color ramped tiles on the fly. So COGs can be stored on a simple file server (like Amazon S3 or Azure Blob Storage) and served up as interactive web maps with TiTiler as an intermediary between the COG files and the client accessing the interactive Shiny mapping app (Figure 7.1).

sequenceDiagram
  autonumber
  Client      ->> ShinyServer: request Shiny map app with a raster layer
  ShinyServer ->> Client:      return JavaScript (JS) with URL specification for raster tiles
  loop for every tile in the current view (based on zoom level and bounding box)
    Client     ->> TiTiler:    request specific raster tile ({z}/{x}/{y})
    TiTiler    ->> FileServer: read metadata from COG file and request byte range for tile ({z}/{x}/{y})
    FileServer ->> TiTiler:    send byte range for tile
    TiTiler    ->> Client:     apply color ramp to tile and return PNG
  end
%% mmdc -i diagrams/raster.mmd --theme default --backgroundColor transparent --scale 8 -o diagrams/raster.png
Figure 7.1: Sequence diagram implementing large raster interactive display using Cloud-Optimized GeoTIFFs (COGs) and Titiler in a Shiny mapping app.

7.0.2.2 Vector: Mapbox Vector Tiles (MVTs) and pg_tileserv

Although “cloud native” vector formats exist for simple file storage (see Cloud-Optimized Geospatial Formats Guide), none of these allow for flexible filtering and manipulation. Instead, we use PostgreSQL with the spatial extension (PostGIS) to store the vector data and serve it as Mapbox Vector Tiles (MVTs) using the pg_tileserv web service written in the language Go, which is very fast. This means that we don’t have to pre-render the MVTs (such as you might do with tippecanoe), but can instead serve the raw vector data directly from the database and let pg_tileserv handle the rendering on the fly. Filters (in the form of CQL) can be applied to the request. Symbology is rendered client-side via JavaScript, which allows for interactive hover and click events on vector objects (e.g., BOEM aliquot). Some speed-up is enabled by implementing a Varnish cache service in between. We can even write our own database functions for customized rendering, such as H3 hexagonal summaries. This allows us to serve vector data as web maps with minimal configuration and setup, and it provides a fast and efficient way to display large vector datasets in an interactive web application (Figure 7.2).

sequenceDiagram
  autonumber
  Client        ->> ShinyServer: request Shiny map app with a vector layer
  ShinyServer   ->> Client: return JavaScript (JS) with URL specification for vector tiles
  loop for every tile in the current view (based on zoom level and bounding box)
    Client      ->> VarnishCache: request specific vector tile ({z}/{x}/{y})
    alt cache hit
      VarnishCache ->> Client: return vector tile
    else cache miss
      VarnishCache ->> pg_tileserv:  request specific vector tile ({z}/{x}/{y})
      pg_tileserv  ->> pgDB:         run query to create vector tile in PostGIS database
      pgDB         ->> pg_tileserv:  return vector tile in compressed Protobuf format (*.pbf)
      pg_tileserv  ->> VarnishCache: store vector tile
      VarnishCache ->> Client:       return vector tile
    end
  end
%% mmdc -i diagrams/vector.mmd --theme default --backgroundColor transparent --scale 8 -o diagrams/vector.png
Figure 7.2: Sequence diagram implementing large vector interactive display using Mapbox Vector Tiles (MVTs) and pg_tileserv in a Shiny mapping app.

7.0.3 Github Repositories

repo description
api application programming interface (API) using R Plumber package
apps Shiny applications
docs documentation for BOEM’s offshore environmental sensitivity index products
manuscripts Manuscripts with review of sensitivities by industry and receptors (species, habitats, human uses)
MarineSensitivity.github.io default website
msens R library of functions for mapping marine sensitivities, sponsored by BOEM
objectives repository for issues spanning multiple repositories and doing big picture roadmapping
server server setup for R Shiny apps, RStudio IDE, R Plumber API, PostGIS database, pg_tileserv
workflows scripts for testing data analytics and visualization as well as production workflows

7.0.4 Software Components