6  Software

6.0.1 Interactive Applications

We have developed a series of interactive applications to explore the data and results of the MS project. These applications allow users to visualize the data, explore the results, and interact with the data in a more intuitive way. The applications are built using the shiny package in R, which allows us to easily create a user interface with complex reactivity for an interactive web application easily accessed through a web browser. The applications are designed to be user-friendly and intuitive, with interactive maps, charts, and tables that allow users to explore the data in a more dynamic way.

6.0.2 Overcoming Challenges with Large Spatial Data

The MS project incorporates many large spatial datasets that are problematic to render in a typical interactive application. For instance, the most common interactive mapping R package leaflet has a 4MB limitation for displaying rasters (see “Large Raster Warning” in Raster Images • leaflet). Vectors (i.e., points, lines and polygons) get smoothed when containing many vertices, but contiguity gets lost between polygons and rendering degrades to non-usable depending on the internet speed of the user’s connection.

To work around these limitations, we have implemented “cloud native” web services and formats (see also Cloud-Optimized Geospatial Formats Guide). Our implementations effectively reduce the size of any given spatial object based on the zoom level of the user’s browser. For rasters, we use cloud-optimized GeoTIFFs (COGs) and for vectors, we use Mapbox Vector Tiles (MVT). These formats are designed to be fast and efficient for web mapping applications, and they allow us to display large spatial datasets in an interactive web application without sacrificing performance or usability. Let’s take a closer look at implementation of each.

6.0.2.1 Raster: Cloud-Optimized GeoTIFFs (COGs) and Titiler

Historically, to read a raster, such as a GeoTIFF, from the web, the client software would have to read the entire file before rendering. Cloud Optimized GeoTIFFs (COGs) take advantage of HTTP GET range requests to read only the part of the file needed for rendering. So a COG stores quadtree simplifications of the original raster at multiple zoom levels and metadata for accessing their byte ranges in the file in the metadata header. This allows the client software to request only the parts of the file needed for rendering, which can greatly reduce the amount of data transferred and speed up rendering. This is for accessing the raw data in pixel values, e.g., for a raster of species distribution then the abundance of a species in each cell. We would want to also apply a color ramp to visualize the data. The open-source (TiTiler software is a lightweight web service that serves up these color ramped tiles on the fly. So COGs can be stored on a simple file server (like Amazon S3 or Azure Blob Storage) and served up as interactive web maps with TiTiler as an intermediary between the COG files and the client accessing the interactive Shiny mapping app (Figure 6.1).

sequenceDiagram
  autonumber
  Client      ->> ShinyServer: request Shiny map app with a raster layer
  ShinyServer ->> Client:      return JavaScript (JS) with URL specification for raster tiles
  loop for every tile in the current view (based on zoom level and bounding box)
    Client     ->> TiTiler:    request specific raster tile ({z}/{x}/{y})
    TiTiler    ->> FileServer: read metadata from COG file and request byte range for tile ({z}/{x}/{y})
    FileServer ->> TiTiler:    send byte range for tile
    TiTiler    ->> Client:     apply color ramp to tile and return PNG
  end

Figure 6.1: Sequence diagram implementing large raster interactive display using Cloud-Optimized GeoTIFFs (COGs) and Titiler in a Shiny mapping app.

6.0.2.2 Vector: Mapbox Vector Tiles (MVTs) and pg_tileserv

Although “cloud native” vector formats exist for simple file storage (see Cloud-Optimized Geospatial Formats Guide), none of these allow for flexible filtering and manipulation. Instead, we use PostgreSQL with the spatial extension (PostGIS) to store the vector data and serve it as Mapbox Vector Tiles (MVTs) using the pg_tileserv web service written in the language Go, which is very fast. This means that we don’t have to pre-render the MVTs (such as you might do with tippecanoe), but can instead serve the raw vector data directly from the database and let pg_tileserv handle the rendering on the fly. Filters (in the form of CQL) can be applied to the request. Symbology is rendered client-side via JavaScript, which allows for interactive hover and click events on vector objects (e.g., BOEM aliquot). Some speed-up is enabled by implementing a Varnish cache service in between. We can even write our own database functions for customized rendering, such as H3 hexagonal summaries. This allows us to serve vector data as web maps with minimal configuration and setup, and it provides a fast and efficient way to display large vector datasets in an interactive web application (Figure 6.2).

sequenceDiagram
  autonumber
  Client        ->> ShinyServer: request Shiny map app with a vector layer
  ShinyServer   ->> Client: return JavaScript (JS) with URL specification for vector tiles
  loop for every tile in the current view (based on zoom level and bounding box)
    Client      ->> VarnishCache: request specific vector tile ({z}/{x}/{y})
    alt cache hit
      VarnishCache ->> Client: return vector tile
    else cache miss
      VarnishCache ->> pg_tileserv:  request specific vector tile ({z}/{x}/{y})
      pg_tileserv  ->> pgDB:         run query to create vector tile in PostGIS database
      pgDB         ->> pg_tileserv:  return vector tile in compressed Protobuf format (*.pbf)
      pg_tileserv  ->> VarnishCache: store vector tile
      VarnishCache ->> Client:       return vector tile
    end
  end

Figure 6.2: Sequence diagram implementing large vector interactive display using Mapbox Vector Tiles (MVTs) and pg_tileserv in a Shiny mapping app.

6.0.3 Github Repositories

repo	description
api	application programming interface (API) using R Plumber package
apps	Shiny applications
docs	documentation for BOEM’s offshore environmental sensitivity index products
manuscripts	Manuscripts with review of sensitivities by industry and receptors (species, habitats, human uses)
MarineSensitivity.github.io	default website
msens	R library of functions for mapping marine sensitivities, sponsored by BOEM
objectives	repository for issues spanning multiple repositories and doing big picture roadmapping
server	server setup for R Shiny apps, RStudio IDE, R Plumber API, PostGIS database, pg_tileserv
workflows	scripts for testing data analytics and visualization as well as production workflows

6.0.4 Software Components