8 Server

The server is for serving up any web services outside those of Github (e.g., website, docs and R package msens) using Docker (see the docker-compose.yml; with reverse proxying from subdomains to ports by Caddy).

8.1 Setup

For the latest instructions on launching an Amazon instance and installing the server software, see Server Setup · MarineSensitivity/server Wiki, which is pasted below for convenience…

on AWS as EC2 instance using Docker

8.1.1 launch instance

name: msens1:

Software Image (AMI)
Canonical, Ubuntu, 22.04 LTS, amd64 jammy image build on 2023-09-19 ami-0fc5d935ebf8bc3bc
Virtual server type (instance type)
t2.xlarge (4 vCPU, 16 GB memory)
Firewall (security group)
New security group
Storage (volumes)
2 volume(s)
- 20 GB
  / server software, disposable
- 60 GB
  /share for all data, persistent and to be backed up

8.1.1.1 allocate IP address

Elastic IP addresses | EC2 | us-east-1 for persistent IP address
Allocated IPv4 address: 100.25.173.0
Associate Elastic IP address

8.1.2 ssh to server

pem='/Users/bbest/My Drive/private/msens_key_pair.pem'
ssh -i $pem ubuntu@msens1.marinesensitivity.org

8.1.2.1 set hostname

Change the hostname of your Amazon Linux instance - Amazon Elastic Compute Cloud

sudo vi /etc/cloud/cloud.cfg
# preserve_hostname: true
sudo hostnamectl set-hostname msens1.marinesensitivity.org
sudo reboot

8.1.2.2 mount volume

The extra volume (60 GB for /share) was added during EC2 launch instance wizard, but needs to be mounted before available for use.

Make an Amazon EBS volume available for use on Linux - Amazon Elastic Compute Cloud

df -H

Filesystem      Size  Used Avail Use% Mounted on
/dev/root        21G  2.3G   19G  11% /
tmpfs           8.4G     0  8.4G   0% /dev/shm
tmpfs           3.4G  898k  3.4G   1% /run
tmpfs           5.3M     0  5.3M   0% /run/lock
/dev/xvda15     110M  6.4M  104M   6% /boot/efi
tmpfs           1.7G  4.1k  1.7G   1% /run/user/1000

lsblk

NAME     MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
loop0      7:0    0  24.6M  1 loop /snap/amazon-ssm-agent/7528
loop1      7:1    0  55.7M  1 loop /snap/core18/2790
loop2      7:2    0  63.5M  1 loop /snap/core20/2015
loop3      7:3    0 111.9M  1 loop /snap/lxd/24322
loop4      7:4    0  40.8M  1 loop /snap/snapd/20092
xvda     202:0    0    20G  0 disk 
├─xvda1  202:1    0  19.9G  0 part /
├─xvda14 202:14   0     4M  0 part 
└─xvda15 202:15   0   106M  0 part /boot/efi
xvdb     202:16   0    60G  0 disk

sudo file -s /dev/xvdb
# /dev/xvdb: data

So no file system on /dev/xvdb yet.

sudo mkfs -t xfs /dev/xvdb
sudo mkdir /share
sudo mount /dev/xvdb /share

sudo cp /etc/fstab /etc/fstab.orig
sudo blkid
# /dev/xvdb: UUID="bc766dfb-1c42-49cf-9320-2242a2d48a2e" BLOCK_SIZE="512" TYPE="xfs"
sudo vim /etc/fstab
# UUID=bc766dfb-1c42-49cf-9320-2242a2d48a2e  /share  xfs  defaults,nofail  0  2

df -h
sudo umount /share ; df -h
sudo mount -a      ; df -h

8.1.3 install docker

Following:

Step-by-Step Guide to Install Docker on Ubuntu in AWS | by Srija Anaparthy | Medium

sudo apt-get update
#OLD: sudo apt-get install docker.io -y

NEW: [[Migrate to docker compose]]

sudo systemctl start docker
sudo docker run hello-world
sudo systemctl enable docker
docker --version
# Docker version 24.0.6, build ed223bc
sudo usermod -a -G docker $(whoami)

8.1.3.1 run docker compose

/Users/bbest/My Drive/private/msens_server_env-password.txt

sudo chown -R ubuntu:ubuntu /share
mkdir -p /share/github/MarineSensitivity
cd /share/github/MarineSensitivity
# clone server repo
git clone https://github.com/MarineSensitivity/server.git
cd server

# add password, used as $PASSWORD in docker-compose.yml
echo 'PASSWORD=******' > .env

# launch docker instances
sudo docker-compose up -d

8.2 Docker compose

The Docker compose file is used to define and run multi-container Docker applications. Here is the docker-compose.yml file for the server pasted for convenience …

version: "3.9"

services:
  caddy:
    container_name: caddy
    image: caddy:latest
    ports:
      - 80:80
      - 443:443
    restart: unless-stopped
    volumes:
      - ./caddy/Caddyfile:/etc/caddy/Caddyfile
      - /share:/share
      - /share/caddy/data:/data
      - /share/caddy/config:/config

  rstudio:
    container_name: rstudio
    build: ./rstudio
    environment:
      ROOT: 'true'
      USER: admin
      PASSWORD: ${PASSWORD}
      ADD: shiny
    ports:
      - 8787:8787  # rstudio
      - 3838:3838  # shiny
    restart: unless-stopped
    volumes:
      - /share:/share
      - /share/shiny_apps:/srv/shiny-server

  plumber:
    container_name: plumber
    build: ./plumber
    ports:
      - 8888:8888  # api
    restart: unless-stopped
    volumes:
      - /share:/share
    depends_on:
      - postgis

  postgis:
    container_name: postgis
    image: postgis/postgis:latest
    environment:
      POSTGRES_DB: msens
      POSTGRES_USER: admin
      POSTGRES_PASSWORD: ${PASSWORD}
      ANON_PASSWORD: ${ANON_PASSWORD}
      PGDATA: /share/postgis/data
    volumes:
      # all files in /docker-entrypoint-initdb.dare automatically executed
      #  in alphabetical order on container creation
      - ./postgis/init.sh:/docker-entrypoint-initdb.d/init.sh  # add user anon
      - /share:/share
      - /share/postgis:/var/lib/postgresql
    restart: unless-stopped
    healthcheck:
      test: 'exit 0'
    ports:
      - 5432:5432

  pgadmin:
    container_name: pgadmin
    image: dpage/pgadmin4:8.14
    restart: always
    environment:
      PGADMIN_DEFAULT_EMAIL: ben@ecoquants.com
      PGADMIN_DEFAULT_PASSWORD: ${PASSWORD}
      PGADMIN_LISTEN_PORT: 8088
      # PGADMIN_CONFIG_CONFIG_DATABASE_URI: "'postgresql://admin:$PASSWORD@postgis:5432/msens'"
    ports:
      - 8088:8088
    volumes:
      - /share/pgadmin:/var/lib/pgadmin
    depends_on:
      - postgis

  pgbkups:
    container_name: pgbkups
    image: prodrigestivill/postgres-backup-local
    restart: always
    user: postgres:postgres # Optional: see below
    volumes:
      - /share/postgis_backups:/backups
      # sudo mkdir /share/postgis_backups; sudo chown -R 999:999 /share/postgis_backups
    links:
      - postgis
    depends_on:
      - postgis
    environment:
      - POSTGRES_HOST=postgis
      - POSTGRES_DB=msens
      - POSTGRES_USER=admin
      - POSTGRES_PASSWORD=${PASSWORD}
      - POSTGRES_EXTRA_OPTS=-Z6 --blobs
      - SCHEDULE=@daily
      - BACKUP_KEEP_DAYS=7
      - BACKUP_KEEP_WEEKS=4
      - BACKUP_KEEP_MONTHS=6
      - HEALTHCHECK_PORT=8088

  tile:
    container_name: tile
    environment:
      DATABASE_URL: 'postgresql://admin:${PASSWORD}@postgis:5432/msens'
    image: pramsey/pg_tileserv:latest
    depends_on:
      - postgis
    ports:
      - 7800:7800

  tilecache:
    container_name: tilecache
    image: varnish:latest # 7.4.2  # last updated: 2023-12-26
    volumes:
      - /share:/share
      # - "./varnish/default.vcl:/etc/varnish/default.vcl"
    ports:
      - 6081:6081
    environment:
      # VARNISH_SIZE: '2G'
      VARNISH_BACKEND_HOST: tile # .marinesensitivity.org
      VARNISH_BACKEND_PORT: 7800
      VARNISH_HTTP_PORT: 6081  # VARNISH_PROXY_PORT: '6081'
    #command: "-p default_keep=43200" # 60*60*12 = 43200 sec = 12 hrs
    restart: always
    depends_on:
      - "tile"

  rest:
    container_name: rest
    environment:
      PGRST_DB_URI: 'postgresql://anon:${ANON_PASSWORD}@postgis:5432/msens'
      PGRST_OPENAPI_SERVER_PROXY_URI: http://127.0.0.1:3000
      PGRST_DB_ANON_ROLE: anon  # db-anon-role
    image: postgrest/postgrest
    depends_on:
      - postgis
    ports:
      - "3000:3000"

  swagger:
    container_name: swagger
    image: swaggerapi/swagger-ui
    depends_on:
      - rest
    ports:
      - "8080:8080"
    expose:
      - "8080"
    environment:
      API_URL: https://rest.MarineSensitivity.org/

  titiler:
    container_name: titiler
    image: ghcr.io/developmentseed/titiler:latest
    environment:
      PORT: 8000
      # WORKERS_PER_CORE: 1
    ports:
      - "8000:8000"

8.3 DNS

The domain name server (DNS) records are managed by SquareSpace. The subdomains point to the server on Amazon at 100.25.173.0, whereas the main website is hosted by Github servers, per Managing a custom domain for your GitHub Pages site - GitHub Docs.

Host	Type	Data
@	A	185.199.111.153
@	A	185.199.110.153
@	A	185.199.109.153
@	A	185.199.108.153
api	A	100.25.173.0
file	A	100.25.173.0
msens1	A	100.25.173.0
pgadmin	A	100.25.173.0
rest	A	100.25.173.0
rstudio	A	100.25.173.0
shiny	A	100.25.173.0
swagger	A	100.25.173.0
tile	A	100.25.173.0
tilecache	A	100.25.173.0
titiler	A	100.25.173.0
www	CNAME	marinesensitivity.org

8.4 Caddyfile

The Caddyfile parameterizes the reverse proxying between the external subdomains and the Docker’s internal ports. Here is the Caddyfile pasted for convenience …

api.marinesensitivity.org {
  reverse_proxy plumber:8888
}

file.marinesensitivity.org {
  root * /share/public
  file_server browse {
    # serve *.zst, *.br or *.gz if file exists and client supports precompressed files
    precompressed zstd br gzip
  }
}

pgadmin.marinesensitivity.org {
  reverse_proxy pgadmin:8088
}

rest.marinesensitivity.org {
  reverse_proxy rest:3000
}

rstudio.marinesensitivity.org {
  reverse_proxy rstudio:8787
}

shiny.marinesensitivity.org {
  reverse_proxy rstudio:3838
}

swagger.marinesensitivity.org {
  reverse_proxy swagger:8080
}

tile.marinesensitivity.org {
  reverse_proxy tile:7800
}

tilecache.marinesensitivity.org {
  reverse_proxy tilecache:6081
}

titiler.marinesensitivity.org {
  reverse_proxy titiler:8000
}

8.5 Services

The server is running the following services:

RStudio
integrated development environment (IDE) to code and debug directly on the server

More info..
Shiny
interactive applications
e.g., shiny.marinesensitivity.org/map

More info..
PGadmin
PostGreSQL database administration interface

More info..
api
custom API: using R plumber

More info..
swagger
generic database API: using PostGREST

More info..
tile
spatial database API: using pg_tileserv for serving vector tiles

More info..