turtledata.dbca.wa.gov.au
Architecture of a data dashboard at DBCA
Florian Mayer
turtledata.dbca.wa.gov.au
RShiny {turtleviewer2} https://turtledata.dbca.wa.gov.au/
Layout {bs4Dash} https://rinterface.github.io/bs4Dash/
Bootstrap {golem} https://thinkr-open.github.io/golem/
Data WAStD https://wastd.dbca.wa.gov.au/
Content {wastdr} https://dbca-wa.github.io/wastdr/
IDE RStudio https://www.rstudio.com/
Repo GitHub https://github.com/dbca-wa/turtleviewer2/
ETL GitHub https://github.com/dbca-wa/etlTurtleNesting/
DevOps Rancher https://az-k3s-bcs01.dbca.wa.gov.au/
Infra K3s https://rancher.com/
Data flow
turtledata.dbca.wa.gov.au
RShiny app
Docker image
ghcr.io/dbca-wa/turtleviewer2:0.2.0
Data ETL job cron job every 2h → download_data.R
Docker image
ghcr.io/dbca-wa/etlturtlenesting:0.6.16
Shared storage: Data
Docker volume
/app/inst
ODK to WAStD data & media (photos)
wastd.rds
wamtram.rds
Shared storage: Config
Docker volume
/app/config
.Renviron - WAStD, ODK Central, WAMTRAM credentials
wastd.dbca.wa.gov.au
odkc.dbca.wa.gov.au
API
API
WAMTRAM
Docker images
Docker images
Remember Virtual Machines?
Docker image = operating system + installed software + installed app + run command
Anyone (with Docker) can run any Docker image
Running Docker image = Docker container
Containers can have attached persistent storage
Containers are disposable, storage is persistent
FROM rocker/geospatial:4.1.3 as base
System packages required for R packages
R packages required for turtleviewer2
/app turtleviewer2 code, installed locally
/app/inst empty dir for data snapshots
Run command: turtleviewer2::run_app()
Runs a web server with the RShiny app
FROM rocker/geospatial:4.1.3 as base
System packages required for R packages
R packages required for etlTurtleNesting
/app etlTurtleNesting code, installed locally
/app/inst empty dir for data snapshots
/app/config empty dir for .Renviron config
/app.cron/cronjob cronjob config: run /app/cron/job.R every 2 hrs
/app/cron/job.R R script of ETL jobs: ODKC, WAStD, WAMTRAM
Run command: start cronjob
Storage Volume for data
Mounted to turtleviewer2 at /app/inst
Mounted to etlTurtleNesting at /app/inst
Contains data snapshots
Storage Volume for config
Mounted to etlTurtleNesting at /app/config
Contains .Renviron
cron cannot read environment variables
Development & DevOps
Development: Desktop to Docker DevOps: Docker to Browser
RStudio Desktop
Add code
Refactor > helper functions
Refactor > package
Add tests for helper functions
Use data snapshot
Run locally - works?
Release steps (see README):
Git commit & push > GH action “CI”
Git tag && push > GH action “docker”
GitHub Actions
Configured by .github/workflows/*.yml
Triggered by git push or git tag
CI: test code
Docker: build & push Docker image
Uses GH container registry
Builds:
RShiny app (turtleviewer2)
ETL job (etlTurtleNesting)
Rancher UI for Kubernetes Management
Configure Deployments:
Shell (system, R) - inside the container
Logs - error messages
Hosts:
RShiny app (turtleviewer2)
ETL job (etlTurtleNesting)
WAStD (wastd)
…
Want your own?
mpaviewer.dbca.wa.gov.au
RShiny app
Docker image
docker pull ghcr.io/dbca-wa/mpaviewer
Data ETL job cron job every ??h → download_data.R
Docker image
docker pull ghcr.io/dbca-wa/etlMPA Many CKAN CSVs → data.rda (list of df)
Shared storage: Data
Docker volume
/app/inst
Data snapshot
data.rds
Shared storage: Config
Docker volume
/app/config
.Renviron - CKAN credentials
data.dbca.wa.gov.au
API
Data updates �Whenever new data available�Can automate
Theme and components