1 of 21

Building Efficient and Scalable Data Visualisation Tools

Angus Comrie

2 of 21

Outline

  • Data visualisation challenges in RA
  • IDIA’s work on remote data visualisation
  • Demonstration
  • Other IDIA visualisation projects

3 of 21

Context: Radio Astronomy Image Cubes

  • “Image” of the sky in spatial and spectral dimensions
    • regular spatial coordinates (x- and y-axes)
    • Separated into frequency/velocity bins in spectral axis (z-axis)
    • (Polarization) Separated into “Stokes” axis (w-axis)
  • Rendering a slice: f(v) -> [0, 1] -> RGB
  • Common desktop tools: Karma/Kvis, SAO DS9

​

4 of 21

Context: Radio Astronomy Image Cubes

5 of 21

Common data organisation

  • Cubes stored in FITS or CASA formats
  • Usually single-precision floating point data
  • X usually contiguous in storage
  • Different axes => different I/O (sequential / random)

​

​

6 of 21

The Challenge: Data size

  • Image cubes are getting very large
    • MeerKAT: 16K x 16K x 500 (512 GB, 2TB Full-Stokes, 1GB slice)
    • SKA Phase 1: 64K x 64K x 500? (8 TB, 32 TB Full-Stokes, 16 GB slice)
    • ALMA: 4K x 4K x 4K (256 GB, 1 TB Full-Stokes, 64 MB slice)
  • Could take days to download on 100 mbps connection
  • Existing software ill-equipped to handle enormous images

7 of 21

Common workloads

  • Render 2D subset (slice)
    • Colour mapping
    • Contours
  • Graphs: 1D subset (profile)
  • Graphs: 3D subset and reduction (profile)
  • Histograms
  • Zooming / adjusting FoV

​

​

8 of 21

Slice (2D Subset)

  • Plane-aligned (XY, YZ, XZ)
  • Very common: XY slice (fixed Z)
  • Animate through channels
  • XY slice: single sequential read
  • YZ slice: lots of random reads

​

9 of 21

Profiles (1D subset)

  • Common: Axis-aligned
  • X, Y, Z profiles for given pixel
  • Polarization: Stokes as well
  • Z Axis: 1 single pixel read / channel

​

10 of 21

Regions (3D subset)

  • Common: Small XY subset
  • Z profiles for subset
  • 2D region slice reduced to single value
  • Still a lot of random reads!

​

11 of 21

Histograms (Slice or cube)

  • Used to define colour map bounds
  • Can be used to approximate percentiles
  • Global / Local bounding

​

12 of 21

IDIA’s work on remote data visualisation:�CARTA Viewer

  • Collaborative effort from four institutes
  • Desktop and server/client
  • Version 1.0 “preview” due 21st Dec
  • Server version being deployed to ILIFU early 2019
  • https://dev-viewer.idia.ac.za

13 of 21

IDIA’s work on remote data visualisation:�Viewer Testbed

  • Test ideas for CARTA
  • Many ideas already implemented in CARTA
    • Lossy compression of raw data
    • WebGL rendering
    • WebAssembly for c/c++ libraries and performance
  • Many more still to come
    • Accelerated region calculates
    • Pre-calculating and caching (using HDF5 schema)
  • https://viewer.idia.ac.za

​

14 of 21

Viewer Testbed

  • Files go through an “ingestion process”
    • Histograms for slices and cube pre-calculated
    • Rotated dataset (XYZ -> ZYX) cached
    • Stored in HDF5 according to purpose-built schema
  • Currently just looks at files in BeeGFS volume
  • General approach: Only send what you need (Dynamic cropping / resolution)

​

15 of 21

Viewer Testbed: Data to Image Pipeline

  • Server:
    • Cropping
    • Down-sampling
    • Compressing (ZFP compression)
    • Sending (WebSockets with protocol buffer)
  • Client:
    • Decompressing (WebAssembly)
    • Rendering (WebGL shaders)
    • Data used for (approximate) XY profiles

​

16 of 21

Viewer Testbed: Profiles

  • X, Y and Z profiles from mouse position
  • Full-Stokes files: Z profile for all Stokes values
  • Z Profile read using rotated dataset
    • Orders of magnitude faster
    • Interactive inspection of 3rd (and 4th) dimension

​

​

17 of 21

Viewer Testbed: Regions

  • Regions also use rotated dataset
  • Fixed set of stats operations
  • Basic shapes (rectangle, ellipse)

​

18 of 21

CARTA V1.0 “Preview”

  • Basic viewer functionality
  • WCS overlay (WebAssembly port of AST library)
  • X, Y and Z profiles
    • Not accelerated using rotated data sets
    • Z updated when cursor stops
  • Lots of rendering control
    • Histogram
    • Scaling
    • Percentiles
    • Basic contouring features (GPU-accelerated!)
  • Flexible interface (Floating and docked)

​

19 of 21

CARTA V1.0+ Roadmap

  • Dec 21st Desktop release
  • V1.1 in first half of 2019:
    • Overlay multiple images with same WCS type
    • Contours (limited support in 1.0)
    • Basic regions
  • Later in 2019:
    • Focus on scalability with tiled rendering
    • Integrating missing testbed features
  • TBD:
    • Collaborative features (annotations, overlays)

20 of 21

Visualisation Lab

  • Part of IDIA offices at UCT
  • “Video Wall” project
    • 2x2 array of high end ultra-HD displays (“8k” resolution 7860x4320)
    • Collaborative meeting space. Place to meet and examine large data sets in detail.
    • Plug-and-play with external graphics card support
  • “Cobra” Panoramic screen
    • Immersive single-user mini-dome
  • High end Virtual Reality setup
    • Motion controllers and room-scale tracking
    • Fully wireless

​

​

21 of 21

VR Projects