1 of 37

Unlocking Analysis Ready Cloud Optimized (ARCO) file format in the OpenVisus Framework and NSDF-Data-Portal Update

​

2 of 37

OpenVisus

  • Remote Big Data Visualizations
  • Interactive exploration of massive scientific models

​

OpenVisus Viewer

Jupyter Notebook / OpenVisus Python package

JavaScript Web Viewer

3 of 37

Current OpenVisus Approach

4 of 37

Current OpenVisus Approach

5 of 37

(i) First try to read data from local storage

Current OpenVisus Approach

6 of 37

(i) First try to read data from local storage

(ii) The block is already locally stored

Current OpenVisus Approach

7 of 37

(i) First try to read data from local storage

(ii) The block is already locally stored

(ii) The block isn't cached locally, request the data from the server

Current OpenVisus Approach

8 of 37

(i) First try to read data from local storage

(ii) The block is already locally stored

(ii) The block isn't cached locally, request the data from the server

Current OpenVisus Approach

(iii) The server reads the block from the disk

9 of 37

(i) First try to read data from local storage

(ii) The block is already locally stored

(ii) The block isn't cached locally, request the data from the server

Current OpenVisus Approach

(iii) The server reads the block from the disk

(iv) The block is returned to the server

10 of 37

(i) First try to read data from local storage

(ii) The block is already locally stored

(ii) The block isn't cached locally, request the data from the server

(v) The server finds the data and returns it

Current OpenVisus Approach

(iii) The server reads the block from the disk

(iv) The block is returned to the server

11 of 37

(i) First try to read data from local storage

(ii) The block is already locally stored

(ii) The block isn't cached locally, request the data from the server

(v) The server finds the data and returns it

(vi) cache the new data and return it

Current OpenVisus Approach

(iii) The server reads the block from the disk

(iv) The block is returned to the server

12 of 37

  • Server-full Installation Required

(i) First try to read data from local storage

(ii) The block is already locally stored

(ii) The block isn't cached locally, request the data from the server

(v) The server finds the data and returns it

(vi) cache the new data and return it

Current OpenVisus Approach

(iii) The server reads the block from the disk

(iv) The block is returned to the server

13 of 37

  • Server-full Installation Required
    • difficult to properly horizontally scale nodes

(i) First try to read data from local storage

(ii) The block is already locally stored

(ii) The block isn't cached locally, request the data from the server

(v) The server finds the data and returns it

(vi) cache the new data and return it

Current OpenVisus Approach

(iii) The server reads the block from the disk

(iv) The block is returned to the server

14 of 37

What is Analysis Ready Cloud Optimized (ARCO)

    • Data has been cleaned
    • Rich metadata

Analysis Ready

    • Object storage
    • Support direct access to data subsets

Cloud Optimized

15 of 37

Enabling ARCO in OpenVisus

    • Requires a server-full installation
    • Difficult to properly horizontally scale nodes

OpenVisus limitations

    • Eliminates need for a OpenViSUS server
    • Inherit linear scalability from object storage infrastructure

OpenVisus + ARCO

16 of 37

OpenVisus + ARCO

​

17 of 37

OpenVisus + ARCO

​

(i) First try to read block from local storage

18 of 37

OpenVisus + ARCO

​

(i) First try to read block from local storage

(ii) The block is already locally stored

19 of 37

OpenVisus + ARCO

​

(i) First try to read block from local storage

(ii) The block is already locally stored

(ii) The block isn't cached locally, retrieve from object storage

20 of 37

OpenVisus + ARCO

​

(i) First try to read block from local storage

(ii) The block is already locally stored

(ii) The block isn't cached locally, retrieve from object storage

(iii) Return the block from object storage 

21 of 37

OpenVisus + ARCO

​

(i) First try to read block from local storage

(ii) The block is already locally stored

(ii) The block isn't cached locally, retrieve from object storage

(iv) cache the new data and return the block

(iii) Return the block from object storage 

22 of 37

OpenVisus + ARCO

​

(i) First try to read block from local storage

(ii) The block is already locally stored

(ii) The block isn't cached locally, retrieve from object storage

(iv) cache the new data and return it

No need for a server

(iii) Return the block from object storage 

23 of 37

Sharded Data

24 of 37

Optimal Block Size to maximize bandwidth: [1,8] MB

Similar to Commercial Company Block Size

​

25 of 37

ARCO Compatibility

​

​

​

​

​

With Proper Client Caching results are indistinguishable across ARCO supported frameworks 

​

ARCO

OpenVisus Viewer

YES

Python code

YES

Jupyter Notebook

YES

JavaScript Web Viewer

NO

26 of 37

27 of 37

Future Work

  • Implement ARCO for JavaScript Web Viewer
  • Test with more cloud providers
  • Enable ARCO compatibility for new IDX2 format with independent progressive encoding of resolution and precisions

28 of 37

Sharing Jupyter Notebook Data Visualizations to the Web (NSDF-Data-Portal)

29 of 37

Sharing Interactive Visualizations through the Web

Interactive Visualization and Exploration in the Browser

GitHub Repository

Repository for Jupyter Notebooks

Python Script

Reads in notebooks from GitHub and automatically servers them to the web

Flask and Bokeh

Share interactive visualizations on a centralized website

30 of 37

Web Access

  • Notebooks uploaded to the GitHub repo are automatically served to the web.

31 of 37

Notebook Storage

  • GitHub serves as a central repository for notebooks that have been Bokeh-ifed and can be served with the data portal

32 of 37

Easy to add Notebooks

  • Any changes made to the GitHub notebook repository are automatically reflected on the web data portal.

​

33 of 37

Live Demo

34 of 37

35 of 37

Live Demo

http://155.101.6.68:5999

​

​

36 of 37

Data Portal Links

The GitHub repository the example data portal servers reades files from: https://github.com/okoppe/Juypter-Notebook-Repo

​

Source code for data portal: https://github.com/okoppe/NSDF-Data-Portal

​

37 of 37

Acknowledgments

​

This research was supported by the National Science Foundation (NSF) under grant numbers #2138811; the Extreme Science and Engineering Discovery Environment (XSEDE) under allocation TG-CIS210128; Chameleon Cloud under allocation CHI- 210923; and IBM through a Shared University Research Award.