1 of 14

�deck.gl as a cross-language

visualization architecture

​

Public proposal

​

Last Updated April 15, 2020

2 of 14

Motivations

When we say that a code base is an implementation of “deck.gl”,�what exactly do we mean?�

  • deck.gl v9+ will have a growing set of language bindings
    • JS scripting, JSON playground, pydeck, mobile…
    • Ensure consistent API across bindings
    • Most (but not all) bindings will use the JSON protocol
    • All bindings will need to handle “transport” type issues�
  • deck.gl v9+ will have multiple “renderers” (implementations)
    • JS and C++ renderers

3 of 14

Main Themes

  • Transport protocol
    • Define how styles, data and auxiliary information is exchanged
  • JSON syntax
    • Define “JSON extension” syntax (@@type, @=lambda, constants, etc)
    • Define semantics of multiple JSON updates
  • deck.gl “style” specification
    • Specify deck.gl props as official style spec? Versioning?
    • Specify additional styles, e.g. for multiple base maps?

​

​

4 of 14

Big picture: deck.gl as a cross-language architecture

deck.gl-native

​

​

​

​

​

​

​

dawn (native WebGPU)

​

​

​

​

​

@deck.gl/json

D3D12�Windows

OpenGL

Android

Rendered textures

pydeck (Python)

(Python classes generating deck json)

Interaction (events)

Swiftdeck�(Swift/Cocoa classes generating deck json)

deck.gl (the JS renderer)

​

​

​

WebGL

​

WebGPU

(TBA)

Renderers

@deck.gl/json (JSON)

(deck.gl playground example)

Language Bindings

Javadeck (Android)�(Java classes generating deck json)

...

Protocol

C++ API

JS API

React API

Vulkan�Linux

Metal�iOS/OSX

5 of 14

deck.gl: Protocol and Transport

deck-browser-renderer (JS)

​

​

deck-native-renderer

​

​

deck-native-renderer (C++)

​

​

​

​

​

dawn (native WebGPU)

JSON “styles” initial

pydeck

(Python classes generating deck json)

WebGL

WebGPU

cppdeck�(C++ classes generating deck json)

Renderers

@deck.gl/json

(deck.gl playground)

Language Bindings

swiftdeck (iOS)�(Swift classes generating deck json)

...

Protocol & Transport

deck-node-renderer (JS)

​

​

“WebGL-node” *

JSON “styles” update

JSON “styles” (referencing data blob)

Init: Version, Metadata (prop types, ...)

Backchannel: logs and errors

Transport: data “blob” (JSON,CSV,binary...)_

Backchannel: interaction events

Back transport: rendered textures/buffers...

Back transport: large selections...

javadeck (Android)�(Java classes generating deck json)

6 of 14

Transports: One API Multiple Implementations

​

​

​

​

​

iOS

“Swift” <-> WebView Transport

​

​

​

​

​

​

​

​

​

​

​

​

​

​

iOS WebView

(transport via iOS WebView)

​

​

​

​

Android

Java <-> WebView Transport

​

​

​

​

​

​

​

​

​

​

​

​

​

​

Android WebView

(transport via Android WebView)

​

​

​

​

Python

Notebook <-> Browser Transport

​

​

​

​

​

​

​

​

​

​

​

​

​

​

pydeck JS

(transport via Jupyter Widget hooks)

​

​

​

​

Java Layer classes

Swift Layer classes

pydeck classes

(transport via Jupyter hooks)

Hosted JS app (deck.gl playground fork)

(transport via/ native Android / iOS WebView APIs, accessed via window object)

WebSocket

Socket <-> Browser Transport

​

​

​

​

​

​

​

​

​

​

​

​

​

​

Arbitrary language

(transport via WebSocket)

Hosted deck app

7 of 14

Android “Javadeck” demo

webView = findViewById(R.id.mainWebView);

deckGlView = DeckGlView.configure(webView);

imageView = findViewById(R.id.imageView);

​

// … configure connection to imageView ...

Deck visState = � Deck.builder()

.initialView(ViewState.builder()

.latitude(37.72)

.longitude(-122.45)

.build())

.layers(Arrays.asList(Layer.builder()

.type(Layers.GEO_JSON_LAYER)

.dataUrl(“https://…”)

.args(...)

.build())

.build();

deckGlView.update(visState);

Code: https://github.com/UnfoldedInc/unfolded.gl/tree/master/javadeck

8 of 14

Requirements for a transport layer / protocol

  • Support successive JSON “style sheet” updates
    • Define semantics of multiple JSON styles: “incremental” setProps (Note: playground is React).
  • Allow for separate data transports
    • Efficiently support large data transfer (binary or text) to deck (next slide):
  • Back-channel: Initial metadata
    • Class and prop type metadata can be forwarded for better error checking / dynamic type creation in client languages (e.g. Python/Java/...) etc.
  • Back-channel: Logs and errors
    • Non-JavaScript devs do not look in browser console
  • Back-channel: Textures and Buffers
    • Mobile: Allows using deck.gl as an offline renderer, for integration with native mobile code
  • Back-channel: Large selections
    • A pydeck use case is to show a large point cloud in deck.gl, let the user select points and return the selected indices to the client language.

​

9 of 14

Data “Blob” Transport Messages

  • Use-cases for data “blob” messages
    • Binary data
    • Sending JSON data only once when updating JSON layer “styles” repeatedly
    • Non-JSON data
  • Possible types of data “blobs”
    • Text: JSON payloads (send big payloads only once, out of line with JSON “styles”)
    • Text: Non-JSON text payloads (CSV, …) - (support via loaders.gl?)
    • Binary payloads (arbitrary, no metadata)
    • Binary payloads (GLTF encoded, with metadata per glTF conventions)
    • Binary payloads (Arrow encoded, with metadata per Arrow and deck+Arrow conventions)
  • Introduce “data source” concept to deck.gl?
    • New deck.dataSources prop? Layer.data can reference 1) data 2) url 3) named data source?

​

10 of 14

Relevant documents / RFCs

  • deck.gl Component Wrapping RFC
    • Many of the use cases for the transport back-channel are laid out here
  • deck.gl Python Integration RFC
    • Many transport emerged from the pydeck implementation)
  • deck.gl Mobile Platform Support RFC (WebView Approach)
    • The ideas that led to javadeck which now provides a second transport implementation
  • deck.gl Transport format RFC
    • Proposes deck.gl mappings for metadata-less binary buffer “blob”s

​

11 of 14

deck.gl native renderer

12 of 14

deck-native-renderer: Internal Structure

@deck.gl/json

deck-native-renderer

(C++? Rust? …)

​

​

​

​

​

​

​

\

​

C++/JS code sharing ideas...

Share GLSL code (transpile to MSL on iOS)?

Rewrite tesselation algorithms in C++ and use in JS via WASM?

​

​

​

@luma.gl replacement

Metal / OpenGL / Vulkan Backends (iOS + Android)

Rendered textures,

Interaction events?

...

@deck.gl/json: layer protocol parser (PORT)

@deck.gl/layers: port subset of layers

@deck.gl/core: port subset of core

From external apps

In various languages

13 of 14

dawn: Cross-platform native WebGPU

dawn is an open-source cross-platform implementation of WebGPU providing:

  • WebGPU C/C++ headers (webgpu.h, maps one-to-one to the WebGPU IDL)
  • Native implementations of WebGPU using platforms' GPU APIs:
    • D3D12 on Windows 10
    • Metal on OSX (iOS under development)
    • Vulkan on Windows, Linux (ChromeOS and Android under development)
    • OpenGL as best effort where available
  • A client-server implementation of WebGPU for applications that are in a sandbox without access to native drivers

14 of 14

@luma.gl JS directions - WebGPU as “abstraction”?

Currently based on a WebGL class interface. Move to WebGPU-ish class interface?

  • V9 - Expose a set of WebGPU-oriented classes rather than WebGL2 classes.
    • Most concepts still there (Buffer, Texture, …) just a “finer” (more detailed) API.
  • deck.gl moves to use a more WebGPU styled API?
  • Provide WebGL-based fallback of WebGPU, dynamically load as needed?

Alternative: luma.gl abstraction layer completely hides WebGPU w/ custom classes.

  • Dawn already provides WebGPU as cross platform native abstraction layer…
  • Both JS and native deck impldmentations converging on WebGPU...
  • Would native deck renderer duplicate a luma.gl abstraction layer?
  • One “important user” wants WebGPU level access to deck layers...

​