Continental scale point cloud data management with Entwine

Connor Manning

Open source software tools

PDAL

  • "Point Data Abstraction Library"
  • Translate & manipulate point cloud data

PDAL

  • Analogous to GDAL
  • GDAL → raster/vector
  • PDAL → point cloud

PDAL

Pipelines: workflows composed of stages

  • readers/writers/filters
  • Specified as JSON

Pipelines: templatizable, repeatable

PDAL - pipelines

PDAL - pipelines

PDAL

  • Powerful!
  • Building blocks to implement workflows

But what about...

Cities?

Countries?

States?

  • Point cloud organization software
  • Efficiently enables:
    • Querying
    • Analyzing
    • Visualizing
    • Enriching
  • Scalable to trillions of points
  • Built with cloud parallelization in mind

Entwine Point Tiles (EPT)

  • Static octree-based structure
  • Encoding-agnostic
  • Flexible attribute schema
  • Lossless

Losslessness:

  • Input dataset is fully reconstructible from EPT
    • Precision
    • Attributes
    • Ordering
    • Metadata
  • "Slippy Map Tiles" for point clouds

Entwine’s scope:

^

EPT

Visualization

Exploitation

Exploitation & Analysis

  • EPT reader with PDAL

[

{

"type": "readers.ept",

"filename": "https://<endpoint>/denmark/ept.json",

"bounds": [1076483, 7572563, 1077548, 7573369]

},

{ "type": "filters.outlier" },

{ "type": "filters.smrf" },

{ "type": "filters.range", "limits": "Classification[2:2]" },

{

"type": "writers.gdal",

"filename": "denmark-hill.tif",

"window_size": 2,

"resolution": 1

}

]

[

{

"type": "readers.ept",

"filename": "https://<endpoint>/kentucky/ept.json",

"resolution": 400

},

{

"type": "filters.hexbin",

"edge_size": 600,

"threshold": 1

}

]

EPT exploitation - enrichment

Infinitely append attributes ("addons")

  • Algorithm primitives
    • Normals, eigens, PCAs, etc.
  • Workflow results
    • Classifiers, annotations, error models
  • No write-access needed - dynamically mapped

[

"https://na.entwine.io/autzen/ept.json",

{ "type": "filters.assign", "assignment": "Classification[:]=0" },

{ "type": "filters.my-awesome-ground-algorithm" },

{

"type": "writers.ept_addon",

"addons": { "~/entwine/addons/autzen/my-stuff": "Classification" }

}

]

[

{

"type": "readers.ept",

"filename": "https://na.entwine.io/autzen/ept.json",

"addons": { "Classification": "~/entwine/addons/autzen/my-stuff" }

},

// ...pipeline things here.

]

Cesium + 3D Tiles

  • Great for mixed media
    • Building models, terrain data, point clouds, etc.
  • Flexible tiling formats
  • Robust rendering software

Cesium + 3D Tiles - point clouds (vs. EPT)

  • No open source tooling to build/exploit
  • Format is oriented for visualization
    • No industry standard encodings (e.g. LASzip)
    • Compression is optimized for GPU
    • Non-renderable attributes are deprioritized
  • Large metadata (no implicit octree)
    • On the roadmap

EPT Tools

EPT Tools

  • Javascript/NodeJS utility suite to work with EPT data
    • Can run in browser
  • Tools:
    • validate - check EPT metadata
    • tile - translate EPT to 3D Tiles on disk
    • serve - live translation of EPT to 3D Tiles
  • More to come soon...

EPT Tools

ept serve https://<ept-project-root>

├── denmark/

├── netherlands/

└── romania/

├── ept.json

└── // EPT here

ept serve → λ-capable!

EPT Tools Lambda

  • AWS Lambda + API Gateway (or other cloud equivalent)
    • Serverless EPT → 3D Tiles transformation

+

+

+

USGS 3DEP

USGS LiDAR holdings

  • 1.4 million tiles (LAZ)
  • 14 trillion points
  • 1200 collections
  • 70+ terabytes
  • 2006-2019
  • S3 requester-pays bucket

Leveraging this data

  • Could we look at all of it?
  • Could we write software against it?
  • Could we query & filter it?
  • Could we get Amazon to pay for it?

AWS Public Datasets Program

Z sampling

> pdal info \

https://<endpoint>/IA_FullState/ept.json \

--stats --dimensions=Z \

--readers.ept.resolution=500 \

--readers.ept.threads=12

{

"average": 407.5926459,

"count": 3213897,

"maximum": 2010.49,

"minimum": -1896.85,

"name": "Z",

"stddev": 309.0541156,

"variance": 95514.44639

}

Classification sampling

> pdal info \

https://<endpoint>/IA_FullState/ept.json --stats

--dimensions=Classification \

--readers.ept.resolution=500 --readers.ept.threads=12 \

--filters.stats.count=Classification

"0/20394",

"1/618216",

"2/1825406",

"4/36",

"5/105",

"7/153474",

"12/596072",

"13/193",

"229/1"

Thanks!

entwine.io

FOSS4G 2019 Bucharest - Google Slides