1 of 26

Sharing with Europeana:

Depositing and publishing 3D datasets for preservation and future access

Kate Fernie, CARARE

2 of 26

Introduction

Twin iT! marks a massive expansion of 3D in the cultural heritage sector
The webinars in this series by 4CH have covered the digitisation workflow from the initial plans to modelling techniques
Today I will talk about the data that you create and strategies for depositing and publishing your results

Oxbow Books, 15 Dec. 2003 ISBN: ‎ 1842170406

3 of 26

From fieldwork

to depositing your 3D dataset and providing access

3

4 of 26

3D Workflow

CAPTURE

MODELLING

TEXTURING

OUTPUTs

Point

Clouds

High Poly

Model

Low Poly

Model

Textures

Photos

5 of 26

Workflow and data

3D workflow uses various equipment and software

Survey equipment which produces raw data in various file types
Software to process the data to produce the desired results which is capable of

accepting data in certain file types
exporting data in certain file types

6 of 26

From fieldwork

to depositing your 3D dataset and providing access

Interoperability matters

during the project and for future access

https://open-science-training-handbook.github.io/Open-Science-Training-Handbook_EN/

7 of 26

What’s in a 3D dataset?

7

8 of 26

Data capture

Laser scanning Thousands of points representing X, Y, Z coordinates, colour and intensity Multiple scans Imported into software for registration and processing	Photogrammetry Thousands of pictures taken at different angles around a CH object Imported into software which uses a Structure from Motion (SfM) algorithm to produce a point cloud or a mesh	Reference measures GNSS or GPS ground control points TPS points on building features and elements Used for registering laser scan data and photogrammetry

9 of 26

Data - initial processing

Data produced in 3D projects is rarely used in raw, unedited form
In most projects there will be multiple datasets (e.g. overview and detail scans, photogrammetry, drone imagery, etc.) which are each registered to a coordinate system to create a unified point cloud
Point clouds are used to generate various products (such as polygonal meshes, CAD drawings and elevation models)

10 of 26

File types/formats

… Meshes

A lot of different hardware and software is involved. There are many file formats but common file types have potential for deposit and archiving.

Photogrammetry Import - images (proprietary raw or standard JPG/TIFF) Export - point cloud/, meshes and textures	Point clouds Original raw scan data (proprietary format) + XYZ (ASCII text) Registered point cloud: XYZ (ASCII text) E57 (evolving as a standard) LAS proprietary format

https://archaeologydataservice.ac.uk/help-guidance/guides-to-good-practice/data-collection-and-fieldwork/laser-scanning-for-archaeology/archiving-laser-scan-data/file-formats-for-archiving-datasets/

A lot of different hardware and software is involved which doesn’t all play well together. Most software has file export and import options. There are many file types but among them there are some common types which have potential for archiving.

Photogrammetry begins by collecting a lot of images, which are imported either as camera raw files (proprietary) or as standard JPGs. Standard archival recommendations for image files apply. The images are used to generate a point cloud or a mesh and textures. These are usually output from photogrammetry software and there is generally a choice of file formats for export, choosing open file formats (discussed next) is recommended.

Scanner manufacturers have their own software for processing the point clouds generated; the individual scans are registered and brought together to create a unified point cloud. The data can be exported in the proprietary format, or in a format such as XYZ (an ASCII text file), E57 or LAS. XYZ, E57 and LAS are not yet formal standards but they are on a promising track and are being widely adopted by manufacturers. For archiving the recommendation is to deposit the original raw scan data in both the original proprietary format and as ASCII text (XYZ); plus the registered point cloud - again as ASCII text plus proprietary / E57 and/or LAS.

Multiple formats are recommended for archiving as these datasets undergo considerable editing/processing, which can potentially remove valuable data. The ASCII version of a scan is good for long term preservation but it may look different to the version in the proprietary format or a format such as E57. This is a more is more approach to a challenging preservation scenario.

ASTM E57 is a promising option as a standard for laser scan data as it is supported by a number of vendors, scanners and systems. Files can store 3D point data with attributes (colour and intensity) and also 2D imagery obtained from the system. As well as being used in laser scanners, E57 can be used in flash LIDAR systems, structured light scanners and other systems.

The LAS format was developed by ASPRS to store aerial LIDAR data but is gaining support in the terrestrial scanning community.

11 of 26

From data to 3D model

In reality capture the point cloud is used to generate a mesh which defines the shape of an object
The data undergoes processing/editing
Textures are applied and
The mesh is optimized
A high poly count mesh is the first output

from this you can generate lower polygon count meshes
models in various formats (according to the planned uses)

137 million polygons

Discovery Programme

12 of 26

File types/formats

Meshes Open formats: OBJ + MTL Collada (.dae) X3D Other formats: PLY glTF STL (for printing)	Textures Can be saved as: JPEG PNG TIFF TARGA (TGA)

13 of 26

Need for archiving

Lower priority

Derivatives - 3D models for web; for 3D printing etc.

High priority

Original raw scan data (proprietary & ASCII)
Registered point cloud (ASCII + Open format)
Original images (photogrammetry)
High Poly mesh + textures
Low poly mesh

14 of 26

Data files plus metadata

What’s in a 3D dataset?

https://open-science-training-handbook.github.io/Open-Science-Training-Handbook_EN/

15 of 26

Metadata

When depositing 3D datasets archives need good documentation to enable preservation and access, this includes:

Details about the project and how the content was produced
The technical characteristics – tools and software used, data processing, control, geometry, textures, etc.
Info to enable discovery and access – the CH subject, period, copyright, access conditions, media, links etc.

16 of 26

Depositing for future access

Sean Doran, Discovery Programme

17 of 26

Plan for archiving from the start of the project

University repositories
(larger) cultural institutional repositories
National repositories such as DANS Data Vault, SND and DRI
International repositories such as https://zenodo.org/
Specialist data archives such as ADS

Image: European Data Journalism Network

Find a suitable digital archive or repository and check their requirements and what level of access they can offer.

18 of 26

Preparing for deposit

Don’t leave preparation of the dataset for archiving to the end of the project

Specify the data for archiving
Specify formats which have most potential for archiving

open, industry standard

Specify the documentation and metadata that’s needed
Agree the copyright status and access permissions

19 of 26

Deposit the dataset at the end of the project

https://doi.org 10.5281/zenodo.77350786

Now you can plan to publish other outputs!

20 of 26

Publishing

The 3D workflow can produce many outputs – from high poly models for HBIM, low poly models for publishing online to 3D models for printing and more.

These can be uploaded with texture files for publication on hosting platforms:

Commercial
National
Local

21 of 26

Viewers

Ideally you would select a hosting platform that can render your 3D model in a viewers.

There are 3D viewers available as open source software which can be installed on your local server/repository. For example, 3D Hop and the Smithsonian’s 3D viewer.

The alternative is to upload the content to a hosting platform (commercial or non-commercial) and take advantage of their viewer

22 of 26

Share3D

I’ve spoken previously about Share3D, which aims to help CH Sketchfab users to share their 3D models with Europeana.

Sign up for an account here: https://dashboard.share3d.eu/login.php

23 of 26

More publishing options

Smithsonian Voyager

https://smithsonian.github.io/dpo-voyager/

24 of 26

Why storing and preserving your results matters

EC directive focusses on cultural heritage at risk and the most visited sites
3D digitisation enables monitoring of the heritage (changes over time) and increases access
Securing your results for the future
Increases the possibility for reuse
Leads to future research
Supports tourism, education, enjoyment

Monitoring

25 of 26

Acknowledgements and references

Anthony Corns, Technology Manager, Discovery Programme

Marco Medici, INCEPTION

4CH Deliverable D3.1 Design of the CH Cloud and 4CH platform

4CH Deliverable D4.1 Report on standards, procedures and protocols

ADS Guides to Good Practice

25

26 of 26

Kate Fernie

kate@carare.eu

Thanks for your attention!

https://www.carare.eu

Training hub:

https://pro.carare.eu/en/training-hub/

CARARE is a non-profit membership association which offers advice, guidance and training in the creation, publication and use of digital data. A Europeana aggregator CARARE offers technical services to help institutions share their archaeological and architectural heritage content.