High-performance Seismology CyberTraining 2023�Cross-correlation of �Distributed Acoustic Sensing on the Cloud

Yiyu Ni^1*, Marine A. Denolle¹, Chengxin Jiang²

​

​

​

¹Department of Earth and Space Sciences, University of Washington

²Research School of Earth Sciences, the Australian National University

*niyiyu@uw.edu

​

May 10, 2023

​

​

Seismic Dense Array

2

​

​

• Several thousand stations
• Several hundred meter spacing
• Several months recording

Distributed Acoustic Sensing

3

Zhan, 2019

Distributed Acoustic Sensing

4

Zhan, 2019; Nayak et al., 2021

​

• Several thousand channels
• Several months recording
• Several meters spacing
• Much less field work

Distributed Acoustic Sensing

5

Zhan, 2019; Nayak et al., 2021

Distributed Acoustic Sensing

6

Zhan, 2019; Nayak et al., 2021

Ambient Noise Seismology + Distributed Acoustic Sensing

7

DAS data

• Big data

​

• Better cyber-infrastructure and formats

• Large-N

​

• Efficient tools

​

• Advanced computing infrastructure

Distributed Acoustic Sensing

8

Modified from Spica et al., 2023

Data volume rate: 10 to 1600 GB per day

6-7 years of DAS > 31 years of IRIS archive

DAS data format

9

Acquisition

Structure of a Sintela Onyx DAS HDF5 file

Attributes

RawData

(ncha, nt, int/float32)

nt: #time sample

ncha: #channel

DAS data format

10

Time

Channel

​

00:00:00

1 minute: 1 file (50 MB) [nCha, fs*60]

1 day: 1440 files (70 GB) [nCha, fs*60*60*24]

1 month: 44640 files (2.2 TB) [nCha, fs*60*60*24*31]

​

sub-array

DAS data format

11

• Self-describing hierarchical structure

​

• Support compression

​

• Allow user-defined attributes

​

• No upper limit of file size

• Complex structure on disks

​

• Hard for range request (sub-array request)

​

• Expensive (or prohibitive) to make the full copy of the data

​

Object Storage for DAS

12

1. Convert HDF5 into Cloud-optimized format

​

​

​

• Deploy MinIO as a local object storage

​

​

Cloud-optimized format

13

​

00:00:00

​

00:01:00

​

23:59:00

Time

Channel

······

data chunk

data chunk

data chunk

​

• Compression

​

• Key-value attributes

​

• Self-describing hierarchical structure

​

• Sub-array request

​

​

Chunking

14

Chunking

15

Small chunk

Large chunk

Optimize based on

• Data set
• Storage hardware
• Reading pattern

​

Object Storage

16

• Open-source

​

• Easy to deploy

​

• Good scalability

​

• Access control

​

• S3-compatible

17

Local Storage

Dept. B

Group B

Dept. C

Group C

​

Group A

Data Query

A PI level data server that serves the group and other departments, and other institutions.

18

Local Storage

Dept. B

Group B

Dept. C

Group C

Group A

A PI level data server that serves the group and other departments, and other institutions.

Data Query

Institution D

Institution E

Institution F

Remote

Local

19

Local Storage

Dept. B

Group B

Dept. C

Group C

Group A

Local

Cloud

……

Cloud Storage

Institution D

Institution E

Institution F

Remote

Local

A S3-compatible storage for developing and data sharing.

Data Query

20

Local Storage

Local

Cloud

Cloud Storage

Institution D

Institution E

Institution F

Remote

Local

S3 as a data center

Data Query

21

NoisePy for DAS

Application: SeaDAS-N

22

• UW Seattle - Bothell
• April 2022 – March 2023 (11 months)

​

​

• 2089 channels (4.78 m) @ 100Hz
• Raw data in minute-long HDF5 (72 GB/day)

UW - Photonic Sensing Facility

https://psf.uw.edu

23

NoisePy for DAS

2089x2090/2 = 2.18 million corr

Application: SeaDAS-N�Client to query data

24

subarray

600 channels

Application: SeaDAS-N �Cross-correlation on AWS Batch

25

Containerized NoisPy4DAS

One-bit, spectrum whitening

496 vCPUs, 12h

< $ 100 EC2

One-month SeaDAS-N

Chunked in Zarr format

~ $ 1.3 per day

Hourly stacking CCF

~ $ 1.8 per day

Local

Cloud

1 2 3 4 5 6

7 8 9 10 11 12

Job array

Application: SeaDAS-N �Cross-correlation on AWS Batch

26

• Virtual source gather
• Hourly linear stacking
• ~100 billion correlation

Application: SeaDAS-N �Cross-correlation on AWS Batch

27

• Virtual source gather
• Hourly linear stacking
• ~100 billion correlation

Thanks

Follow project on GitHub

https://github.com/niyiyu/DASstore

https://github.com/niyiyu/NoisePy4DAS-SeaDAS/

Hands-on Google Colab

https://colab.research.google.com/

File -> Open notebook

-> GitHub section

-> URL: https://github.com/niyiyu/NoisePy4DAS-SeaDAS/

-> select the example notebook

Hands-on through Docker on AWS EC2

1. Launch a EC2 instance in us-west-2 (Oregon) region, connect to the instance, and install required packages (just like yesterday).

​

2. Run the command in the instance to pull the image

sudo systemctl start docker

sudo docker pull ghcr.io/niyiyu/noisepy4das-seadas:latest

​

3. Run the Jupyter notebook in the instance

sudo docker run --rm -p 8888:8888 ghcr.io/niyiyu/noisepy4das-seadas jupyter notebook --ip 0.0.0.0

​

4. Go to the Jupyter notebook through your local browser.

Tips to launch instances

https://seisscoped.org/HPS/softhardware/AWS_101.html

Hands-on through Docker on local (anyone?)

​

1. Launch a EC2 instance in us-west-2 (Oregon) region, connect to the instance, and install required packages (just like yesterday).

​

2. Run the command in the instance to pull the image

sudo systemctl start docker

sudo docker pull ghcr.io/niyiyu/noisepy4das-seadas:latest

​

3. Run the Jupyter notebook in the instance

sudo docker run --rm -p 8888:8888 ghcr.io/niyiyu/noisepy4das-seadas jupyter notebook --ip 0.0.0.0

​

4. Go to the Jupyter notebook through your local browser.

​

Tips to launch instances

https://seisscoped.org/HPS/softhardware/AWS_101.html