1 of 25

ga4gh.org

2 of 25

DRS - Cold Storage and Bulk Requests

Co-Chairs, Ian Fore, Michele Mattioni, Brian O’Connor

ga4gh.org

3 of 25

Cold Storage

ga4gh.org

4 of 25

Drivers for cold storage

NCBI dbGaP/SRA

  • Uneconomical to store raw formats (fasta, bam, cram) for all projects

​

Other driver projects?

  • Poll for needs/interest

​

​

ga4gh.org

5 of 25

Typical SRA/dbGaP file storage tier strategy

file

type

Size bytes

location

state

17_SR.sorted.bam

bam

77,815,229

s3

cold

SRR1999488

sra

27,930,067

s3

hot

SRR1999488.lite

sra

12,037,768

ncbi,gs,s3

hot

Files for a sequencing run

ga4gh.org

6 of 25

Balancing factors

File Format

Tool capability

Storage tier

Platform flow

Service

ga4gh.org

7 of 25

Related factors

File format

Cram, bam, fasta, sra

Protocol

DRS, htsget, refget, etc

Platform flow

File transfer to working directory

Tool capability

Do tools work with files or protocols?

​

​

ga4gh.org

8 of 25

SRA sam-dump

Converts SRA format

BAM

FASTA

ga4gh.org

9 of 25

Current NCBI DRS response for a file in cold storage

https://locate.ncbi.nlm.nih.gov/ga4gh/drs/v1/objects/f8c6e53425fe82061f792c79c7764c33

​

status_code:409

​

{

"msg":"The file has been moved offline.",

"Status_code":409

}

ga4gh.org

10 of 25

Workaround at client level - simulated response

https://locate.ncbi.nlm.nih.gov/ga4gh/drs/v1/objects/f8c6e53425fe82061f792c79c7764c33

​

status_code:200

​

{� "access_methods": [� {� "access_id": "1",� "region": "unknown",� "type": "https",� "storage": "cold"� }� ],� "id": "f8c6e53425fe82061f792c79c7764c33",� "self_url": "drs://locate.ncbi.nlm.nih.gov/f8c6e53425fe82061f792c79c7764c33"�}

​

​

​

ga4gh.org

11 of 25

Proposal: support attribute on access_method

{ "access_methods": [� {� "access_id": "e93724",� "region": "ncbi",� "type": "https"� },� {� "access_id": "fbd466",� "region": "gs.US",� "type": "https",� "storage": "cold"� },� {� "access_id": "0da151",� "region": "s3.us-east-1",� "type": "https"� }� ],� "checksums": [{ "checksum": "044e759c2e430c3db049392b181f6f5a", "type": "md5"} ],� "created_time": "2022-06-03T14:07:27Z",� "id": "044e759c2e430c3db049392b181f6f5a",� "name": "SRR000066.lite",� "self_url": "drs://locate.ncbi.nlm.nih.gov/044e759c2e430c3db049392b181f6f5a",� "size": 118588310�}

​

​

See DRS issue 395

ga4gh.org

12 of 25

Key questions

Mapping to the DRS model

DRS Object or Access Method

Optional attribute

I.e. only when needed for cold

Indicating availability

Binary - hot or cold

Accommodating different tiers

Tiers are cloud provider specific

Indicate service level e.g. time to make available

How long should a thawed file remain hot?

​

​

​

ga4gh.org

13 of 25

Factors in supporting cold storage

NCBI Cloud Data Delivery Service (CDDS)

  • Delivery from NCBI cold storage buckets
  • To consumers' buckets in AWS and GCP
    • Target bucket
  • Public and authorized access (dbGaP) data

​

Download and hot storage at user cost

Admin access on consumer bucket must be granted to CDDS

​

https://www.ncbi.nlm.nih.gov/sra/docs/data-delivery/

​

​

ga4gh.org

14 of 25

NCBI Cloud Data Delivery Service

SRA accession (run) based not file based

Accession-to-file (drs_id) changes over time

​

​

ga4gh.org

15 of 25

NCBI Cloud Data Delivery Service - API methods

Workorder status

Email, workorder id

Workorder History

Email

Submit workorder

Email, target bucket

Verify target bucket (has it been configured for access)

Email

​

​

​

Largely cloud provider agnostic

ga4gh.org

16 of 25

Cloud Data Delivery Service

Cloud provider specific Access Management

​

Accession (run) based not file based

​

Issues with granting required access

​

​

​

ga4gh.org

17 of 25

Platform flow - hot storage

NCBI bucket

NCBI DRS

Seven Bridges bucket

samtools

Seven Bridges bucket

File path

transfer

Task environment in Cavatica

User’s project in BioData Catalyst

Seven Bridges DRS

download

Signed URL

WES

User script

ga4gh.org

18 of 25

Platform flow - direct transfer to target bucket?

NCBI bucket

cold

NCBI DRS

User script

samtools

Seven Bridges bucke�hot

File path

transfer

Task environment in Cavatica

WES

ga4gh.org

19 of 25

Bulk Operations

ga4gh.org

20 of 25

Bulk Operations

  • Bulk operations has been a frequent request for DRS to help scale�
  • Use case: systems that process thousands of files in DRS 1.3 must make multiple calls in order to resolve an object → signed URL or equivalent
    • Bulk operations doesn’t necessarily replace multiple calls, but the calls can be done for multiple objects in parallel
    • To keep things simple, we did not include pagination but focused on bulk calls and allowed the client/implementation to chunk requests

​

​

Multiple object IDs

Multiple access methods

Multiple signed URLs

ga4gh.org

21 of 25

Status of Bulk Operations in DRS 1.4

  • PR created
    • “Ready” to merge PR 365 (length of bulk_objects_ids in service info)
  • DRS Release
    • DRS 1.3 (last release)
      • OPTS for providing authorization information (Passports)
    • DRS 1.4 (develop)
      • Bundle documentation/best practice recommendation
      • Bulk operations
    • DRS 1.5
      • Come to the meeting at 4pm Pacific today!
        • “DRS v1.5 road-mapping and discussion”
        • Room of the Dons
        • 4:00 PM - 5:30 PM (PDT)

ga4gh.org

22 of 25

Status of DRS 1.4 in the Starter Kit

  • Issue created during eLwazi GA4GH meeting (July)
    • See issue 79
  • Feature Branch

​

​

ga4gh.org

23 of 25

ga4gh.org

24 of 25

ga4gh.org

25 of 25

ga4gh.org