1 of 5

Data Carousel R&D ideas

Alexei Klimentov (BNL), Mario Lassnig (CERN), Xin Zhao (BNL)

​

DDM weekly meeting, September 6th, 2022

1

2 of 5

Data Carousel next steps

  • Several areas as the next steps of the Data Carousel
    • Smart writing
    • Control tape write rates to sites
    • non-Data-Carousel tape staging
    • Long tail effects
    • …

(See this talk for more details)

  • Today we would like to focus on two concrete ideas, as potential demonstrator projects for Run4.

2

3 of 5

Tape Smart writing

  • Purpose
    • To demonstrate the smart writing mechanism and its effects in improving tape throughputs
  • Components
    • Metainfo provided by DDM/FTS when writing to tapes
      • Per file basis
      • Name of the dataset, size of the dataset (total volume, # of files), dataset status (open/ closed)
      • Metainfo can be extended to include container layer if applicable
    • Smart writing mechanism provided by sites
      • To group files on tape based on datasets/containers
  • Participants
    • DDM team (+ FTS)
    • FZK Tier-1 (+ dCache dev)
  • Metrics
    • Overall tape throughput (added after the meeting discussion)
    • Tape throughput per tape drive
    • Number of tape (re)mounts
  • Timeline
    • TBD

3

4 of 5

“Unpopular*” DAODs to tape

  • Purpose
    • Part of another demonstrator proposal (On demand data demo)
  • Similar to the short-lived DAODs to tape proposal
    • See slide 5 of this talk for details
    • Tried at PIC_SLTAPE, but didn’t get exercised thoroughly (many DAODs still on disk)
  • Participants
    • WFM/DDM
    • FZK Tier-1
  • Metrics (part of the metrics of the On demand data demo)
    • User tasks using DAODs from tape directly
    • User tasks with DAODs (re-)produced on the fly, which requires staging AODs from tape
    • Load on tape systems
      • Tape bandwidth requirements (to stage DAODs vs to stage AODs)
    • Total volume of such unpopular DAODs (ie. disk space saved) (added after the meeting discussion)
  • Timeline
    • After the next lifetime model campaign (in 1~2 months) ?

4

*) Unpopular == DAOD from lifetime model exception list

5 of 5

  • The above two demonstrators may be carried out in one exercise, e.g. to use the short-lived DAODs as data sample for testing the smart writing mechanism, but they are in principle two separate topics, with their own timeline and metrics.
  • If any other sites want to join these R&D, they are more than welcome!

​

5