1 of 4

Testing PanDA GPU queues

USATLAS Facility Coordination meeting

1 December 2021

Johannes Elmsheuser (BNL)

USATLAS Facility Coordination meeting, 1 December 2021

2 of 4

Introduction

  • The following is a test as a “naive user” through PanDA
  • Athena,master nightlies contain cuda and cudnn since a couple of weeks - I was curious to test a small “GPU burner” script at all available GPU PanDA queues, since also the local cuda installation differs at various sites
  • Available GPU PanDA queues:

​

ANALY_BNL_GPU_ARC : test

ANALY_INFN-T1_GPU : brokeroff

ANALY_MANC_GPU : online

ANALY_MWT2_GPU : online

ANALY_OU_OSCER_GPU_TEST : test

ANALY_QMUL_GPU : test

ANALY_SLAC_GPU : online

DESY-HH_GPU : online

GOOGLE_GPU : brokeroff

​

​

​

2

USATLAS Facility Coordination meeting, 1 December 2021

3 of 4

Job submission

  • Job submission:

prun --exec="./run.sh" --outDS user.elmsheus.gputest.0090 --outputs=my-outputs.tar.gz --disableAutoRetry --noBuild --extFile=gpu-burn.tar.gz,run.sh --site ANALY_BNL_GPU_ARC –architecture …

  • Only succeeded to run jobs at ANALY_MANC_GPU and GOOGLE_GPU
  • All other jobs failed in job brokering or went into exhausted status after 2-3 days: see https://bigpanda.cern.ch/user/?user=elmsheuser
  • User.elmsheus.gputest.0070 - 0078 submitted with --architecture "&nvidia-gpu" - only MANC and GOOGLE worked
  • User.elmsheus.gputest.0080 - 0088 submitted with --architecture "&gpu" - some working task brokering for BNL, OU_OSCER, QMUL, but the tasks went into closed/exhausted status later - only MANC and GOOGLE worked
  • user.elmsheus.gputest.0090 - 0098 submitted without --architecture - some working task brokering for SLAC and DESY-HH but again the tasks went into closed/exhausted status later - only MANC and GOOGLE worked
  • Overall there might some queue definitions/configurations missing in CRIC for some queues

3

USATLAS Facility Coordination meeting, 1 December 2021

4 of 4

Some items

  • Are the US GPU PanDA queues working ?
  • Overall not much GPU PanDA queue usage (see right plot) since they are not accessible or not known ?
  • I did not and will not test in the near future any GPUs interactively at AnalysisFacilties

​

  • N.B. there is at present no change in accounting towards pledges for GPUs, as far as I know

4

USATLAS Facility Coordination meeting, 1 December 2021