Task definition strategy (aka Mixed campaign/sub-campaign containers)

D. Golubkov on behalf of ProdSys dev. team

1

DCC meeting, CERN, 4 October 2017

Intro

• Production System (ProdSys) splits input containers except “Simul” and “Evgen” steps
• Container can be used as input for “Simul” and “Evgen” tasks
• For each TID-dataset in the container the task is created
• Offsets are calculated using information about the previous tasks with the same configuration
• TID-datasets are used in series
• https://its.cern.ch/jira/browse/PRODSYS-799
• “In mc16, the datasets are mutually exclusive so each dataset must have campaign and sub-campaign attributes”
• mc15_13TeV.361026.Pythia8EvtGen_A14NNPDF23LO_jetjet_JZ6W.evgen.EVNT.e3569/
• _tid04904963 - campaign is None (in Rucio)
• _tid09471170 - campaign is MC15:MC15c
• _tid09822654 - campaign is MC15:MC15c
• In MC16 ProdSys should process datasets of different sub-campaigns separately

2

DCC meeting, CERN, 4 October 2017

Intro (Cont.)

• Datasets produced years ago have no campaign/sub-campaign metadata
• Doug made the ProdSys sub-campaign definitions
• https://twiki.cern.ch/twiki/bin/view/AtlasProtected/AtlasProductionGroupMC16#Prodsys_sub_campaign_definitions

3

• campaign=None can be interpreted as MC16a sub-campaign

DCC meeting, CERN, 4 October 2017

Implementation

• The first iteration is to use a special project_mode parameter to specify the campaign values of the input datasets
• runOnlyCampaign. For example, MC16:MC16a,None,MC15:MC15c
• The value should be the same as in Rucio metadata and specified by user
• The second iteration is automatic splitting of the container with mixed campaign and taking TID-datasets according to “Subcampaign” property of production request and ProdSys sub-campaign definitions table
• Enable for all MC16 requests
• First checking a special hashtag (associated with the task with corresponding TID)
• XXX_sc_102017_mixed_cont, where XXX - sub-campaign value
• If there is no hashtag then checking Rucio metadata
• Setting hashtag XXX_102017 automatically using ProdSys sub-campaign definitions table to prevent uncertainty with ‘None’ and easy future bookkeeping

4

DCC meeting, CERN, 4 October 2017

Implementation (Cont.)

• Changes in the algorithm to prevent incorrect offset calculation
• Additional protection for regular production: the containers with mixed campaign TID-datasets cannot be used as input without splitting
• ProdSys checks every TID-dataset in mixed campaign container and reject the task if it was used as input separately in the previous task with the same parameters

5

DCC meeting, CERN, 4 October 2017