HS³

^{High Energy Physics �Statistics Serialization Standard}

Carsten Burgard

Tomas Dado, Jonas Eschle, Matthew Feickert, Cornelius Grunwald, Alexander Held, Robin Pelkner, Jonas Rembser, Oliver Schulz

1

Stating the problem

• RooFit has been an extremely successful backend for statistical modeling for the last two decades
• The RooWorkspace is the de-facto standard format for exchanging statistical models
• for example: CMS+ATLAS combination efforts
• also used for CMS+LHCb

2

RooWorkspace

Stating the problem

• The community is larger than just the LHC Experiments
• Phenomenologists outside the large collaborations
• Astroparticle experimentalists
• …
• Not everyone uses ROOT
• not even in atlas: pyhf!

3

3

?

?

?

?

?

RooWorkspace

Publications as a channel of communication

• As a science community, we communicate our results via publications
• We have generally agreed to also publish numbers shown in plots & tables in a machine-readable form
• The numbers we publish are typically far away from being a complete representation of our statistical model
• Most often choose to publish “simplified representations”
• most common: multivariate gaussian models

4

4

Simplified Models: Multivariate Gaussians

• The simplest method: Multivariate Gaussians

5

5

covariance matrix

correlations

uncertainties

central values

Publications as a channel of communication

• As a science community, we communicate our results via publications
• We have generally agreed to also publish numbers shown in plots & tables in a machine-readable form
• The numbers we publish are typically far away from being a complete representation of our statistical model
• Most often choose to publish “simplified representations”
• most common: multivariate gaussian models
• Theorists repeatedly ask for more complete representations
• if we actually publish full models, they use them happily!
• decision to publish likelihoods was made a long time ago

6

6

A long journey

7

7

Massimo Corradi

​

It seems to me that there is a general consensus that what is really meaningful for an experiment is likelihood, and almost everybody would agree on the prescription that experiments should give their likelihood function for these kinds of results. Does everybody agree on this statement, to publish likelihoods?

​

Louis Lyons

​

Any disagreement? Carried unanimously. That’s actually quite an achievement for this Workshop.

this slide was stolen from Lukas Heinrich & Kyle Cranmer

pyhf and its influence on likelihood publishing

• text-models have been published on HEPdata
• only for pyhf models so far
• possible for ROOT workspaces since very recently
• these results have been enthusiastically embraced by theorists

8

8

Paper combining full likelihoods published by ATLAS with simplified models published by CMS

Look out for this button here!

There is significant tooling available around pyhf & JSON:

• cabinetry: build & use pyhf models
• workspaceexplorer: examine pyhf models
• also a julia version available: LiteHF.jl

Even though pyhf is not the right tool for every analysis, the documentation quality is excellent and it’s a great tool for simple & quick analyses!

pyhf: A success story

• Since about 2018, pyhf exists as a pure-python implementation of the HistFactory modeling framework
• very useful for stacks of histograms
• somewhat limited in scope
• adopted by large parts of ATLAS (mostly SUSY+Exotics)
• Models can be written down descriptively as JSON files
• resounding adoption by the theory community

9

9

Purely descriptive, static model representations are extremely useful

The common problem(s)

• All of the approaches taken so far suffer from several shortcomings

• Bound to a specific piece of software
• ROOT, pyhf, combine, …
• What about version compatibility in 1y (5y, 20y)?
• What about dependencies & their versions?
• Limited versatility
• phyf: only histogram stacks
• ROOT: no purely descriptive, human-readable representation
• Functionality often opaque
• What does the code do in a specific case vs. what should it do (in principle)
• How well documented is the descriptive representation

10

10

What we want

• A descriptive, domain-specific language for serializing models, and it should be
• feature-complete with respect to ROOT (and all other common frameworks)
• human-readable (at least in principle)
• machine-readable with little effort
• well documented to the point where someone implementing a new tool can reproduce the functionality with just the documentation
• Ideally, it should be suitable to
• exchange models between analysts inside collaborations & outside
• capable of being published on HEPdata or similar places
• usable and modifiable by outsiders with little to no domain knowledge
• useful for outreach and teaching activities
• carry the legacy of the LHC experiments & their publications

11

11

Where to learn from

• pyhf
• descriptive
• human-readable & editable
• excellent documentation, mostly mathematical
• ROOT
• RooWorkspace de-facto standard & rich in features

12

12

If we can have the features of a RooWorkspace

convertible to and from ROOT

in a format that resembles pyhf JSON

with similar quality documentation,

we’re basically there!

ROOT Workspaces

• statistical models are computational graphs
• directed graphs: no cyclic dependencies
• leaves are variables
• intermediate nodes are operations
• +,-,*,/,...
• higher-level nodes allow to simply the graph
• exp, Gauss, Poisson, …

13

13

computational graph of a Higgs combination workspace

this graphic was stolen from Wouter Verkerke

ROOT Workspaces ⇔ JSON Files

• can easily render dependency graph as text file
• use JSON, since format is well-received so far
• graphs can be very large
• what can we do to keep them simple?
• … other than build simpler models
• common patterns can be abstracted as high-level nodes
• e.g. HistFactory bin prediction
• work together with ROOT developers to make sure JSON representation is fully isomorphic to ROOT Workspace

14

14

JSON representation of an ATLAS measurement

mix of HiFa & non-HiFa components

PreprocessFunctions & more

“ModelConfig”

obs, asimov, toys, …

pdfs

param. ranges

functions

connecting data+model

Top-level components

• distributions, functions: mathematical model
• data: observations and toy data
• likelihoods: defining loss, constraints
• parameter points, domains: info on parameters
• analyses: string everything together, identify POIs for interpretations
• metadata: version numbers and required packages
• misc: custom information dump

15

15

likelihoods

domains

parameter�points

analyses

metadata

misc

this graphic was stolen from Robin Pelkner

Example: Distributions

16

16

definition in the standard

actual usage in JSON

More examples

17

17

All of these are suggestions, �feel free to suggest improvements!

This is not a pipe-dream!

• Need to make sure the standard works for everyone
• get on board major tool developers
• exercise roundtrips with complicated models
• obtain & incorporate suggestions
• So far, reasonably successful
• fully working implementation in ROOT
• also available in combine!
• BAT.jl implementation is WIP
• pyhf+zfit have vowed to implement

18

18

take a look at the overleaf or github to view the current draft!

It’s alive: Higgs discovery workspaces from ATLAS

• export of the combined ATLAS Higgs discovery workspace from 2012
• an important document of science history
• a landmark success: full feature-completeness with respect to the requirements of the ATLAS Higgs group
• Roundtrip closure within ROOT
• pushing for a release of these workspaces in the JSON format
• ROOT team would be happy to use them as a unit test
• will ensure long-term stability of the most important features
• Draft implementation in julia
• not efficient enough yet to run the full closure test (sadly)
• closure on individual components already successful
• Bidirectional conversion from HS³ JSON to pyhf JSON exercised
• only works for the pure HistFactory components, but is an option!

19

19

analyses

• if you know RooFit:
• analysis = ModelConfig
• contains inference suggestions
• list of POIs
• parameter domains (ranges)
• point to likelihood
• optional element

20

20

you can always leave it to the user to come up with descriptions of what they want to do

likelihoods

• list of likelihoods in the model
• each one identifies
• list of distributions (named)
• list of datasets (named)

21

21

data

• one entry per dataset
• one per “channel” or “region”
• binned and unbinned supported
• regular and irregular binnings
• only rectangular for now
• use axes to identify dimensions
• arbitrary dimensionality
• name-matching to observables in pdfs

​

22

22

distributions

• one entry per pdf
• “type” encodes the type of Pdf
• definition + explanation found in the standard document (evolving)

23

23

Please take a look at the standard and find whether any of the distributions important to you is not yet included

Long term, we strive to include all reasonably frequently used pdf types �in the standard!

distributions

• one pdf type called histfactory_dist
• close resemblance of current pyhf JSON
• a few core differences
• parameter names (modifiers)
• constraint types
• observable names (axes)
• difference in data syntax for modifiers
• observed data not included�this is done in a separate section
• bidirectional conversion ot pyhf JSON possible (converter script exists)
• changes planned for the future:
• will (re)move constraint type and replace with pdf constraint name

24

24

functions

• very similar to distributions
• not self-normalizing
• allowed to yield negative numbers
• auxiliary objects
• not needed for many models
• main use case: reparametrization
• arithmetic functions to transform variables (product, sum, …)
• special feature: generic_function
• needed to cover RooFormulaVar
• simple arithmetic transformations
• use should be avoided if possible

25

25

parameter_points

domains

• store parameter configurations
• needed for some inference techniques
• convenience feature
• starting points for minimizations
• best-fit-values for easy reruns
• true parameter sets for toy data
• …
• maps to snapshots in RooFit

• store parameter ranges
• needed for some bayesian inference �techniques
• useful for closure with RooFit

26

26

metadata

• track model provenance
• who created this model?
• when?
• how?
• which packages were used?
• will be extended in the future
• include HEPdata fields?
• …?
• heavily formalized
• suggestions welcome

27

27

misc

• additional field to store arbitrary, possibly framework-related infomation
• tags for categorization of objects
• pretty-print titles
• plotting colors
• …
• contents are not standardized
• must not affect the result of inference�in any way

28

28

anything that is nice-to-have but not standardizable enough to fit in metadata goes here!

Top-level components

• distributions, functions: mathematical model
• data: observations and toy data
• likelihoods: defining loss, constraints
• parameter points, domains: info on parameters
• analyses: string everything together, identify POIs for interpretations
• metadata: version numbers and required packages
• misc: custom information dump

29

29

likelihoods

domains

parameter�points

analyses

metadata

misc

this graphic was stolen from Robin Pelkner

Conclusions

• HS³ is a new and developing standard to serialize statistical models
• isomorphic to RooWorkspace, fully implemented in ROOT
• ROOT 6.30/00 implements HS³ v 0.2 (to be released in the next days)
• adoption by the community slow but steady
• It should cover a wide range of requirements
• useful for publishing full statistical models of combined analyses
• useful for long-term storage of LHC legacy results
• useful for outreach and teaching statistics
• We’re open for feedback and input of any type
• If you think we’re headed down the wrong direction, tell us now and tell us why!
• Please read our draft and provide feedback on our github & participate in discussions!

30

30