1

Geoffrey Fox , Indiana University, gcfexchange@gmail.com

​

​

SBI: Surrogate Benchmark Initiative

FAIR Surrogate Benchmarks Supporting AI and Simulation Research

​

​

​

Digital Science Center

SBI Meeting March 22-2021

​

​

Status of work at IU

• Continued good progress with MLCommons -- Science WG expects to have 4 Benchmarks -- probably one from Indiana on Time Series-- by July for SC21 results
• No surrogate benchmark chosen
• FAIR metadata talk to MLCommons April 6 by Christine Kirkpatrick SDSC to MLCommons Benchmark/Infrastructure working group
• Evaluating and Extending Benchmark Technology from MLCommons and UK

2

Digital Science Center

SBI Meeting March 22-2021

​

​

Science Data MLCommons working group

• Science like industry involves edge and data-center issues, end-to-end systems, inference, and training, There are some similarities in the datasets and analytics as both industry and science involve image data but also differences; science data associated with simulations and particle physics experiments are quite different from most industry exemplars
• When fully contributed, the benchmark suite will cover (at least) the following domains: material sciences, environmental sciences, life sciences, fusion, particle physics, astronomy, earthquake and earth sciences, with more than one representative problem from each of these domains

​

3

• One aim is to provide a mechanism for assessing the capability of different ML models in addressing different scientific problem
• Divide problems into classes and try to cover rich range of classes
• “End-to-end” is one class
• Provide common environment to store and run benchmarks
• https://mlcommons.org/en/groups/research-science/
• Surrogates Included

Digital Science Center

SBI Meeting March 22-2021

​

​

4

• Technical Specification: An open link is https://mlcommons.github.io/mlcube/
• Vibhatha Abeykoon, Indiana University looking at MLCube for EM-Noise benchmark. Contrast Docker and Singularity https://docs.google.com/presentation/d/19XAtMRPXXZG6DCzSjVzlJMqkkCGXbhTPGWOXTvsQ_Rg/edit?usp=sharing

Digital Science Center

SBI Meeting March 22-2021

​

​

General Issues

• Google-group https://groups.google.com/g/sbi-fair
• Website https://sbi-fair.github.io/
• Directory from proposal writing DOE_FAIR2020-Surrogates
• Directory for this proposal Afteraward
• Need to help set up outreach with working groups as in proposal; MLCommons working groups have some value here
• We need discussion groups?
• Surrogates
• FAIR
• Surrogate Software and Benchmark software
• Suggest best initial surrogate benchmark
• Need to implement SBI repository and look at MLCommons metadata wrt FAIR principles

5

Digital Science Center

SBI Meeting March 22-2021

​

​