1 of 11

Status of L1/L2/L3 Submissions

N. Bates, W. Feng,

E. Strohmaier, and T. Scogland

SC 2021 Green500 BoF

synergy.cs.vt.edu

2 of 11

Status of L1/L2/L3 Submissions

N. Bates, W. Feng,

E. Strohmaier, and T. Scogland

SC 2021 Green500 BoF

synergy.cs.vt.edu

3 of 11

Submission Types and Status

Power Source

Measured (182)
Derived (318)

Power Quality Level

Level 1 (139)
Level 2 (28)
Level 3 (14)

(June2021 List entries)

Just like the Top500 List, the Green500 List has 500 entries. The power numbers listed for those 500 entries are not, however, all measured power submissions, some of the power numbers are derived. There is an excel spreadsheet that you can download for the Green500 List. One of the columns in that spreadsheet is called Power Source and it identifies whether or not a submission was measured or derived. For the June2021 List, there were 182 entries with measured submissions and 318 with derived submissions.

For those entries with measured submissions, there are differences in the rigor of the methodology used for making the measurement. There are three levels of rigor, Levels 1-3 with three being the most rigorous. The Level information is also found in the excel spreadsheet under a column called Power Quality Level. For the June2020 List, there were 139 Level 1 entries, 28 Level 2 entries and 14 Level 3 entries.

4 of 11

Differences Between Levels?

	Level 1	Level 2	Level 3
Machine fraction	Largest of: 2 kW 1/10 of the system 15 nodes	Largest of: 10 kW 1/8 of the system 15 nodes	Whole system
Subsystems included	Only compute and network required, estimates allowed	All participating subsystems, estimates allowed	All participating subsystems must be measured
Meter accuracy	Minimum 5%	Minimum 2%	Revenue grade
Measurements to report	Average power, core phase	Average power, full run	Energy, full run

You might ask, why the different levels and what are the differences?

When the EE HPC WG started working with the Green500 on improving their Power Measurement Methodology, we wanted to have a high-bar. To set an example of what ‘current best’ practices allowed. This was what led to the Level 3. Level 1 was generally available to any submitter with a minimum of cost and effort. Level 2 was in-between. The criteria for these different levels has changed over the years based on submitter feedback and continuous improvement. It has also changed because of architectural trends in supercomputers over the past decade. The power draw of the network relative to the compute system, for example, was negligible a decade ago. The network power was not required for Level 1 in the first iteration of the methodology. Increases in the power draw of the network relative to the compute system is no longer negligible. So, the network power is now required for all three levels.

I am going to give a high level description of the differences between levels 1, 2 and 3. One parameter is the fraction of the compute subsystem that is measured. A second parameter is what subsystems are included and how they are measured. The third parameter is meter accuracy and, finally, the fourth is what measurements are reported.

To give an example, the first row is the fraction of the machine that is measured. For level 1, that is the largest of 2kW, 1/10 of the system or 15 nodes. The entire system power is then extrapolated from that fractional measurement. This is most contrasted with a level , which requires that the entire system is measured. Level 2 is in-between level 1 and level 3.

The other rows have similar distinctions between the levels.

5 of 11

Why Make L2/L3 Submission?

More accurate and precise information
Ability to measure system-level power & energy

Some use-case examples:

Architectural trending & system modeling
Procurement & data-center provisioning
Operational improvements
Validate component-level measurement

Why make a L2 or L3 Submission? Besides providing more accurate and precise information for the Green500 List, there are other reasons to make a L2/L3 submission. There are a lot of capabilities today for measuring component level power. This is especially true for processors and accelerators, but it also includes node and rack level measurements. System level power is still a challenge for many sites and systems. Sites may want to do architectural trending and system modeling for design, selection, upgrades, tuning and analysis. Another example would be for understanding the actual – verses peak – power draw of a site’s HPC system based on a normal site workload. This might be used for provisioning the datacenter as there can be a substantial difference between peak and normal power draw. Operational improvements, such as for cooling systems, can leverage the ability to measure HPC system-level power and energy. Finally, system-level measurements can be used to validate component level measurements by summing them up.

6 of 11

Los Alamos National Laboratory (LANL)

Level 3 measurements encouraged diverse organizational teamwork
Level 3 measurements laid the groundwork for future green monitoring

Swiss Supercomputing Center (CSCS)

It is always good to have reliable information about your data center
Doing a reliable Level 3 measurement is not harder than Level 2 or Level 1

RIKEN

While doing the Shoubu System B Level 3 measurement, the submission team realized an opportunity for optimizing their cooling sub-system

Why Make a Level 3 Submission?

(Feedback from Previous Green500 BoFs)

More specifically, why a Level 3 and not a Level 2? Here are some answers from Los Alamos National Laboratory, the Swiss Supercomputing Center and RIKEN. One reason is that taking a Level 3 measurement often requires people from diverse organizations to come together to make the measurement and that this cross-fertilization is valuable in and of itself. In fact, it is a critical organizational capability for improving operational efficiency. Another reason to take a level 3 measurement is that it is easy to do. In fact, as we mentioned earlier, the ability to make an accurate system level measurement is a necessary capability for data center optimization. Finally, sometimes the data is unexpected and revealing, so taking a Level 3 measurement can allow for immediate optimization opportunities.

7 of 11

All L2/L3 Submissions by Site

AIST
AWE
Calcul Québec/Compute Canada
CSC (Center for Scientific Computing)
CEA/TGCC-GENCI
Commissariat a l'Energie Atomique (CEA)
Facebook
Forschungszentrum Juelich (FZJ)
Fujitsu Numazu Plant
HLRN at ZIB/Konrad Zuse-Zentrum Berlin
Joint Center for Advanced HPC
Lawrence Livermore National Laboratory
MIT/MGHPCC
National Supercomputing Center in Wuxi
Sandia National Laboratories
SENAI CIMATEC

Lawrence Livermore National Laboratory
Los Alamos National Laboratory
Oak Ridge National Laboratory
Rensselaer Polytechnic Institute (RPI)
Sandia National Laboratories (SNL)
Swiss National Supercomputing Centre, CSCS
Technische Universitaet Darmstadt
RIKEN
Preferred Networks
NVIDIA Corporation
Science and Technology Facilities Council
Universitaet Mainz
University of Tokyo

(through June2021

Green500 List)

Lawrence Berkeley National Laboratory
Technical University Dresden
Deutscher Wetterdienst
Japan Agency for Marine & Earth Science & Technology

University of Florida
Purdue University
Cyfronet
New York University

New November 2020-Present

8 of 11

L2/L3 Submissions by Vendor**

(Based on June2021 Green500 List)

* Combined vendors count twice

** 1 Self-made not listed

Since 2016	Past Year (June)
Fujitsu (7)	4
Atos (6)	1
NEC (5)	4
Penguin (5)	1
HPE (5)	2
NVIDIA* (4)	3
IBM* (3)	0
Dell (2)	1
Megware* (2)	1
Preferred Networks (1)	1
Huawei (1)	1
Lenovo (1)	1
NRCPC (1)	0

9 of 11

Gaining Traction

(through June2020

Green500 List)

June 2018	June 2019	June 2020	June 2021
23	26	31	42

Total number of Level 2 and Level 3 entries is growing

10 of 11

What Needs to be Improved in the L2/L3 Methodology and Submission Process?

Swiss National Supercomputing Centre (CSCS)

[Submission] Only use only L3 measurement. “No harder than L1 or L2 …”
[Submission] Need a better way to provide a report and supporting files. The free form box is insufficient.

Fujitsu Numazu Plant

[Methodology] L3 too difficult from an infrastructure perspective. L2 good.
[Submission] Make power reporting mandatory for every system.

Los Alamos National Laboratory (LANL)

[Methodology] Useful document on L2/L3 but intimidating for first-time users
[Methodology] Need contact info for quick questions or detailed discussions
[Methodology] Need a list of known metering/measurement equipment

LANL needed to contact vendors to ensure that meters met the requirements

Technical University Dresden

[Methodology] Add ISO/IEC 62053-21, drop the internal sampling rate requirements, clarify aggregation of multiple measurements

11 of 11

Your thoughts?
EE HPC WG will kick-off effort to revise methodology, everyone’s participation is welcome

What’s Next?

Discussion

Thank you!

http://eehpcwg.llnl.gov

natalie.jean.bates@gmail.com