1 of 13

Demystifying In-band/Out-of-band Methods for Hyperscale Fleet Management

BO - Title Slide

2 of 13

Demystifying In-band/Out-of-band Methods for Hyperscale Fleet Management

George Kola, Senior Staff SW Engineer, Google

Anil Agrawal, HW Systems Engineer, Meta

Future Technology Initiatives

BO – Title/Presenter Slide

3 of 13

Agenda

  • Hyperscale Fleet Deployment Models
  • Hardware Management Scope
  • In-band/Out-of-band Management Methods
  • Q&A

​

BO - Content

4 of 13

Hyperscale Fleet Deployment Models

Hyper Scaler ownership varies based on the Deployment Model

Hardware Management Methods Varies

(In-band and Out-of-band)

Customer

Hyper Scaler

End User

Operating System(OS)

Firmware

Hardware

Services

Applications(SW)

Bare-Metal

Hypervisor (optional)

Customer

Hyper Scaler

End User

Operating System(OS)

Firmware

Hardware

Services

Applications(SW)

Infrastructure

Hypervisor

Hyper Scaler

End User

Operating System(OS)

Firmware

Hardware

Services

Applications(SW)

Application

Hypervisor (optional)

∴

5 of 13

Hardware Management Scope

System Configuration

(E.g., FW Upgrades/downgrades, etc)

Hardware Management

System Monitoring

(E.g., FW versions, PCIe device inventory, sensors, etc)

Hardware Fault Management

(e.g., Core, Cache, memory, PCIe, internal error handling, etc)

Keeping the Scope of Discussion 🡪 System Monitoring and Hardware Fault Management

6 of 13

In-band and Out-of-band Management

In-band

In-band

Out-of-band

Out-of-band

Hyperscaler Management Infra

In-band

IPMI/Redfish

​

On-Platform

Off-Platform

In-band

IMA ←→ Devices, BMC

IMA ←→ HS Infra

Out-of-band

BMC ←→ Devices

BMC ←→ HS Infra

IMA: In-band Management Agent

IPMI: Intelligent Platform Management Interface

MCTP: Management Component Transport Protocol

HS: Hyper Scaler

7 of 13

In-band/Out-of-band Management Methods

Auto-remediation

System Monitoring

IMA & BMC are complementary

System-level

Aggregator

Unified model

Hyperscale Storage

Hyper Scaler Management Infrastructure

In-band

Out-of-band

8 of 13

In-band/Out-of-band Management Methods

BO - Content

Hardware Fault Management

A Hybrid Method - Provides Flexibility in Hardware Fault Management

In-band FW

OS/Drivers

Applications/ Services

Error Event/

RAS Action

IMA

In-band, OS-First

In-band, FW-First

Method

Usage

In-band, OS-First

  • Higher rate of errors handling(e.g., Corrected Errors)

In-band, FW-First

  • Failure cases where OS-First may not be successful (e.g., Uncorrected Errors)

Out-of-band

  • Failure cases where In-band methods may not be successful (e.g., Catastrophic errors)
  • Required for ‘Bare-metal’ Deployment

Hyper Scaler Management Infrastructure

Host System

CPU, Memory

BMC

Out-of-band

Devices

9 of 13

Summary

  • Challenges in using a single method for Hardware Management (Monitoring and/or Fault Handling)
  • Out-of-band HW Fault Management Requirements - Currently driven by “OCP HW Fault Management Sub-project” team
  • In-band HW Management Requirements - Currently driven by CSM workstream

In-band Management

Not Feasible

Selectively Desirable

Selectively Desirable

Out-of-band Management

Highly Desirable

Selectively Desirable

Selectively Desirable

Customer

Hyper Scaler

End User

Operating System(OS)

Firmware

Hardware

Services

Applications(SW)

Bare-Metal

Infrastructure

Customer

Hyper Scaler

End User

Operating System(OS)

Firmware

Hardware

Services

Applications(SW)

Hypervisor

Application

Hyper Scaler

End User

Operating System(OS)

Firmware

Hardware

Services

Applications(SW)

Hypervisor (optional)

Hypervisor (optional)

10 of 13

Questions To the Invited Guests

Panos Christeas, Meta

Drew Walton, Google

Future Technology Initiatives

BO – Title/Presenter Slide

11 of 13

Questions

  1. Have you deployed in-band or out-of-band or both methods in the fleet?

​

  1. Moving forward, any preferences in using in-band and out-of-band methods?

​

12 of 13

Call to Action

  • Engage with “OCP - Cloud Service Model” workstream to participate in defining “In-band Hardware Management” requirements
  • Participate in HW Fault Management Sub-project working meetings to define “Out-of-band HW Fault Management” requirements.

BO - Content

13 of 13

Open Discussion