1 of 22

OPEN SYSTEMS FOR AI

Overview for Plenary Session at OCP Global Summit 2024

Community-driven hyperscale innovation for all

2 of 22

Tackling the Next DC Technology Shift

Challenges:

Hyperscale DC Required

Scale and efficiency of IT and physical DC through openness
Flexible disaggregation and SD-x systems management value on open hardware

At-scale AI Clusters

Specialization

Economics at Scale

Speed to Market

Sustainable

Connect. Collaborate. Accelerate.

Community-driven hyperscale innovation for all

AI and HPC workloads are also distributed and diverse, much more so than other virtualized and cloud-native applications. In the pursuit of greater performance, infrastructure specialization for some workload categories is burgeoning, but risks fracturing the supply chain into silos, raising costs and reducing efficiencies. This is exacerbated by the accelerated drive for innovation to meet new requirements in the hardware ecosystem and rush for all players in the market to provide solutions.

�To prevent these downsides, the OCP facilitates open-sourcing hardware specifications and standardizing building blocks that are forged for the most demanding hyperscale solutions. The OCP’s trusted IP model allows multiple vendors to safely collaborate and specialize their products with greater interoperability and composability which benefits those vendors as well as the customers, moving the entire market forward. Spanning the OCP Projects, the OCP “Open Systems for AI” Strategic Initiative brings together the work of the Community to meet AI's scale and workload diversity.

�In summary, as open-source and standardized systems improve economies and efficiencies for hyperscale data centers and in turn data centers more generally, they do so even more so for AI clusters’ requirements of even more massive resource scale and varying infrastructure needs.

3 of 22

Open Systems for AI

Determine commonalities from�at-scale AI challenges

Open source & standards�enable multi-vendor supply chain

Improve sustainability

Drive efficiencies

Connect. Collaborate. Accelerate.

Community-driven hyperscale innovation for all

4 of 22

Open Systems for AI Strategic Initiative

Open Systems for AI – an umbrella OCP Strategic Initiative

WORK AREAS

COMMUNITY WORK

ADOPTION & ECONOMY

Execution in�OCP Project workstreams

Contributions in OCP database

OCP Marketplace:

chiplets, products, solutions

and OCP Ready facilities

Systems Management

DC IT Infrastructure

DC Physical Infrastructure

+ OCP Future

Technologies Initiative

Connect. Collaborate. Accelerate.

Community-driven hyperscale innovation for all

5 of 22

Open Systems for AI Strategic Initiative

WORK AREAS

COMMUNITY WORK

ADOPTION & ECONOMY

Execution in�OCP Project workstreams

Contributions in OCP database

OCP Marketplace:

chiplets, products, solutions

and OCP Ready facilities

Systems Management

DC IT Infrastructure

DC Physical Infrastructure

+ OCP Future

Technologies Initiative

OPEN SYSTEMS FOR AI

OCP’s Strategic Initiative Across the Community

Connect. Collaborate. Accelerate.

Community-driven hyperscale innovation for all

6 of 22

Open Systems for AI: Work Areas

Cross-stack work will link together open systems within AI clusters

SYSTEMS MANAGEMENT

DC IT INFRASTRUCTURE

DC PHYSICAL INFRASTRUCTURE

DC Facilities

Server

Test and Validation

Rack and Power

Cooling Environments

AI Automation

Storage

Networking

Hardware Management

Open Platform Firmware

Memory Systems

Open Chiplet Economy

OCP

Future

Technologies Initiative

Sustainability

Security

Connect. Collaborate. Accelerate.

Community-driven hyperscale innovation for all

7 of 22

Assembling Open Systems of Systems

Project level

Cross-stack

Cluster level

Progressively Getting to Holistic AI Clusters Design with Open Source

Connect. Collaborate. Accelerate.

Community-driven hyperscale innovation for all

8 of 22

Cross-stack Innovation Examples for AI Clusters

Interconnect

Power

Liquid Cooling

Fast, high-capacity memory, storage and access of models, checkpoints, RAG vector DBs, datasets impacts:

Composable Memory Systems
Server NIC and baseboard
Storage devices
Dedicated networking
Rack/row placement

Compute density, driving up system power impacts:

DC power distribution
DC power backup
Rack/row power (AC vs DC) and batteries
Server PSU
Chip power delivery, e.g. vertical delivery
And cooling…

Liquid cooling demands to satisfy power requirements impact:

DC Facility CDU
Liquid distribution to and within racks (e.g. BMQC)
… within servers
Cold plate
Chip design and testing

…the SI should encourage alignment and engagement across projects

Connect. Collaborate. Accelerate.

Community-driven hyperscale innovation for all

9 of 22

Putting the OCP SI In Action

Define Challenges and Commonalities

Bridge Inter-Project Communications

Share Throughout the Community

new muscle within the OCP Community to reach across projects

Connect. Collaborate. Accelerate.

Community-driven hyperscale innovation for all

10 of 22

Open Systems for AI: Top Contributions

Falcon RT, NIC 3.0, Minipack and Wedge with TH5

OFP8 format, Micro-scaling MX format

Data-centric computing, short-reach optics, polymorphic architecture

Cooling Distribution, BMQC and UQDs, floats, liquids, plates, + news

Open Rack v3, + news

Grand Teton, HGX, DC-SCM, CMS + news

Quantization

Networking

Servers

Rack & Power

Cooling

Future Technologies (FTI)

Connect. Collaborate. Accelerate.

Community-driven hyperscale innovation for all

11 of 22

Open Systems for AI: Top Contributions

Falcon RT, NIC 3.0, Minipack and Wedge with TH5

OFP8 format, Micro-scaling MX format

Data-centric computing, short-reach optics, polymorphic architecture

Cooling Distribution, BMQC and UQDs, floats, liquids, plates

OpenRack v3, 19” MGX expanded rack, 1400A bus bar

Grand Teton (‘24), MGX compute/switch tray, HGX, DC-SCM, CMS

Quantization

Networking

Servers

Rack & Power

Cooling

Future Technologies (FTI)

Connect. Collaborate. Accelerate.

Community-driven hyperscale innovation for all

12 of 22

NEW CONTRIBUTIONS

NVIDIA and Meta

13 of 22

OCP Contribution by

NVIDIA has contributed MGX Accelerated GB200-NVL72 Computing Rack technologies and compute trays, building on ORv3

MGX Rack Infrastructure 19” EIA Support with Expanded Front Cable Volume

1400A electrical bus bar
Liquid cooling blind mate multi-node interconnect volumetrics and mounting, and blind mate manifolds

MGX compute and switch trays with modular front IO Bay design

Compute board form factors with space for a 1RU OCP DC-SCM
Liquid cooling multi-node connector volumetrics and blind mate UQD float mechanisms for seamless liquid cooling integration

NVIDIA is contributing all necessary design documents�(2D, 3D, specs, etc.) to enable scale production and there are already multiple sources that can deliver

Connect. Collaborate. Accelerate.

Community-driven hyperscale innovation for all

14 of 22

OCP Contribution by

Meta is introducing “Catalina” AI Rack ‘commons’ architecture for AI clusters
Catalina AI Rack architecture is specifically configured to deliver a high-density AI system that supports NVIDIA GB200
Past contribution of their “Grand Teton” server for AI clusters is being updated to 2024 specifications

Connect. Collaborate. Accelerate.

Community-driven hyperscale innovation for all

15 of 22

Open Systems for AI within OCP Marketplace

Networking

Rack & Power

Cooling

Server

400G+ switches

400G+ Cables

NICs v3.0

ORv3 racks

BBUs

Power Shelves

DC-SCM

MGX/HGX compute�boards and servers

Grand Teton�2024

Chiplets

*coming soon

BMQC

UQD

*

Fluids

Connect. Collaborate. Accelerate.

Community-driven hyperscale innovation for all

16 of 22

Expanding Reach with Alliances

SYSTEMS MANAGEMENT

DC IT INFRASTRUCTURE

DC PHYSICAL INFRASTRUCTURE

Connect. Collaborate. Accelerate.

Community-driven hyperscale innovation for all

17 of 22

Open Systems for AI: Status

Position Paper v1 underway, about 2/3^rd draft-complete
Marketing communications plan done in Q3-24. Continue to execute to plan and expand for 2025
Good participation in monthly SI meetings… which needs to continue with community sharing and guest speakers from OCP alliances and relevant players
Solid list of germane contributions and two new multi-faceted additions

Connect. Collaborate. Accelerate.

Community-driven hyperscale innovation for all

18 of 22

Open Systems for AI Highlights at Summit

and track + panel: “Special Focus: AI”

FTI workshop on�AI HW/SW Co-design

FTS AI/HPC Track

Server track OAI

Liquid Cooling

Networking�with UEC etc.

Storage for AI

Rack & Power for�400Vdc and >200kW

Keynotes

Photonics

Connect. Collaborate. Accelerate.

Community-driven hyperscale innovation for all

19 of 22

Community-driven hyperscale innovation for all

THANK YOU

opencompute.org/ai

20 of 22

BACKUP SLIDES

21 of 22

Open Systems for AI: Priorities

Close our v1 of the Position Paper
Increase marketplace listings to match contributions with member GTM reach via our engineering participants
Identify and operationalize cross-stack workstreams
Earlier planning of contributions for 25’ Summits
Showcase a few examples of OCP Inspired AI Clusters with SC leaders and proof points like MLPerf benchmarking and testing of up-stack software programming tools

Align with Foundation KPIs

Connect. Collaborate. Accelerate.

Community-driven hyperscale innovation for all

22 of 22

What Impacts Open Infrastructure Requirements?

For AI HW/SW Co-design, which up-stack pieces may impact us?

Training vs Inference vs Tuning

Workload Task and Model Type

Model Size

Data Sets� and Size

Other :�frameworks, libraries, MoE, RAG, ZSL, SL/USL/RL, …

Connect. Collaborate. Accelerate.

Community-driven hyperscale innovation for all