Rapid growth of GPUs in DCs: �With great power comes great responsibility

Esha Choukse

Azure Research- Systems

Microsoft

How rapid is this growth?

350,000 NVIDIA H100 GPUs

• 437 MW of additional power in 1 year at Meta
• Enough for a town of 150,000 humans

Across Azure, AWS, GCP, and other DCs very conservatively, assuming 8 GW per year

>0.6% of total US grid

​

Projected annual growth in US electricity demand until 2050 is 1% per year

​

Virginia’s data center alley

Source: PJM grid operator data

Prime power

• We are running out, and utility companies are unable to meet demands
• In 5-10 years, we will see a lot of
• Requirements for flexibility and load shaping from the grid
• DCs with their own energy sources

​

Until then?

​

​

​

Datacenters today

Power efficiency in CPUs

• Has been around for decades
• Several works from CSPs have used various power features for efficiency

Power-performance

1 core active

n core active

Power

Frequency

We need to act fast

How rapid is this growth?

This just in, yesterday:

350,000 NVIDIA H100 GPUs

• 437 MW of additional DC power in 1 year for Meta
• Enough for a town of 150,000 humans
• Do we really need all that hardware?
• And does that hardware really need all that power?

Across Azure, AWS, GCP, and other DCs very conservatively, assuming 6 GW per year

~0.5% of total US grid

​

Projected annual growth in US electricity demand until 2050 is 1% per year

​

Do we need all that power?

• For training … yes
• We saw opportunity in LLM inference
• Max power utilization in a DC row was < 80%
• Inherent nature of the workload: statistical multiplexing of low and high power phases
• We should be able to oversubscribe power!
• And then we see the challenges
• GPUs are direct-device-access mode into VMs – no in-band access to cloud provider
• Out-of-band mechanism takes 40 seconds!

Power oversubscription despite the limitations of control/telemetry

• Despite the 40 seconds, we guarantee no power overdraw
• Choose the thresholds based on known maximum power spikes in 40 seconds
• Use heavy hammer of power brake, but sparingly
• While being able to oversubscribe power by 30%
• With just 5% latency impact to P99 latency of high priority workloads
• Can be better with better telemetry and control

Characterizing Power Management Opportunities for LLMs in the Cloud, ASPLOS 2024 – with our intern Pratyush Patel, UW

Do we need all that hardware?

• Better to use 70 servers at 70% utilization than 100 servers at 50% utilization

​

• LLMs can run memory capacity bound
• Memory is just 20-25% of GPU power
• Forces the GPU compute to run underutilized

1 core active

n core active

Power

Frequency

Break open the box to find efficiency

All the transformer-based generative LLMs in the past 4-5 years have the same behaviors from power/energy perspective

Motivation I

Motivation II

BLOOM-176B with 4k input & 512 output – * From per-VM pricing in CoreWeave

Prompt

Token

H100 better

A100 better

Break open the box to find efficiency

Unlocks upto 80% more throughput at same power!

Important to optimize for power, and not just cost

Splitwise: Efficient generative LLM inference using phase splitting, ISCA, 2024 – with our intern Pratyush Patel, UW

Splitwise: Layerwise KV-cache transfer

17

CPU

GPU

GPU

GPU

GPU

GPU

GPU

GPU

GPU

CPU

GPU

GPU

GPU

GPU

GPU

GPU

GPU

GPU

: 1/8^th of the KV-cache for an 8-GPU model

Layer 1

Layer 2

Splitwise has less than 0.8% impact on the end-to-end inference time

Prompt Machine

Token Generation Machine

Splitwise: Evaluated designs

Evaluated Splitwise designs all normalized to DGX-A100

SW-only solutions

Need infrastructure changes

Splitwise: System diagram

1. Route incoming requests
2. Maintain pending queue and batching
3. Mixed pool to avoid perf impact at high load
4. Pending queue for local batching
5. KV-cache transfer using MSCCL++

Splitwise: High-level results

Same power, but maximize throughput (requests per second are shown)

​

​

​

​

​

​

​

​

​

Other directions we are looking at

Azure Research – Systems

• Project: Efficient AI
• Extremely fast research 🡪 Product impact
• Sometimes faster than paper acceptances ☺
• We are hiring!

AI will make the world more efficient, but we need to keep making AI efficient

Thanks!

AI will make the world more efficient, but we need to keep making AI efficient

Splitwise Simulator

• Optimization functions
• Throughput, cost, and provisioned power
• Traces from production clusters running GPT-4
• Different distributions of prompt and generated sizes
• Bing (conversation)
• Github (coding)
• Service level objectives
• P50, P90, P99 for TTFT, TBT, E2E

Splitwise Simulator

Splitwise: High-level results

​

Same cost, but maximize throughput

• Splitwise-AA gets 40% more throughput than Baseline-H100 at 25% more power

​

Same throughput, but

• Lower power – win for Azure
• Splitwise-AA needs 50% less power (fewer machines) than Baseline-A100
• Splitwise-HH needs 9% less power (fewer machines) than Baseline-H100
• Splitwise-HHcap needs 20% less power than Baseline-H100
• Lower cost – win for the customer
• Splitwise-AA costs 25% less than Baseline-H100

​

​

Power vs Energy

• Peak power determines provisioning
• We reserve power for DCs
• Reducing peak power has a direct impact on the efficient use of the reserved power sources
• In a DC, there are always some nodes active, some not
• Energy efficiency still brings down peak power
• Secondary effect