NVMe HDD Spin-up options

Spinup ad-hoc team:

Dave Landsman, WD

Yoni Shternhell, WD

Christoph Hellwig, WD

Larry McMillan, WD

Brad Besmer, BRCM

Tim Walker, STX

​

03/03/2022

Agenda

• High Level Functional Requirements
• Example Architectures of Interest
• How Does SAS Do It?
• Option 1: New PCIe Upstream Message
• Option 2: Existing PME# Message
• Option 3: MI
• Option 4: Notify

​

​

High-Level Functional Requirements

• Chassis-managed power budgeting with no host (root complex) knowledge of chassis power state
• Host may not be able to monitor chassis power state and in general does have knowledge of other hosts hitting the same enclosure with spin-up requests
• Direct-attach HDD to CPU PCIe ports
• Low density storage servers and fabric chassis
• Control traffic routable through an industry-standard fanout switch
• High density chassis with multi-level switching
• HDD autonomous spin-up
• Some spin-ups are not host-commanded, either directly (spin up command) or indirectly (media access while spun down).
• Spin-up control totally through power budget & allowance?
• No spin-up token or specific “spin-up” permission is granted – power limits are only control (PCIe backwards compatibility).
• Avoid NVMe-HDD specific PCIe Spec changes if practical.
• We believe that the SIG will be reluctant to support an HDD-specific features

​

• The spin-up control system primarily pertains to io-operation spin-up and spin-down. Initial chassis power up is generally covered by slot power gating and the auto-spin-up feature in NVMe 2.0

Example Architectures of Interest

Simple JBOD

Pizza Box / Fabric Chassis

High Density Fabric Chassis / Storage Server

Switch

PS

HDD

HDD

HDD

HDD

BMC

RC

CPU

RC

CPU

Switch

...

Switch

Switch

Switch

HDD

HDD

HDD

HDD

Switch

...

...

A

B

C

D

High Density JBOD

Use-case extremes

Use-case complexities

NIC

NIC

How Does SAS Do It?

• NOTIFY (ENABLE SPINUP) primitive gives spin up.
• Drives that receive a START/STOP command to spin up will wait until it receives the NOTIFY (ENABLE SPINUP) primitive before spinning up.
• Expanders may routinely send NOTIFY (ENABLE SPINUP) to drives or coordinate NOTIFYs with START/STOP from the host.
• Easy to implement but does not support autonomous spin-ups.
• REQUEST-GRANT
• A set of four primitives allow a device to request power and release the request when done.
• PWR_REQ, PWR_GRANT, PWR_DONE, and PWR_ACK.
• The primitives create a bidirectional handshake with the HBA or expander, which can grant the request based on power management algorithms.

Option 1: New PCIe Upstream Message

• New PCIe upstream message enabling request for more power
• Drive and switch implement protocol
• Potential gaps:
• upstream routing limitations means the first switch layer requires power management hooks.
• CPU vendors would have to implement for direct attach. Handle at driver?

Option 2: Existing PME# Message

• Existing PME# message signals power request
• Drive and host implement protocol, assuming “competent RC” always available.
• Potential Gaps
• Requires a RC to catch the PME#, so host is in the loop

​

PM_PME Message

Option 3: MI

• No PCIe changes or involvement
• Host, BMC, NVM System must implement MI Management Endpoint
• Does switch have to do implement?
• Endpoint in SSD sends MCTP message up through hierarchy; beginning protocol handshake
• Potential gaps:
• Allow endpoint to initiate MI transaction

​

= MI Mgmt Endpoint

Option 4: Notify(ENABLE SPINUP) Emulation

• Host or switch modulates slot_power_limit to each drive round-robin. If drive has pending spin-up it will start.
• Potential gaps:
• Drive must wait for Notify to spin up adding latency coming out of standby

​