VDEV Properties

Date: November 8, 2021

Speakers:

Allan Jude, CTO @ Klara Inc. � Mark Maybee @ Delphix

​

Motivation

• ZFS properties are a good way to express intent
• Filesystems, Volumes, and Pools have properties
• The ZFS UI provides both human and machine readable output for properties
• VDEVs should have properties too

What Properties?

• Many counters in each vdev_t that we can expose
• Writable properties allow control over the vdev
• User properties can store arbitrary data
• What if you could extract the details from zpool status in a machine readable way? #Monitoring #Metrics

What Can You Set?

• Comment – Some free text about each vdev
• Path – Change the path used to open the device
• Allocating – Allow new allocations (more shortly)
• User properties: tld.namespace:property = value
• Future: VDEV Queue Config?

What Can I Read?

• Device ID
• Physical Path
• Enclosure Path
• FRU
• Parent vdev
• Children
• Num Children

​

• Size
• Free
• Allocated
• bootsize
• asize
• psize
• ashift

• Expandsize
• Parity
• Fragmentation
• Capacity
• GUID
• State
• Removing

• Read Errors
• Write Errors
• Cksum Errors
• Initialize Errors
• Operations
• Bytes

What Does It Look Like

# zpool get all pool0 all-vdevs

NAME PROPERTY VALUE SOURCE

sdb capacity 64% -

sdb state ONLINE -

sdb ashift 9 -

sdb size 15.9G -

sdb free 5.69G -

sdb allocated 10.2G -

sdb comment - default

sdb parity 0 -

sdb path /dev/sdb1 -

sdb devid ata-VBOX_HARDDISK_VBe8a6b5f3-3cfbe0e8-part1 -

sdb physpath pci-0000:00:0d.0-ata-2.0 -

sdb parent pool0 -

sdb children - default

sdb numchildren 0 -

sdb read_errors 0 -

sdb write_errors 0 -

sdb checksum_errors 0 -

sdb read_ops 505 -

sdb write_ops 813 -

sdb free_ops 0 -

sdb claim_ops 0 -

sdb trim_ops 0 -

sdb read_bytes 2.43M -

sdb write_bytes 5.10M -

sdb free_bytes 0 -

sdb claim_bytes 0 -

sdb trim_bytes 0 -

sdb removing off -

sdb allocating on default

​

​

Implementation

• Properties are stored in per-vdev ZAP
• ZAP introduced previously, no feature flag required
• Top level vdevs (‘mirror’, ‘RAID-Z*’) can have properties. No inherited currently.

Allocating Property

Allocating vdev Property

• Only applies to top-level vdevs
• Device must have allocatable space
• Default value is “on”
• All top-level vdevs can be allocated from by default
• Setting to “off” prevents further allocations from the vdev
• Any remaining space on the vdev is unavailable
• Impacts the total available pool space
• Must have at least one allocatable device in pool
• This is not a replacement for read-only pools!

Primary use Case: Device Removal

• Current device removal process:
• Request device removal
• verify space available for device removal
• mark device non-allocatable (impacts pool available space)
• remove device (by moving contents to other devices in pool)
• Request another device removal
• same steps as above
• etc.
• New supported process:
• Set one or more devices to allocating=off
• verifies space available (impacts pool available space)
• Request device removal
• no space validation required if device is set non-allocating
• remove device (by moving contents to allocating devices)
• Remove another device
• etc.

Device Removal Example

• Scenario: replacing drives in config
• Replace two old 4TB drives with a new (bigger/faster) 8TB drive
• Assume each old drive starts with 2TB of data

Replace Two Drives (without property)

Two new drives added to config

Copy data from drive A

A

B

Remove drive A

Copy data from drive B

Remove drive B

Request drive A removal

Request drive B removal

Replace Two Drives (without property)

zpool add tank devC

devB

devA

devC

Replace Two Drives (without property)

zpool remove tank devA

devB

devA

devC

Data copied = 2TB

Replace Two Drives (without property)

One drive removed

devB

devC

Replace Two Drives (without property)

zpool remove tank devB

devB

devC

Data copied = 3TB

Replace Two Drives (without property)

FINISHED

​

Total data copied = 5TB

devC

Replace Two Drives (using property)

zpool add tank devC

devB

devA

devC

Replace Two Drives (using property)

zpool set allocating=off tank devA devB

devB

devA

devC

Replace Two Drives (using property)

zpool remove tank devA

devB

devA

devC

Data copied = 2TB

Replace Two Drives (using property)

zpool remove tank devB

devB

devC

Data copied = 2TB

Replace Two Drives (using property)

FINISHED

​

Total data copied = 4TB

devC

Replace Two Drives (using property)

Two new drives added to config

Copy data from drive A

Copy data from drive B

Remove drive A

Remove drive B

Request drive A removal

Request drive B removal

Why “allocating” instead of “read-only”?

• This does not make a device read-only
• Label updates still permitted
• Overwrite still possible (scrub still works)
• Impact on pool space is more than “read-only”
• Expectation for read-only
• Only the unused vdev space would be subtracted from available space
• Actual impact
• Entire drive capacity is subtracted from available space
• Able to accommodate existing content “rewrite”
• It would be a surprise if rewriting a file consumed additional space
• Needed to support device removal (where all content is “rewritten”)
• Allocating is a more accurate name

What Else Could We Do?

• ZCP (Channel Program) Integration
• Mirror Read Bias (Prefer reads from specific member)
• Property Caching (libzfs does this for pool props)
• Inheritance (previously thought to avoid this)
• Persistent Counters (track total writes to SSD etc)

Closing Thoughts

• What other settings could be vdev properties?
• Special vdevs control (metadata or small blocks only)
• How many properties is too many?
• Setting vdev properties are pool creation time
• Should some properties be not-displayed-by-default?

Let’s stay in touch!

• E-mail us: contact@klarasystems.com
• Check out our website: https://klarasystems.com
• Follow us on Twitter: @klarainc and @allanjude
• Follow us on LinkedIn: Klara Inc.

Questions?