1 of 39

Scalable, Global Namespaces with Programmable Storage

Michael A. Sevilla

April 25th, 2018

Dissertation Defense

LA-UR-18-21419

2 of 39

What is a global namespace?

Names ↦ Data; Hierarchical Structure�e.g., DNS, network topologies, URLs, � scoping in programming languages

2

subtree

file

.txt

/dir

In this thesis, we focus on file system namespaces.

Hierarchical Semantics

Global Semantics

inherit

parent's ownership

strong

consistency

durability

Problem! POSIX IO file system metadata access semantics are difficult to scale.

Scalable, Global Namespaces

Dissertation Defense

Michael A. Sevilla

April 24, 2018

Introduction ←

Prototyping Platform

Subtree Load Balancing

API/framework

beyond FSs

Subtree Semantics

mechanisms

guarantees

Subtree Schemas

structure

generators

Conclusion

2identify {rsrcs/data/comps} in computer→ we give it a name

NS: organizes objects so they can be referred to by name, usually hierarchical~tree

group related objects together in a natural way;
we can org obj in any way we want (arbitrary # of levels or objects)

N: portion, N: F: array of bytes; FS: users access Fs {secure/portable/convenient} way

N: FSnspaces have global/hier semantics that dictate how tree can be accessed/modified

G: apply to whole tree; N: R/W to same file/dir → globally ordered, N: durability: users assume updates are saved
H: files/dirs have same properties parent

N: turns out semantics hard to maintain with large sys/wklds

------------------------------

names: designed for humans, so they are a list of characters

Defs: nspace: org data by name N: hierstruct: re-use & group; ~tree

N: (1) mental model (2) standardized 1985; aside: objstr scal/flex storsys {assume FLnspace/GenFields}

EXTRA: fs!dead; legacy apps, users accustomed {files/dirs, hierarchy, snapshots, file shares}

------------------------------

Resay what tanenbaum says about hierarchical namespaces

3 of 39

File system metadata access patterns:

→ Small and frequent requests

→ Target same resource�

3

[Sevilla et. al., SC'15]

Many metadata reads/writes

Fewer metadata reads/writes

Single Node

File System

metadata IO

(permissions, size, atime, etc.)

data IO

client

Distributed

File System

[Mesnier et. al., IEEE Comm.]

data IO

metadata IO

metadata IO does not scale like data IO

… as a result

Scalable, Global Namespaces

Dissertation Defense

Michael A. Sevilla

April 24, 2018

Introduction ←

Prototyping Platform

Subtree Load Balancing

API/framework

beyond FSs

Subtree Semantics

mechanisms

guarantees

Subtree Schemas

structure

generators

Conclusion

4 of 39

Scalable FS Metadata Access

… a brief history

4

'02

'04

'06

'08

'10

'12

'14

'16

'18

Metadata Distribution

Single Node Hash

Subtree Partition Table

CalvinFS

Ursa

Minor

IBRIX

Colossus�FS

Hierarchical Semantics

Global Semantics

inherit

parent's ownership

strong

consistency

durability

lock management
relaxing consistency
caching inodes
journal formats
journal safety
caching paths
metadata distribution
load balancing

GlusterFS

GPFS

Lazy

Hybrid

HBA

Giga+

PVFS2

SkyFS

{Index/Delta/

Batch/Shard}FS

pNFS

MarFS

'clean-slate' file systems

→ migrate? compare?

'dirty-slate' file systems

→ modify code? tunables?

File System

PanFS

Ceph

FS

Two

Tiers

Farsite

HopsFS

HDFS

GFS

Lustre

ADLS

in summary

Scalable, Global Namespaces

Dissertation Defense

Michael A. Sevilla

April 24, 2018

Introduction ←

Prototyping Platform

Subtree Load Balancing

API/framework

beyond FSs

Subtree Semantics

mechanisms

guarantees

Subtree Schemas

structure

generators

Conclusion

survey of RW; 1st look @ how mdata is distr. across cluster

- (2) RR mdata, (3) group mdata, (4) mix/match → muddled

N: sep pubs; but papers for these sys focus on other aspects of FSs

N: propose techs (most unique) → sgnspace N: N:...

This pic SHOULD stress you out…N: What can appdevs do? N: N: N:

sys struggle: today's wrklds (specialized, VolVarVel of data), selecting opt., guiding opt. w/ policies

LH: normalized hash; extra ACL entry for ancestors, lazy updates makes writes slow (perms, rename, rm, ++/--)

Farsite: federates win workstations; no hard links, centralized/global state for dyn redistribution

ADLS: distr. some type of mdata (like PVFS2), centralize others w/ relaxed consistency

HopsFS: multiple R/W, force clients to traverse and lock subtrees; partitioning based on policy

ColossusFS: bigtable (distr map); query tablets; bottlenecks are mitigated by workload specific implementations and aggressive caching

GPFS: global lock manager delagates authority subtrees; cache stat

***pNFS: client caches to support parallel access

***SkyFS: metadata cache

PanFS: specific caching/batching/logging policies (based on how each specific metadata op works); specific hardware

Ursa Minor: policy to assign related inodes together; client caches

HBA: find metadata quickly

PVFS2: distribute metadata w/ data; namespace/attribute cache

CalvinFS WAN replication with sql

Strong Consistency: CephFS, GPFS, PanFS, IndexFS, Lustre

Lock Management: PVFS2 vs. GPFS; same CephFS, PanFS, IndexFS, Lustre

Caching INodes: CephFS, Lustre, IndexFS, PVFS2, GPFS, GFS, Ursa Minor, pNFS

Relaxing Consistency: PVFS2, Panasas, Lustre, Batch/DeltaFS, MarFS, TwoTiers, GFS, HDFS, CephFS

Journal formats: CephFS, IndexFS, PanFS

Journal Safety: GFS, CephFS, HDFS, BatchFS, DeltaFS, PanFS

caching paths: GIGA+, IndexFS, LH, SkyFS, HBA, CephFS

distribution: GIGA+, CephFS, SkyFS, HBA, and Ursa Minor, shardfs, farsite,, calvinfs, pnfs

lb: indexfs, shardfs, cpehfs, GIGA+

Single Node: GFS, HDFS

contend for shared resource
requires extra writes (write amplification)
requires extra checks for parent subtrees

5 of 39

Programmable Storage

(our solution throughout this talk)

5

File System

Hierarchical Semantics

Global Semantics

inherit

parent's ownership

strong

consistency

durability

lock management
relaxing consistency
caching inodes
journal formats
journal safety
caching paths
metadata distribution
load balancing

Scalable, Global Namespaces

Dissertation Defense

Michael A. Sevilla

April 24, 2018

Introduction ←

Prototyping Platform

Subtree Load Balancing

API/framework

beyond FSs

Subtree Semantics

mechanisms

guarantees

Subtree Schemas

structure

generators

Conclusion

6 of 39

Contributions:

6

2

Subtree Semantics

Consistency/Durability

3

Subtree

Schemas

1

Subtree

Load Balancing

design policies that shape

metadata management techniques

1

API for specifying policies;

Policy engine for guiding mechanisms

2

Malacology

[EuroSys '17]

Mantle�[SC '15, CCGrid '18]

Cudele

[IPDPS '18]

Tintenfisch

[HotStorage '18]

7 of 39

Outline: Scalable, Global Namespaces

7

Malacology

Prototyping Platform

Mantle

Cudele

Tintenfisch

Malacology

Prototyping Platform

Scalable Global

Namespaces

1

Subtree

Load Balancing

3

Subtree

Schemas

2

Subtree Semantics

Consistency/Durability

Mantle

Cudele

Tintenfisch

8 of 39

Ceph Background (Used to Build Malacology)

8

Traditional Storage Interfaces

LIB

OBJECT

BLOCK

FILE (hierarchical namespace)

data IO

journal IO

client

metadata IO

RADOS

MDS Cluster

data IO

client

Scalable, Global Namespaces

Dissertation Defense

Michael A. Sevilla

April 24, 2018

Introduction

Prototyping Platform ←

Subtree Load Balancing

API/framework

beyond FSs

Subtree Semantics

mechanisms

guarantees

Subtree Schemas

structure

generators

Conclusion

Malacology�[EuroSys '17]

9 of 39

Malacology: A Programmable Storage System

9

application

developer

RADOS

✓ atomic ops

✓ batching

✓ data access

✓ consensus

✓ migration

My App

Traditional Storage Interfaces

LIB

OBJ

BLOCK

FILE

Scalable, Global Namespaces

Dissertation Defense

Michael A. Sevilla

April 24, 2018

Introduction

Prototyping Platform ←

Subtree Load Balancing

API/framework

beyond FSs

Subtree Semantics

mechanisms

guarantees

Subtree Schemas

structure

generators

Conclusion

Malacology�[EuroSys '17]

app that needs distrstor; 1st iter may work fine; BUT ++opt/fxn

N: 4example: impr. perf migratdata in/out system; integint2app (trick2getright, esp distr.) OR ++layer SW

------------------------------

N: N: N: N: prob: fxn instorsys !expos, API 2storsys-nwaist→ app has2do own dataMgmt

TR: What we did was...

N: maintain cstate → consensus engine N: many fxn shipping (apps useDis2 atomOps on servstordata)�storsys N: guard SR; 4perf impl~batching (group opsbulkup) N: access m/data diff thruout sys N: rebal m/data�N: N: handling stream; versC code w/data; distr. configs; txns on obj.�N: layers impl fxn 2provide theirServ; frustrating: ~fxn dup inStorsys

complex,bloated w/ redundant fxn; maintain SW evolves diff
Notsay nwaist bad; nwaists~POSIX; facilitates abtr.complsys & helpsSWevolv; NOT BAD; LIM
soln: reduce size, complexity, amount redundant comp

10 of 39

Malacology: A Programmable Storage System

10

My App

RADOS

✓ atomic ops

✓ batching

✓ data access

✓ consensus

✓ migration

My App

✓ consensus

✓ atomic operations

✓ batching

✓ migration

Traditional Storage Interfaces

LIB

OBJ

BLK

FILE

application

developer

Scalable, Global Namespaces

Dissertation Defense

Michael A. Sevilla

April 24, 2018

Introduction

Prototyping Platform ←

Subtree Load Balancing

API/framework

beyond FSs

Subtree Semantics

mechanisms

guarantees

Subtree Schemas

structure

generators

Conclusion

Malacology�[EuroSys '17]

11 of 39

Outline: Scalable, Global Namespaces

11

Mantle

Malacology

Prototyping Platform

Scalable Global

Namespaces

1

Subtree

Load Balancing

3

Subtree

Schemas

2

Subtree Semantics

Consistency/Durability

Mantle

Cudele

Tintenfisch

12 of 39

Data IO does not scale like metadata IO

→ Distribute File System Metadata Across Cluster

Solution 1: Hash File ID Solution 2: Subtree Partitioning

Dynamic version of these approaches

12

Current Approaches:�

File systems have mechanisms for metadata migration but it is the policies that determine performance

fundamental insight

locality

balance

locality

balance

Scalable, Global Namespaces

Dissertation Defense

Michael A. Sevilla

April 24, 2018

Introduction

Prototyping Platform

Subtree Load Balancing ←

API/framework

beyond FSs

Subtree Semantics

mechanisms

guarantees

Subtree Schemas

structure

generators

Conclusion

Mantle�[SC '15, CCGrid '18]

13 of 39

Mantle: Programmable Load Balancer

CephFS's Subtree Partitioning

Current Approach:

13

MDS Cluster

Policies

rebalance

recv HB

fragment

where

how much

when

migrate!

Mantle

API

where

how much

when

admin

locality

balance

Our Approach:

Scalable, Global Namespaces

Dissertation Defense

Michael A. Sevilla

April 24, 2018

Introduction

Prototyping Platform

Subtree Load Balancing

API/framework ←

beyond FSs

Subtree Semantics

mechanisms

guarantees

Subtree Schemas

structure

generators

Conclusion

Mantle�[SC '15, CCGrid '18]

14 of 39

Policies Expressed w/ Storage-agnostic Language

Mantle: API & Policy Engine for FS Metadata Load Balancing (our solution)

14

MDS

Cluster

# of inode writes

metadata load on subtrees

load on myself

load on neighbor

Good for mixed workloads

Good for create-heavy workload

Simple implementation

Scalable, Global Namespaces

Dissertation Defense

Michael A. Sevilla

April 24, 2018

Introduction

Prototyping Platform

Subtree Load Balancing

API/framework ←

beyond FSs

Subtree Semantics

mechanisms

guarantees

Subtree Schemas

structure

generators

Conclusion

Mantle�[SC '15, CCGrid '18]

15 of 39

Evaluating the Mantle API & Policy Engine

Mantle: API & Policy Engine for FS Metadata Load Balancing (our solution)

15

policy

GIGA+

modified

LARD

MDS0

MDS1

MDS2

MDS3

% of total load

100

½ load

50

¼ load

25

⅛ load

13

25

% of total load

75

25

% of total load

Time(minutes)

Metadata

(reqs/s)

Metadata

(reqs/s)

Metadata

(reqs/s)

File System

File systems have mechanisms for metadata migration but it is the policies that determine performance

fundamental insight

MDS capacity → 9% speedup

Conservative vs. Aggressive → 6-9% speedup

Metadata Protocols → 40% slowdown

Performance Summary

Scalable, Global Namespaces

Dissertation Defense

Michael A. Sevilla

April 24, 2018

Introduction

Prototyping Platform

Subtree Load Balancing

API/framework ←

beyond FSs

Subtree Semantics

mechanisms

guarantees

Subtree Schemas

structure

generators

Conclusion

Mantle�[SC '15, CCGrid '18]

Performance profile to show how we control behaviour

I’ll show how load gets split; load isn't perf metric (just use to design policy)�L: diff bal (1st show what happened w/ GS) XY,COL: per-mds tput (!stacked), purple �- thruput is uneven – each MDS gets diff load�R: diagram explains why; box: MDS; #: % of total load on that MDS

MDS0 starts w/ all load, but realizes he's overloaded so it sheds half its load to neighbor

Now MDS0/1 have ½ the total load�- MDS0/1 realize they want to shed load, BUT only programmed to look @ neighbor

...sep policy/mech w/ prog → implement diff policies w/ API/PEngine�----------------------------------------

F&S: spill part, only when I am overlaoded�But if you notice, where each of these finish… �performance not what we expected�Fleshed out in our perf analysis

16 of 39

Load Balance ZLog Sequencers

Cache Management for ParSplice

Using API & Policy Engine in Other Domains

16

↓ = workload access pattern detected

customized for sequencer workload

Customized to be Less Aggressive

distributed shared commit log

molecular dynamics simulation

Thanks, Michael Leece!

App-specific balancer → 1.5X speedup

Performance Summary

App-specific cache → 32-66% less memory

→ 0% perf. degradation

Memory Savings Summary

Scalable, Global Namespaces

Dissertation Defense

Michael A. Sevilla

April 24, 2018

Introduction

Prototyping Platform

Subtree Load Balancing

API/framework

beyond FSs ←

Subtree Semantics

mechanisms

guarantees

Subtree Schemas

structure

generators

Conclusion

Mantle�[SC '15, CCGrid '18]

Looked at proj we knew that needed LB; found ZLog, which is...

E comp in ZLog called seq; network process that hands out positions for end of log

- inprac; storsys impl CORFUAPI must support a multiplicity of ind. totally-ordered logs

- so assoc log/app, need seqr/log; prob: co-loc same serv

API/PEngine help us navigate design space; bal wkld & locality; caching looks @ same thing

AXES!!!

Mantle = control plane; injects policies into a running storage system (e.g. FS, kv-store)

→ app-specific storage stacks.

------------------------------�3 seq, eachw/ 4clis: keep all 3 seqs (12clis tot) - 1serv�N: X: time, Y: thruput�N: loadbals pkgd w/ CephFS (bals designed for POSIX FS metadata workloads)�N: → perf improve cuz seqr migr off 1st server; perf drops (sim to nobal) �N: Mantle service to custombals; N: write bal for RD: perf gets higher → 1st: recv/farms reqs, other: actual tail finding�N: another benefit to using Mantle is ability to control how aggr/cons bals are�perf stays lower for longer cuz want policy to wait to see how mig affect sys�CephFS curve, all bal trigg every 10 sec

C: LB helps improve perf; programmable LB can help you tailor to your serv/app; all configurable w/ Malac�

17 of 39

Mantle Takeaway

17

Mantle�general data management API

overhead of POSIX IO semantics

Malacology

Prototyping Platform

Scalable Global

Namespaces

1

Subtree

Load Balancing

3

Subtree

Schemas

2

Subtree Semantics

Consistency/Durability

Mantle

Cudele

Tintenfisch

18 of 39

Outline: Scalable, Global Namespaces

18

Mantle�general data management API

overhead of POSIX IO semantics

Cudele

Malacology

Prototyping Platform

Scalable Global

Namespaces

1

Subtree

Load Balancing

3

Subtree

Schemas

2

Subtree Semantics

Consistency/Durability

Mantle

Cudele

Tintenfisch

19 of 39

Global FS consistency/durability semantics

… which hurts performance OR correctness

Current Approaches:�

19

decoupled, no durability

RAMDisk semantics

decoupled, durable

DeltaFS semantics

weak consistency

HDFS semantics

strong consistency

POSIX IO semantics

Scalable, Global Namespaces

Dissertation Defense

Michael A. Sevilla

April 24, 2018

Introduction

Prototyping Platform

Subtree Load Balancing

API/framework

beyond FSs

Subtree Semantics ←

mechanisms

guarantees

Subtree Schemas

structure

generators

Conclusion

Cudele�[IPDPS '18]

20 of 39

Dynamically Assign Semantics to Subtrees

Cudele: API/policy engine for app-specific customizations (our solution)

20

�

With Cudele, clients can:�

relax consistency�

decouple subtrees�
lock subtrees�

adjust durability

decoupled, no durability

RAMDisk semantics

decoupled, durable

DeltaFS semantics

weak consistency

HDFS semantics

strong consistency

POSIX IO semantics

Scalable, Global Namespaces

Dissertation Defense

Michael A. Sevilla

April 24, 2018

Introduction

Prototyping Platform

Subtree Load Balancing

API/framework

beyond FSs

Subtree Semantics ←

mechanisms

guarantees

Subtree Schemas

structure

generators

Conclusion

Cudele�[IPDPS '18]

21 of 39

Composable Interfaces for Building Guarantees

… with Cudele (our solution)

21

Metadata Servers

Client

local�persist

append client journal

Object

Store

RPCs

stream

volatile

apply

Global

persist

Leveraged Ceph Internal Subsystem

Inode Cache

Journal

Metadata Store

Journal Tool

Consistency

C

Durability

D

Scalable, Global Namespaces

Dissertation Defense

Michael A. Sevilla

April 24, 2018

Introduction

Prototyping Platform

Subtree Load Balancing

API/framework

beyond FSs

Subtree Semantics

mechanisms ←

guarantees

Subtree Schemas

structure

generators

Conclusion

Cudele�[IPDPS '18]

L: interfaces we implemented to satisfy diff C/D guarantees that apps may need ®

B: subsys we leveraged inside Ceph; hack vs. principled approach

Recap: re-used READ; built mechanisms that can be composed to get diff guarantees

N: Strong C (all clients can see), Global D (save no mattter who fails); mapping btwn physical namespace ds and what users think of it

N: We added a bunch of mechanisms, using existing subsys in Ceph, to get a range of guarantees that apps can choose, depending on perf req

------------------------------

C/D guarantees: "local": updates lost of client/server dies AND stays down

“invisible”: sys !handle merging updates; assumes middleware/app manages consistency lazily
“weak” merges updates at some time in the future (e.g., when the system has time, when the number of updates reaches a certain threshold, when the client is done writing, etc.)

same strategy we used in EuroSys paper; dirty-slate approach

Used existing internal subsystems to implement DN in 354 loc lib, 4 loc in metadata server, 219 loc in

N: how Ceph normally works; clients send RPCs to metadata server, and it saves it as a journal in object store

N: mstore: namespace data struct; dirs in data structs in mem (MDS) or objects (objstore)

code that inserts/lookup stuff in metadata store

N: RPC mechanism gets strong C, no dur

N: to get D/fault tol, MDSs stream journal into obj store

N: journal: log of metadata updates; when applied, get back namespace

same state as mstore but it's better for writing quickly

N: stream mech → global durability; where client/server fail and no data loss

N: using journal tool (examine/manipulate log), export namespace and materialize journal in client memory

N: client appends new updates to its journal; this models DN of related work

just appending (not merging or saving) means that when the job is done, no one can see it and its volatile

N: so we fill in that cell in spectrum of C/D

N: journal tool → save to local disk (again using journal tool, also used Ceph's (d)encode)

N: to get some durability

N: journal tool/librados→ object store

N: better durability; so client/server can die and anyone can recover updates

N: leverage IC (used pre-alloc inos to clients) to alloc/skip inos for DN, journal code → replay on mstore

N: journal → apply directly to metadata store in MDS cluster (could also do through objstore)

N: gets us the rest of spectrum

Recap: re-used inode cache, journal replay and export, journal tool

22 of 39

Composable Interfaces for Building Guarantees

… with Cudele (our solution)

22

Strong consistency

Weak consistency

Invisible consistency

Meta-

data

Server

Clients

Custom fit subtree semantics

→ checkpoint-restart (91.7x speedup)

→ user home directories (0.03 stddev from optimal)

→ users checking partial results (2% overhead)

Performance Summary

Scalable, Global Namespaces

Dissertation Defense

Michael A. Sevilla

April 24, 2018

Introduction

Prototyping Platform

Subtree Load Balancing

API/framework

beyond FSs

Subtree Semantics

mechanisms

guarantees ←

Subtree Schemas

structure

generators

Conclusion

Cudele�[IPDPS '18]

23 of 39

Composable Interfaces for Building Guarantees

… with Cudele (our solution)

23

File System

Strong consistency

Weak consistency

Invisible consistency

Meta-

data

Server

Clients

Custom fit subtree semantics

→ checkpoint-restart (91.7x speedup)

→ user home directories (0.03 stddev from optimal)

→ users checking partial results (2% overhead)

Performance Summary

Scalable, Global Namespaces

Dissertation Defense

Michael A. Sevilla

April 24, 2018

Introduction

Prototyping Platform

Subtree Load Balancing

API/framework

beyond FSs

Subtree Semantics

mechanisms

guarantees ←

Subtree Schemas

structure

generators

Conclusion

Cudele�[IPDPS '18]

24 of 39

Cudele Takeaway

24

Mantle�general data management API

overhead of POSIX IO semantics

Cudele

different semantics can co-exist

read overheads (manage, mater-

ialize, transfer)

Malacology

Prototyping Platform

Scalable Global

Namespaces

1

Subtree

Load Balancing

3

Subtree

Schemas

2

Subtree Semantics

Consistency/Durability

Mantle

Cudele

Tintenfisch

25 of 39

Outline: Scalable, Global Namespaces

25

Mantle�general data management API

overhead of POSIX IO semantics

Cudele

different semantics can co-exist

read overheads (manage, mater-

ialize, transfer)

Tintenfisch

Malacology

Prototyping Platform

Scalable Global

Namespaces

1

Subtree

Load Balancing

3

Subtree

Schemas

2

Subtree Semantics

Consistency/Durability

Mantle

Cudele

Tintenfisch

26 of 39

Transferring and Materializing Large Lists

Current Approaches:

26

Client

Metadata Server

Traditional

Client

RPCs

High Performance Computing
High Energy Physics
Fusion Simulation

Scalable, Global Namespaces

Dissertation Defense

Michael A. Sevilla

April 24, 2018

Introduction

Prototyping Platform

Subtree Load Balancing

API/framework

beyond FSs

Subtree Semantics

mechanisms

guarantees

Subtree Schemas ←

structure

generators

Conclusion

Tintenfisch�[HotStorage '18]

27 of 39

Generate Namespaces for Large Lists

Our Solution:

27

Client

Metadata Server

Traditional

Client

RPCs

High Performance Computing
High Energy Physics
Fusion Simulation

Scalable, Global Namespaces

Dissertation Defense

Michael A. Sevilla

April 24, 2018

Introduction

Prototyping Platform

Subtree Load Balancing

API/framework

beyond FSs

Subtree Semantics

mechanisms

guarantees

Subtree Schemas ←

structure

generators

Conclusion

Tintenfisch�[HotStorage '18]

28 of 39

Example: PLFS Namespace

Middleware used in HPC �for checkpoint-restart

28

pattern

Repeat Pattern twice

PLFS specific metadata

Scalable, Global Namespaces

Dissertation Defense

Michael A. Sevilla

April 24, 2018

Introduction

Prototyping Platform

Subtree Load Balancing

API/framework

beyond FSs

Subtree Semantics

mechanisms

guarantees

Subtree Schemas

structure ←

generators

Conclusion

Tintenfisch�[HotStorage '18]

29 of 39

Example: Namespace Generators

Our Solution:

29

3. Pointer

for ROOT

2. Code for SIRIUS

1. Formula�for PLFS

High Performance �Computing

Fusion �Simulation

High Energy Physics

Obj Store

*

Scalable, Global Namespaces

Dissertation Defense

Michael A. Sevilla

April 24, 2018

Introduction

Prototyping Platform

Subtree Load Balancing

API/framework

beyond FSs

Subtree Semantics

mechanisms

guarantees

Subtree Schemas

structure

generators ←

Conclusion

Tintenfisch�[HotStorage '18]

30 of 39

Tintenfisch Takeaway

30

Mantle�general data management API

overhead of POSIX IO semantics

Cudele

different semantics can co-exist

read overheads (manage, mater-

ialize, transfer)

Tintenfisch

metadata struct. → generators

Malacology

Prototyping Platform

Scalable Global

Namespaces

1

Subtree

Load Balancing

3

Subtree

Schemas

2

Subtree Semantics

Consistency/Durability

Mantle

Cudele

Tintenfisch

31 of 39

Academic/Community Impact

31

Malacology

[EuroSys '17]

Mantle�[SC '15, CCGrid '18]

Cudele

[IPDPS '18]

Tintenfisch

[HotStorage '18]

Community

Impact

Funding:

Merged into

Presented to community

Featured in

"Reproducible" papers (Popper-compliant)

Malacology

Prototyping Platform

Scalable Global

Namespaces

1

Subtree

Load Balancing

3

Subtree

Schemas

2

Subtree Semantics

Consistency/Durability

Mantle

Cudele

Tintenfisch

32 of 39

Conclusion

32

policies that shape metadata

management techniques

subtree load balancing
subtree semantics
subtree schemas

1

API for specifying policies;

Policy engine for guiding mechanisms

2

→ facilitates application-specific software stacks

Scalable FS�metadata�techniques exist�(clean-slate/� dirty-slate)�

Problem! POSIX IO file system metadata access semantics are difficult to scale.

File System

Scalable, Global Namespaces

Dissertation Defense

Michael A. Sevilla

April 24, 2018

Introduction

Prototyping Platform

Subtree Load Balancing

API/framework

beyond FSs

Subtree Semantics

mechanisms

guarantees

Subtree Schemas

structure

generators

Conclusion ←

33 of 39

Future work

33

Scalable, Global Namespaces

Dissertation Defense

Michael A. Sevilla

April 24, 2018

Introduction

Prototyping Platform

Subtree Load Balancing

API/framework

beyond FSs

Subtree Semantics

mechanisms

guarantees

Subtree Schemas

structure

generators

Conclusion ←

34 of 39

What's Next?

34

Future work for me and this project:

academic interest [publications, conferences]
community interest [co-authors, pull requests]

important part of Ceph project
other communities

Future work for Scalable Global Namespaces

Subtree Load Balancing: general data management policies that can be used across apps/storage systems
Subtree semantics: dynamically changing subtree semantics; embeddable policies (child subtrees)
Subtree schemas: prototype; storage system agnostic metadata generation

35 of 39

Publications

[SC '15] Mantle: A Programmable Metadata Load Balancer for the Ceph File System � M. Sevilla, N. Watkins, C. Maltzahn, I. Nassi, S. Brandt, S. Weil, G. Farnum, S. Fineberg�[EuroSys '17] Malacology: A Programmable Storage System � M. Sevilla, N. Watkins, I. Jimenez, P. Alvaro, S. Finkelstein, J. LeFevre, C. Maltzahn�[HotStorage '17] DeclStore: Layering is for the Faint of Heart � N. Watkins, M. Sevilla, I. Jimenez, K. Dahlgren, P. Alvaro, Shel Finkelstein, C. Maltzahn �[IPDPS '18] Cudele: An API and Framework for Programmable Consistency and Durability in a Global Namespace � M. Sevilla, I. Jimenez, N. Watkins, J. LeFevre, P. Alvaro, S. Finkelstein, P. Donnelly, C. Maltzahn �[HotStorage '18] Tintenfisch: File System Namespace Schemas and Generators� M. Sevilla, R. Nasirigerdeh, C. Maltzahn, J. LeFevre, N. Watkins, P. Alvaro, M. Lawson, J. Lofstead, J. Pivarski��[;login '16] Standing on the Shoulders of Giants by Managing Scientific Experiments Like Software � I. Jimenez, M. Sevilla, N. Watkins, C. Maltzahn, J. Lofstead, K. Mohror, R. Arpaci-Dusseau, A. Arpaci-Dusseau�[IPDPSW '17] The Popper Convention: Making Reproducible Systems Evaluation Practical � I. Jimenez, M. Sevilla, N. Watkins, C. Maltzahn, J. Lofstead, K. Mohror, A. Arpaci-Dusseau, R. Arpaci-Dusseau �[ICPE '18] quiho: Automated Performance Regression Testing Using Fine Granularity Resource Utilization Profiles � I. Jimenez, N. Watkins, M. Sevilla, J. Lofstead, C. Maltzahn

[DISCS '13] A Framework for an In-depth Comparison of Scale-up and Scale-out � M. Sevilla I. Nassi, K. Ioannidou, S. Brandt, C. Maltzahn�[LSPP '14] SupMR:Circumventing Disk and Memory Bandwidth Bottlenecks for Scale-up MapReduce � M. Sevilla, I. Nassi, K. Ioannidou, S. Brandt, C. Maltzahn��

35

Popper

This Thesis

Big Data

36 of 39

Thanks

Mentors: Carlos Maltzahn, Scott Brandt, Jeff LeFevre, Peter Alvaro, Ike Nassi, Shel Finkelstein, and Kleoni Ioannidou

Industry Peers: Sam Fineberg, Bob Franks, Brad Settlemyer, Sage Weil, Greg Farnum, John Spray, Patrick Donnelly, Danny Perez, David Rich, Galen Shipman, Jim Pivarski, Margaret Lawson

SRL team: Noah Watkins, Ivo Jimenez, Joe Buck, Dimitris Skourtis, Adam Crume, Andrew Shewmaker, Jianshen Liu, Reza Nasirigerdeh, Ken Iizawa, and Lucho Ionkov

36

37 of 39

Lessons Learned

Checklist for success (2012)

High paying job
Certainty for future
Pass coding interview

2015

High paying job
Certainty for future
Pass coding interview

2018

High paying job
Certainty for future
Pass coding interview

37

This was me in 2012 when I started my PhD; I was an overachiever -- really good at school and taking tests…; the main reasons I started my PhD was because I was scared of failure (and I thought success was those things in the checklist); so I turned back to my comfort zone, which was school; I didn't know that research and school were inherently different; but I started climbing N: the ivory tower; maybe for all the wrong reasons, but that's what I did it.

Today I'm here. At the top of the ivory tower, but it's a N: long road and N: climbing it was the hardest thing I've ever done -- take a kid who was good at everything he does and start pounding failure into him. Halfway through, I got what I wanted. N: An offer that ticked off all the boxes. But I did a peculiar thing... I rejected it. Because I enjoyed what I was doing and I knew who I was. And now, N: here I am; we've come full circle… but did I fail? Failure is all about perspective

grad school pounded failure into me so much and cultivated failure that I learned that it's ok to fail; that it's about taking those failures in stride and maintaining humility and charity; to the point where we create a dark sense of humor just to handle our failures
you tell me at the beginning that I'd have 62% of my papers rejected, I'd have been horrified.
… but failure is not something to be scared of; you should be scared of doing things that make you unhappy

38 of 39

38

This was me in 2012 when I started my PhD; I was an overachiever -- really good at school and taking tests…; the main reasons I started my PhD was because I was scared of failure (and I thought success was those things in the checklist); so I turned back to my comfort zone, which was school; I didn't know that research and school were inherently different; but I started climbing N: the ivory tower; maybe for all the wrong reasons, but that's what I did it.

Today I'm here. At the top of the ivory tower, but it's a N: long road and N: climbing it was the hardest thing I've ever done -- take a kid who was good at everything he does and start pounding failure into him. Halfway through, I got what I wanted. N: An offer that ticked off all the boxes. But I did a peculiar thing... I rejected it. Because I enjoyed what I was doing and I knew who I was. And now, N: here I am; we've come full circle… but did I fail? Failure is all about perspective

grad school pounded failure into me so much and cultivated failure that I learned that it's ok to fail; that it's about taking those failures in stride and maintaining humility and charity; to the point where we create a dark sense of humor just to handle our failures
you tell me at the beginning that I'd have 62% of my papers rejected, I'd have been horrified.
… but failure is not something to be scared of; you should be scared of doing things that make you unhappy

39 of 39

Questions?

39