STAR-lite

STreaming ARchive format

atproto sync

integrity checks over per-repository data updates (firehose, space repo ops)

​

​

repair via whole-repo exports

repo

app

incremental

(small)

repo

app

resync, backfill

(whole repo archive)

steady state

repair

atproto sync

integrity checks over per-repository data updates (firehose, space repo ops)

​

​

repair via whole-repo exports

repo

app

incremental

(small)

repo

app

resync, backfill

(whole repo archive)

steady state

repair

✨

✨

atproto sync

integrity checks over per-repository data updates (firehose, space repo ops)

​

​

repair via whole-repo exports

repo

app

incremental

(small)

repo

app

resync, backfill

(whole repo archive)

steady state

repair

✨

✨

✨

✨

✨

atproto sync

integrity checks over per-repository data updates (firehose, space repo ops)

​

​

repair via whole-repo exports, up to a few GiB

repo

app

resync, backfill

(whole repo archive)

repo

app

incremental

(small)

steady state

repair

✨

✨

✨

✨

✨

✨

✨

atproto CARs

Serialized Merkle Search Tree

​

​

• Block CID verification footgun
• Full archive buffering required
• stream-ordering is tricky
• Garbage blocks -> resource DOS
• A lot of incompressible content

atproto CARs

Serialized Merkle Search Tree

​

redundant, MST is deterministic

• Block CID verification footgun
• Full archive buffering required
• stream-ordering is tricky
• Garbage blocks -> resource DOS
• A lot of incompressible content
• Streaming verification is possible

STAR-lite

STAR-lite

v0

public atproto repos

​

tall spec

v0

public atproto repos

​

tall spec

commit

v0

public atproto repos

​

tall spec

​

bleh

v0

public atproto repos

​

also algorithms

v0

public atproto repos only

​

with algorithms

verify MST

reconstruct

shipped.

hubble.microcosm.blue

Total exports: 32TiB

CAR: 3.8 billion records

STAR-lite: 330 billion?!

GET /xrpc/com.atproto.sync.getRepo

?did=did:plc:hdhoaan3xa3jiuq4fg4mefi

&accept=star-lite

atproto spaces???

space repo exports???????

space repos are

key-value set (not MST)

kv format -> kv format, easy?

“

The CAR header declares two roots, in order:

​

1. the signed commit
2. the index: a DRISL (DAG-CBOR) map from "{collection}/{rkey}" to the record's CID, with�keys in canonical DAG-CBOR map order (shortest key first, then bytewise)

​

record blocks MUST appear in that same order

sorting:

​

normal CBOR map

aaaaaaa ...

key001 key1

key1 key2

key10 ...

key2

zzz

​

“

The CAR header declares two roots, in order:

​

• the signed commit
• the index: a DRISL (DAG-CBOR) map from "{collection}/{rkey}" to the record's CID, with�keys in canonical DAG-CBOR map order (shortest key first, then bytewise)

​

record blocks MUST appear in that same order

sorting:

​

normal CBOR map

aaaaaaa ...

key001 key1

key1 key2

key10 key10

key2 key001

zzz ...

​

“

The CAR header declares two roots, in order:

​

• the signed commit
• the index: a DRISL (DAG-CBOR) map from "{collection}/{rkey}" to the record's CID, with�keys in canonical DAG-CBOR map order (shortest key first, then bytewise)

​

record blocks MUST appear in that same order

sorting:

​

normal CBOR map

aaaaaaa zzz

key001 key1

key1 key2

key10 key10

key2 key001

zzz aaaaaaa

​

v1

first attempt

​

reject cbor map sort order

​

still very atproto-coupled

v1

+ generic header meta

​

yay collaboration!

v1

+ generic header meta

+ plain bytes k/v

+ bigger size limits

+ ordered, not sorted

v1

STAR takes over the world??

v1

Application profiles

​

{

$type: “atproto-space-repo”,

...

}

v1

Application profiles

​

{

$type: “car-like”,

roots: [...]

}

as STAR: [ len CID len data ]

as CAR: [ len CID data ]

• +0.89 bytes per block
• +0.30% total overhead

(5.2M atproto repo block samples)

v1

Application profiles

​

{

$type: “scar”,

roots: [...]

}

STAR: [ len CIDpre len data ]

as CAR: [ len CID data ]

footgun-less CAR equiv?

questions

• $type name authority?
• content negotiation?
• no separator/terminator ok?
• define size limits or nah?

​

​

generally for DASL

• bijou64??

next steps

• Open DASL PR
• Hot new stuff status?
• Stabilizing with atproto spaces?
• …
• profit??

​

thanks especially: dholms, bnewbold, robin, volker, mosh, eve osman, nelind, sekoia.dev, a lot more people <3