1 of 61

How IPFS Works

(approximately)

Name (@github)

organization

original deck by @stebalien

2 of 61

3 of 61

4 of 61

5 of 61

6 of 61

IPFS:

Lifecycle

Adding

Files

Getting

Files

7 of 61

IPFS:

Adding Files

8 of 61

IPFS:

Adding Files

-> CID: Content Identifier

-> IPFS Path: /ipfs/QmS4ustL54uo8FzR9455qaxZwuMiUhyvMcX9Ba8nUH4uVv

-> Gateway URL: https://ipfs.io/ipfs/QmS4ustL54uo8FzR9455qaxZwuMiUhyvMcX9Ba8nUH4uVv

CID

9 of 61

IPFS:

Getting Files

CID

10 of 61

IPFS:

Lifecycle

Import

Name

Find

Fetch

Adding

Files

Getting

Files

11 of 61

Import

Name

Find

Fetch

Chunking

UnixFS

IPLD

CID

Path

IPNS

Routing

DHT

Kademlia

Bitswap

12 of 61

Chunking

UnixFS

IPLD

CID

Path

IPNS

Bitswap

Import

Fetch

Find

Name

Contiguous File:

Chunked File:

  • Deduplication
  • Piecewise Transfer
  • Seeking

(each chunk is hashed)

Routing

DHT

Kademlia

13 of 61

Chunking

UnixFS

IPLD

CID

Path

IPNS

Bitswap

Import

Fetch

Find

Name

Contiguous File:

Chunked File:

  • Deduplication
  • Piecewise Transfer
  • Seeking

Routing

DHT

Kademlia

14 of 61

Chunking

UnixFS

IPLD

CID

Path

IPNS

Bitswap

Import

Fetch

Find

Name

Contiguous File:

Chunked File:

  • Deduplication
  • Piecewise Transfer
  • Seeking

Routing

DHT

Kademlia

15 of 61

Chunking

UnixFS

IPLD

CID

Path

IPNS

Bitswap

Import

Fetch

Find

Name

Contiguous File:

Chunked File:

  • Deduplication
  • Piecewise Transfer
  • Seeking

Deduplicated:

Routing

DHT

Kademlia

16 of 61

Chunking

UnixFS

IPLD

CID

Path

IPNS

Bitswap

Import

Fetch

Find

Name

Contiguous File:

Fetched:

  • Deduplication
  • Piecewise Transfer
  • Seeking

Chunked File:

Routing

DHT

Kademlia

17 of 61

Chunking

UnixFS

IPLD

CID

Path

IPNS

Bitswap

Import

Fetch

Find

Name

Contiguous File:

Chunked File:

  • Deduplication
  • Piecewise Transfer
  • Seeking

Routing

DHT

Kademlia

18 of 61

Chunking

UnixFS

IPLD

CID

Path

IPNS

Bitswap

Import

Fetch

Find

Name

Contiguous File:

Chunked File:

  • Deduplication
  • Piecewise Transfer
  • Seeking

Routing

DHT

Kademlia

19 of 61

Chunking

UnixFS

IPLD

CID

Path

IPNS

Bitswap

Import

Fetch

Find

Name

File Chunks:

Routing

DHT

Kademlia

20 of 61

Chunking

UnixFS

IPLD

CID

Path

IPNS

Bitswap

Import

Fetch

Find

Name

File Chunks:

UnixFS File:

(merkle-link)

(a hash)

(merkle-tree)

0-200

200-350

0-100

100-200

100-200

200-250

Routing

DHT

Kademlia

21 of 61

Chunking

UnixFS

IPLD

CID

Path

IPNS

Bitswap

Import

Fetch

Find

Name

File Chunks:

UnixFS File:

(merkle-tree-dag) - directed acyclic graph

(merkle-link)

0-200

200-350

0-100

100-200

100-200

200-250

Routing

DHT

Kademlia

22 of 61

Chunking

UnixFS

IPLD

CID

Path

IPNS

Bitswap

Import

Fetch

Find

Name

UnixFS File(s):

UnixFS Directory:

README.md

main.go

go.mod

File Chunks:

(merkle-link)

Routing

DHT

Kademlia

23 of 61

Chunking

UnixFS

IPLD

CID

Path

IPNS

Bitswap

Import

Fetch

Find

Name

Meta-format for understanding, encoding, and decoding merkle-linked data.

Routing

DHT

Kademlia

24 of 61

http://b.com/Bar.json -> {

"foo": http://a.com/Foo.json

}

http://a.com/Foo.json -> {

"content": "I am foo"

}

Chunking

UnixFS

IPLD

CID

Path

IPNS

Bitswap

Import

Fetch

Find

Name

Linked Data

Routing

DHT

Kademlia

25 of 61

http://b.com/Bar.json -> {

"foo": http://a.com/Foo.json

}

http://a.com/Foo.json -> {

"content": "I am foo"

}

Chunking

UnixFS

IPLD

CID

Path

IPNS

Bitswap

Import

Fetch

Find

Name

Linked Data

Authority

Routing

DHT

Kademlia

26 of 61

QmBar -> {

"foo": QmFoo

}

QmFoo -> {

"content": "I am foo"

}

Chunking

UnixFS

IPLD

CID

Path

IPNS

Bitswap

Import

Fetch

Find

Name

Merkle-Linked Data

  • Immutable
  • Authority Less

Routing

DHT

Kademlia

27 of 61

Chunking

UnixFS

IPLD

CID

Path

IPNS

Bitswap

Import

Fetch

Find

Name

Content Identifier

  • Used for content addressing
  • Are self describing
  • Used to name every piece of data in IPFS/IPLD
  • Are basically a hash with some metadata

QmS4ustL54uo8FzR9455qaxZwuMiUhyvMcX9Ba8nUH4uVv

bafybeibxm2nsadl3fnxv2sxcxmxaco2jl53wpeorjdzidjwf5aqdg7wa6u

Routing

DHT

Kademlia

28 of 61

Chunking

UnixFS

IPLD

CID

Path

IPNS

Bitswap

Import

Fetch

Find

Name

Digression:

​

Content Addressing / Location Addressing

Routing

DHT

Kademlia

29 of 61

Chunking

UnixFS

IPLD

CID

Path

IPNS

Bitswap

Import

Fetch

Find

Name

Location

Addressing

"My cat, Ozzy, is here."

Digression: Content Addressing

Routing

DHT

Kademlia

30 of 61

Chunking

UnixFS

IPLD

CID

Path

IPNS

Bitswap

Import

Fetch

Find

Name

Content Addressing

"This is my cat, Ozzy."

Routing

DHT

Kademlia

31 of 61

Chunking

UnixFS

IPLD

CID

Path

IPNS

Bitswap

Import

Fetch

Find

Name

Location

Addressing

Digression: Content Addressing

Routing

DHT

Kademlia

32 of 61

Chunking

UnixFS

IPLD

CID

Path

IPNS

Bitswap

Import

Fetch

Find

Name

Location

Addressing

Digression: Content Addressing

Routing

DHT

Kademlia

33 of 61

Chunking

UnixFS

IPLD

CID

Path

IPNS

Bitswap

Import

Fetch

Find

Name

Location

Addressing

Digression: Content Addressing

Routing

DHT

Kademlia

34 of 61

Chunking

UnixFS

IPLD

CID

Path

IPNS

Bitswap

Import

Fetch

Find

Name

Location

Addressing

"That's the wrong cat!"

Digression: Content Addressing

Routing

DHT

Kademlia

35 of 61

Chunking

UnixFS

IPLD

CID

Path

IPNS

Bitswap

Import

Fetch

Find

Name

Location

Addressing

"That's the wrong cat!"

(But you can't know that!)

Digression: Content Addressing

Routing

DHT

Kademlia

36 of 61

Verifiable, Immutable, Trustless

​

Permanent

Routing

DHT

Kademlia

Chunking

UnixFS

IPLD

CID

Path

IPNS

Bitswap

Import

Fetch

Find

Name

37 of 61

Chunking

UnixFS

IPLD

CID

Path

IPNS

Bitswap

Import

Fetch

Find

Name

Digression:

​

Multiformats: Self Describing Data

Routing

DHT

Kademlia

38 of 61

Chunking

UnixFS

IPLD

CID

Path

IPNS

Bitswap

Import

Fetch

Find

Name

Digression: Multiformats

  • Multicodec: a non-magic number to uniquely identify a format, protocol, etc.
  • Multihash: a self describing hash digest.
  • Multibase: a self describing base-encoded string.

Routing

DHT

Kademlia

39 of 61

Chunking

UnixFS

IPLD

CID

Path

IPNS

Bitswap

Import

Fetch

Find

Name

Digression: Multiformats

Multicodec: a non-magic number.

name, tag, code, description

identity, multihash, 0x00, raw binary

ip4, multiaddr, 0x04,

dccp, multiaddr, 0x21,

dnsaddr, multiaddr, 0x38,

protobuf, serialization, 0x50, Protocol Buffers

cbor, serialization, 0x51, CBOR

raw, ipld, 0x55, raw binary

...

github.com/multiformats/multicodec

Routing

DHT

Kademlia

40 of 61

Chunking

UnixFS

IPLD

CID

Path

IPNS

Bitswap

Import

Fetch

Find

Name

Digression: Multiformats

Multihash: a self-describing hash digest:

  • Hash Function (multicodec)
  • Hash Digest Length
  • Hash Digest

Routing

DHT

Kademlia

github.com/multiformats/multihash

41 of 61

Chunking

UnixFS

IPLD

CID

Path

IPNS

Bitswap

Import

Fetch

Find

Name

Digression: A bit of metadata

Multibase: a self-describing base encoding.

  • A multibase prefix.
    • b - base32
    • z - base58
    • f - base16
  • Followed by the base encoded data.

bafybeibxm2...

Routing

DHT

Kademlia

42 of 61

Chunking

UnixFS

IPLD

CID

Path

IPNS

Bitswap

Import

Fetch

Find

Name

Self Describing

  • CIDv0: QmS4u...
    • Base58 encoded sha256 multihash
  • CIDv1: bafybei...
    • Multibase encoded (ipld format multicodec, multihash) tuple.
  • Why CIDv1?
    • Can be encoded in arbitrary bases (base32, base58, etc.).
    • Can link between merkle-dag formats using the ipld format multicodec.

Routing

DHT

Kademlia

43 of 61

Chunking

UnixFS

IPLD

CID

Path

IPNS

Routing

DHT

Kademlia

Bitswap

Import

Fetch

Find

Name

IPFS uses paths, not URIs/URLs:

Unlike URLs, paths are recursive:

​

/dns/github.com/tcp/22/ssh/git

​

Versus:

​

git+ssh://github.com:22

Like URLs, paths are namespaced:

​

/ipfs/QmFoo/welcome.txt

/ipns/QmBar/index.html

Not Composable!

44 of 61

Chunking

UnixFS

IPLD

CID

Path

IPNS

Routing

DHT

Kademlia

Bitswap

Import

Fetch

Find

Name

IPNS maps Public Keys to paths

/ipns/QmMyKey -> /ipfs/QmFoo (signed)

/ipns/QmMyKey -> /ipfs/QmSomethingNew

IPNS is mutable

/ipns/QmMyKey -> /ipns/QmYourKey

IPNS can point to arbitrary paths

45 of 61

Chunking

UnixFS

IPLD

CID

Path

IPNS

Bitswap

Import

Fetch

Find

Name

Content Address (CID)

Location Address (Peer)

Routing

DHT

Kademlia

46 of 61

Solution: Keep a "routing table"

Chunking

UnixFS

IPLD

CID

Path

IPNS

Bitswap

Import

Fetch

Find

Name

What

Who

QmFoo

Ozzy

QmBar

Izzy

Routing

DHT

Kademlia

47 of 61

But the table is too BIG!

Chunking

UnixFS

IPLD

CID

Path

IPNS

Routing

DHT

Kademlia

Bitswap

Import

Fetch

Find

Name

What

Who

QmFoo

Ozzy

QmBar

Izzy

... millions of lines later ...

​

QmXXXXXXXXXXXXXXXX

Seth (not a cat!)

48 of 61

Solution: Distribute the routing table and give a little bit to each peer.

Ozzy Knows

Izzy Knows

Chunking

UnixFS

IPLD

CID

Path

IPNS

Routing

DHT

Kademlia

Bitswap

Import

Fetch

Find

Name

What

Who

QmBar

Izzy

...

​

What

Who

QmFoo

Ozzy

...

​

49 of 61

How do we know who has what piece of the routing table?

Chunking

UnixFS

IPLD

CID

Path

IPNS

Routing

DHT

Kademlia

Bitswap

Import

Fetch

Find

Name

50 of 61

How do we know who has what piece of the routing table?

Solution: Deterministically distribute the routing table.

​

Chunking

UnixFS

IPLD

CID

Path

IPNS

Routing

DHT

Kademlia

Bitswap

Import

Fetch

Find

Name

51 of 61

Distance Metric: Is peer X closer to content C than peer Y?

Query Algorithm: Given the distance metric, how do we find the peers closest to C.

​

Chunking

UnixFS

IPLD

CID

Path

IPNS

Routing

DHT

Kademlia

Bitswap

Import

Fetch

Find

Name

52 of 61

Distance Metric: XOR(HASH(C), HASH(Peer))

Query Algorithm:

  • Ask the closest peers you know for closer peers.
  • Remember the closest peers.

Chunking

UnixFS

IPLD

CID

Path

IPNS

Routing

DHT

Kademlia

Bitswap

Import

Fetch

Find

Name

53 of 61

Distance Metric: "Is this closer?"

Query Algorithm: "How do I get closer?"

Chunking

UnixFS

IPLD

CID

Path

IPNS

Routing

DHT

Kademlia

Bitswap

Import

Fetch

Find

Name

54 of 61

Chunking

UnixFS

IPLD

CID

Path

IPNS

Bitswap

Import

Fetch

Find

Name

Routing

DHT

Kademlia

55 of 61

Izzy Wants

  • QmTreats
  • QmToy

Ozzy Wants

  • QmCuddles
  • QmFood
  • QmAttention

Izzy

Ozzy

Chunking

UnixFS

IPLD

CID

Path

IPNS

Bitswap

Import

Fetch

Find

Name

Routing

DHT

Kademlia

56 of 61

Izzy Wants

  • QmTreats
  • QmToy

Ozzy Wants

  • QmCuddles
  • QmFood
  • QmAttention

Izzy

Ozzy

Routing

DHT

Kademlia

Chunking

UnixFS

IPLD

CID

Path

IPNS

Bitswap

Import

Fetch

Find

Name

57 of 61

Izzy Wants

  • QmTreats
  • QmToy

Ozzy Wants

  • QmCuddles
  • QmFood
  • QmAttention

Izzy

Ozzy

Routing

DHT

Kademlia

Chunking

UnixFS

IPLD

CID

Path

IPNS

Bitswap

Import

Fetch

Find

Name

58 of 61

Izzy Wants

  • QmTreats
  • QmToy

Ozzy Wants

  • QmCuddles
  • QmFood
  • QmAttention

Izzy

Ozzy

Routing

DHT

Kademlia

Chunking

UnixFS

IPLD

CID

Path

IPNS

Bitswap

Import

Fetch

Find

Name

QmFood

QmAttention

QmToy

59 of 61

Izzy Wants

  • QmTreats

Ozzy Wants

  • QmCuddles

Izzy

Ozzy

Routing

DHT

Kademlia

Chunking

UnixFS

IPLD

CID

Path

IPNS

Bitswap

Import

Fetch

Find

Name

60 of 61

Chunking

UnixFS

IPLD

CID

Path

IPNS

Bitswap

Import

Fetch

Find

Name

Routing

DHT

Kademlia

61 of 61

How IPFS Works

(approximately)

Name (@github)

organization