How IPFS Works
(approximately)
Name (@github)
organization
original deck by @stebalien
IPFS:
Lifecycle
Adding
Files
Getting
Files
IPFS:
Adding Files
IPFS:
Adding Files
-> CID: Content Identifier
-> IPFS Path: /ipfs/QmS4ustL54uo8FzR9455qaxZwuMiUhyvMcX9Ba8nUH4uVv
-> Gateway URL: https://ipfs.io/ipfs/QmS4ustL54uo8FzR9455qaxZwuMiUhyvMcX9Ba8nUH4uVv
CID
IPFS:
Getting Files
CID
IPFS:
Lifecycle
Import
Name
Find
Fetch
Adding
Files
Getting
Files
Import
Name
Find
Fetch
Chunking
UnixFS
IPLD
CID
Path
IPNS
Routing
DHT
Kademlia
Bitswap
Chunking
UnixFS
IPLD
CID
Path
IPNS
Bitswap
Import
Fetch
Find
Name
Contiguous File:
Chunked File:
(each chunk is hashed)
Routing
DHT
Kademlia
Chunking
UnixFS
IPLD
CID
Path
IPNS
Bitswap
Import
Fetch
Find
Name
Contiguous File:
Chunked File:
Routing
DHT
Kademlia
Chunking
UnixFS
IPLD
CID
Path
IPNS
Bitswap
Import
Fetch
Find
Name
Contiguous File:
Chunked File:
Routing
DHT
Kademlia
Chunking
UnixFS
IPLD
CID
Path
IPNS
Bitswap
Import
Fetch
Find
Name
Contiguous File:
Chunked File:
Deduplicated:
Routing
DHT
Kademlia
Chunking
UnixFS
IPLD
CID
Path
IPNS
Bitswap
Import
Fetch
Find
Name
Contiguous File:
Fetched:
Chunked File:
Routing
DHT
Kademlia
Chunking
UnixFS
IPLD
CID
Path
IPNS
Bitswap
Import
Fetch
Find
Name
Contiguous File:
Chunked File:
Routing
DHT
Kademlia
Chunking
UnixFS
IPLD
CID
Path
IPNS
Bitswap
Import
Fetch
Find
Name
Contiguous File:
Chunked File:
Routing
DHT
Kademlia
Chunking
UnixFS
IPLD
CID
Path
IPNS
Bitswap
Import
Fetch
Find
Name
File Chunks:
Routing
DHT
Kademlia
Chunking
UnixFS
IPLD
CID
Path
IPNS
Bitswap
Import
Fetch
Find
Name
File Chunks:
UnixFS File:
(merkle-link)
(a hash)
(merkle-tree)
0-200 | 200-350 |
0-100 | 100-200 |
100-200 | 200-250 |
Routing
DHT
Kademlia
Chunking
UnixFS
IPLD
CID
Path
IPNS
Bitswap
Import
Fetch
Find
Name
File Chunks:
UnixFS File:
(merkle-tree-dag) - directed acyclic graph
(merkle-link)
0-200 | 200-350 |
0-100 | 100-200 |
100-200 | 200-250 |
Routing
DHT
Kademlia
Chunking
UnixFS
IPLD
CID
Path
IPNS
Bitswap
Import
Fetch
Find
Name
UnixFS File(s):
UnixFS Directory:
README.md | main.go | go.mod |
File Chunks:
(merkle-link)
Routing
DHT
Kademlia
Chunking
UnixFS
IPLD
CID
Path
IPNS
Bitswap
Import
Fetch
Find
Name
Meta-format for understanding, encoding, and decoding merkle-linked data.
Routing
DHT
Kademlia
http://b.com/Bar.json -> {
"foo": http://a.com/Foo.json
}
http://a.com/Foo.json -> {
"content": "I am foo"
}
Chunking
UnixFS
IPLD
CID
Path
IPNS
Bitswap
Import
Fetch
Find
Name
Linked Data
Routing
DHT
Kademlia
http://b.com/Bar.json -> {
"foo": http://a.com/Foo.json
}
http://a.com/Foo.json -> {
"content": "I am foo"
}
Chunking
UnixFS
IPLD
CID
Path
IPNS
Bitswap
Import
Fetch
Find
Name
Linked Data
Authority
Routing
DHT
Kademlia
QmBar -> {
"foo": QmFoo
}
QmFoo -> {
"content": "I am foo"
}
Chunking
UnixFS
IPLD
CID
Path
IPNS
Bitswap
Import
Fetch
Find
Name
Merkle-Linked Data
Routing
DHT
Kademlia
Chunking
UnixFS
IPLD
CID
Path
IPNS
Bitswap
Import
Fetch
Find
Name
Content Identifier
QmS4ustL54uo8FzR9455qaxZwuMiUhyvMcX9Ba8nUH4uVv
bafybeibxm2nsadl3fnxv2sxcxmxaco2jl53wpeorjdzidjwf5aqdg7wa6u
Routing
DHT
Kademlia
Chunking
UnixFS
IPLD
CID
Path
IPNS
Bitswap
Import
Fetch
Find
Name
Digression:
Content Addressing / Location Addressing
Routing
DHT
Kademlia
Chunking
UnixFS
IPLD
CID
Path
IPNS
Bitswap
Import
Fetch
Find
Name
Location
Addressing
"My cat, Ozzy, is here."
Digression: Content Addressing
Routing
DHT
Kademlia
Chunking
UnixFS
IPLD
CID
Path
IPNS
Bitswap
Import
Fetch
Find
Name
Content Addressing
"This is my cat, Ozzy."
Routing
DHT
Kademlia
Chunking
UnixFS
IPLD
CID
Path
IPNS
Bitswap
Import
Fetch
Find
Name
Location
Addressing
Digression: Content Addressing
Routing
DHT
Kademlia
Chunking
UnixFS
IPLD
CID
Path
IPNS
Bitswap
Import
Fetch
Find
Name
Location
Addressing
Digression: Content Addressing
Routing
DHT
Kademlia
Chunking
UnixFS
IPLD
CID
Path
IPNS
Bitswap
Import
Fetch
Find
Name
Location
Addressing
Digression: Content Addressing
Routing
DHT
Kademlia
Chunking
UnixFS
IPLD
CID
Path
IPNS
Bitswap
Import
Fetch
Find
Name
Location
Addressing
"That's the wrong cat!"
Digression: Content Addressing
Routing
DHT
Kademlia
Chunking
UnixFS
IPLD
CID
Path
IPNS
Bitswap
Import
Fetch
Find
Name
Location
Addressing
"That's the wrong cat!"
(But you can't know that!)
Digression: Content Addressing
Routing
DHT
Kademlia
Verifiable, Immutable, Trustless
Permanent
Routing
DHT
Kademlia
Chunking
UnixFS
IPLD
CID
Path
IPNS
Bitswap
Import
Fetch
Find
Name
Chunking
UnixFS
IPLD
CID
Path
IPNS
Bitswap
Import
Fetch
Find
Name
Digression:
Multiformats: Self Describing Data
Routing
DHT
Kademlia
Chunking
UnixFS
IPLD
CID
Path
IPNS
Bitswap
Import
Fetch
Find
Name
Digression: Multiformats
Routing
DHT
Kademlia
Chunking
UnixFS
IPLD
CID
Path
IPNS
Bitswap
Import
Fetch
Find
Name
Digression: Multiformats
Multicodec: a non-magic number.
name, tag, code, description
identity, multihash, 0x00, raw binary
ip4, multiaddr, 0x04,
dccp, multiaddr, 0x21,
dnsaddr, multiaddr, 0x38,
protobuf, serialization, 0x50, Protocol Buffers
cbor, serialization, 0x51, CBOR
raw, ipld, 0x55, raw binary
...
github.com/multiformats/multicodec
Routing
DHT
Kademlia
Chunking
UnixFS
IPLD
CID
Path
IPNS
Bitswap
Import
Fetch
Find
Name
Digression: Multiformats
Multihash: a self-describing hash digest:
Routing
DHT
Kademlia
github.com/multiformats/multihash
Chunking
UnixFS
IPLD
CID
Path
IPNS
Bitswap
Import
Fetch
Find
Name
Digression: A bit of metadata
Multibase: a self-describing base encoding.
bafybeibxm2...
Routing
DHT
Kademlia
Chunking
UnixFS
IPLD
CID
Path
IPNS
Bitswap
Import
Fetch
Find
Name
Self Describing
Routing
DHT
Kademlia
Chunking
UnixFS
IPLD
CID
Path
IPNS
Routing
DHT
Kademlia
Bitswap
Import
Fetch
Find
Name
IPFS uses paths, not URIs/URLs:
Unlike URLs, paths are recursive:
/dns/github.com/tcp/22/ssh/git
Versus:
git+ssh://github.com:22
Like URLs, paths are namespaced:
/ipfs/QmFoo/welcome.txt
/ipns/QmBar/index.html
Not Composable!
Chunking
UnixFS
IPLD
CID
Path
IPNS
Routing
DHT
Kademlia
Bitswap
Import
Fetch
Find
Name
IPNS maps Public Keys to paths
/ipns/QmMyKey -> /ipfs/QmFoo (signed)
/ipns/QmMyKey -> /ipfs/QmSomethingNew
IPNS is mutable
/ipns/QmMyKey -> /ipns/QmYourKey
IPNS can point to arbitrary paths
Chunking
UnixFS
IPLD
CID
Path
IPNS
Bitswap
Import
Fetch
Find
Name
Content Address (CID)
Location Address (Peer)
Routing
DHT
Kademlia
Solution: Keep a "routing table"
Chunking
UnixFS
IPLD
CID
Path
IPNS
Bitswap
Import
Fetch
Find
Name
What | Who |
QmFoo | Ozzy |
QmBar | Izzy |
Routing
DHT
Kademlia
But the table is too BIG!
Chunking
UnixFS
IPLD
CID
Path
IPNS
Routing
DHT
Kademlia
Bitswap
Import
Fetch
Find
Name
What | Who |
QmFoo | Ozzy |
QmBar | Izzy |
... millions of lines later ... | |
QmXXXXXXXXXXXXXXXX | Seth (not a cat!) |
Solution: Distribute the routing table and give a little bit to each peer.
Ozzy Knows
Izzy Knows
Chunking
UnixFS
IPLD
CID
Path
IPNS
Routing
DHT
Kademlia
Bitswap
Import
Fetch
Find
Name
What | Who |
QmBar | Izzy |
... | |
What | Who |
QmFoo | Ozzy |
... | |
How do we know who has what piece of the routing table?
Chunking
UnixFS
IPLD
CID
Path
IPNS
Routing
DHT
Kademlia
Bitswap
Import
Fetch
Find
Name
How do we know who has what piece of the routing table?
Solution: Deterministically distribute the routing table.
Chunking
UnixFS
IPLD
CID
Path
IPNS
Routing
DHT
Kademlia
Bitswap
Import
Fetch
Find
Name
Distance Metric: Is peer X closer to content C than peer Y?
Query Algorithm: Given the distance metric, how do we find the peers closest to C.
Chunking
UnixFS
IPLD
CID
Path
IPNS
Routing
DHT
Kademlia
Bitswap
Import
Fetch
Find
Name
Distance Metric: XOR(HASH(C), HASH(Peer))
Query Algorithm:
Chunking
UnixFS
IPLD
CID
Path
IPNS
Routing
DHT
Kademlia
Bitswap
Import
Fetch
Find
Name
Distance Metric: "Is this closer?"
Query Algorithm: "How do I get closer?"
Chunking
UnixFS
IPLD
CID
Path
IPNS
Routing
DHT
Kademlia
Bitswap
Import
Fetch
Find
Name
Chunking
UnixFS
IPLD
CID
Path
IPNS
Bitswap
Import
Fetch
Find
Name
Routing
DHT
Kademlia
Izzy Wants
Ozzy Wants
Izzy
Ozzy
Chunking
UnixFS
IPLD
CID
Path
IPNS
Bitswap
Import
Fetch
Find
Name
Routing
DHT
Kademlia
Izzy Wants
Ozzy Wants
Izzy
Ozzy
Routing
DHT
Kademlia
Chunking
UnixFS
IPLD
CID
Path
IPNS
Bitswap
Import
Fetch
Find
Name
Izzy Wants
Ozzy Wants
Izzy
Ozzy
Routing
DHT
Kademlia
Chunking
UnixFS
IPLD
CID
Path
IPNS
Bitswap
Import
Fetch
Find
Name
Izzy Wants
Ozzy Wants
Izzy
Ozzy
Routing
DHT
Kademlia
Chunking
UnixFS
IPLD
CID
Path
IPNS
Bitswap
Import
Fetch
Find
Name
QmFood
QmAttention
QmToy
Izzy Wants
Ozzy Wants
Izzy
Ozzy
Routing
DHT
Kademlia
Chunking
UnixFS
IPLD
CID
Path
IPNS
Bitswap
Import
Fetch
Find
Name
Chunking
UnixFS
IPLD
CID
Path
IPNS
Bitswap
Import
Fetch
Find
Name
Routing
DHT
Kademlia
How IPFS Works
(approximately)
Name (@github)
organization