IPFS 0.5: The Fastest Way to Find, Fetch, & Share Data Peer-to-Peer

IPFS Ecosystem

​

30x increase in 2019

100Ks of nodes

IPFS Public Network

​

13M req / day

~5 TB of data / day

IPFS HTTP Gateway

(↑4-5x this year)

Q1 Highlights

Opera launched default IPFS:// support in their android browser!

Brave launched their official store on ORIGIN swag (built on IPFS)

Wolfram language supports PFS as the default external storage system

Ethereum.org put their vuepress website on IPFS & ENS

Use Fleek to automate website deployments from github to IPFS

Upload your website to IPFS within the ENS Domain Manager

Easy IPFS sites with templates, drag & drop upload, & more

Equilibrium Labs kicked off the new rust-ipfs implementation & community

Our collab w Netflix resulted in making Bitswap transfer speeds ~3x faster

Content Routing

Areas of Improvement

AutoNAT:

• IPFS’s Kademlia DHT is a single network of peers where peers participate in routing content requests & peer look-up requests between others in the network.
• Problem: What happens when not all peers can connect to all other peers? The network structure starts to degrade.

​

• Solution:
• Ask other peers about our dialable status
• If we determine we’re not dialable, stop advertising that we can respond to queries
• Backup Plan: All peers should defensively examine their connections from other peers to determine if they look like they are connectable

Cleaning up the DHT

Routing Tables:

• Routing tables form the basis of how peers find each other in the DHT
• So many peers were undialable that we decided to only tell people about peers we had active connections to
• Problem: Forgetting Good Peers
• Intermittent network disconnects
• Heavy load of non-DHT connections

​

• Solution: Only remove peers if...
• ...we try to ask them a question and fail
• ...or, they haven’t been useful to us in a while and we discover a new peer to replace them with

Keeping good Peers

Lookup Algorithm:

• The Kademlia lookup logic repeatedly asks peers: “Do you know anyone closer to you than X?”

​

• Problem:
• Routing tables full of “bad” peers causes the query to take a very LONG time.
• Many failed dials to peers
• Decreases confidence that we have reached the closest peers to X
• Requires searching more of the network

​

• Solution: With improved routing tables we can...
• Query the closest peers we know to X, until they don’t know anyone closer

So, who do we ask?

Testing:

Let’s take it out for a spin!

Testing:

20-30x faster

DHT Results

Testing:

2-6x faster

DHT Results

Testing:

~5x faster

DHT Results

Content Routing:

• Notes:
• These numbers are highly variable.
• They'll get better as the network upgrades.
• More improvements on the way.

Public Network

42 sec

IPFS 0.4.23

14-20 sec

IPFS 0.5.0

vs

Avg

95th%

8 sec

1.5 sec

Content Exchange

Content Exchange:

• Substantial protocol changes to:
• Reduce wasted bandwidth due to duplicate blocks.
• Increase parallelism and throughput.

Bitswap

Content Exchange:

• Go-ipfs 0.5 can respond to graphsync requests (when the experiment is enabled).

​

• What is Graphsync?
• Successor to bitswap.
• Syncs at the IPLD graph layer instead of the block layer.
• Can request an entire graph all at once (e.g., an entire file or directory tree).

​

• What's missing?
• Requesting data via graphsync.
• Parallelizing graphsync requests.
• All the "smarts" that makes bitswap "just work".

Graphsync

Import/Export

Import/Export:

• Async datastore writes in badger with explicit sync points.
• 2-3x performance improvement over badger with sync writes.
• Performance on-par with copy/paste.

​

• Note: Badger not enabled by default due to two outstanding issues:
• 1GiB Memory Spikes
• GC not effective

​

Add Performance

Import/Export:

~$ ipfs dag export QmQPeNsJPyVWPFDVHb77w8G42Fvo15z4bG2X8D2GhfbSXc \

| xz > welcome_to_ipfs.car.xz

​

0s 6.73 KiB / ? [-------=-------------------------------------] 5.16 MiB/s 0s

​

~$ xz -dc welcome_to_ipfs.car.xz | ipfs dag import

​

Pinned root QmQPeNsJPyVWPFDVHb77w8G42Fvo15z4bG2X8D2GhfbSXc success

​

~$ ipfs dag export QmQPeNsJPyVWPFDVHb77w8G42Fvo15z4bG2X8D2GhfbSXc | xz | nc ...

Dag Import/Export

Libp2p

Libp2p:

• In 0.5
• Faster NAT detection.
• Rate limiting.
• Dial-back filtering and restrictions.
• AutoNAT service enabled by default on all go-ipfs nodes (highly rate-limited).

​

• Next Steps:
• Even faster NAT detection in 0.6
• Deeper integration into libp2p to avoid advertising undialable addresses.

Improved AutoNAT

Libp2p:

• TLS is now the default transport. 🎉
• Well understood security properties.
• Faster.

​

• Next up:
• Noise js interop.
• Secio deprecation in 0.7.0 likely.

TLS By Default

Libp2p:

• New QUIC protocol version (with breaking changes)
• Final update before we stabilize it.

​

• Next up:
• Will be enabled by default in 0.6 🎉

QUIC Draft 27

Gateway

Gateway:

Subdomain Gateway

IPNS

IPNS:

/ipns/ipfs.eth

Transparent ENS

IPNS:

/ipns/bafzbeibxfjp4gaxc4cdn57257cyvc7jfa4rlp4e5min6geg44m57g6nx7e

Base32 IPNS

IPNS:

• Fetches the latest record from pubsub peers instead of the DHT.
• Ensures we remain connected to pubsub peers interested in the same IPNS key using the DHT.
• Will be stabilized in go-ipfs 0.6.0 (along with pubsub).

Improved IPNS over PubSub

• Reduced badger memory usage.
• Flatfs improvements: avoid leaving temporary files behind and better handle file descriptor limits.
• Dial backoff logic is now per address instead of per-peer.
• Minimum RSA key size has been bumped to 2048.
• The default PubSub router has been changed from floodsub to gossipsub.
• API now only accepts POST requests.
• "Error: api not running" error has been killed off along with the ipfs repo fsck command.
• The ipfs add command has gained --ignore and --ignore-rules-path flags to ignore files on add.
• Significantly improved systemd support.
• With systemd notification & socket activation support.
• The API can now listen on unix domain sockets.
• Docker container is built with OpenSSL by default.

But wait, there’s more...

• Two repo migrations
• Bootstrapper migration to migrate to the new bootstrappers. The old ones used 1024bit RSA keys.
• Keystore migration to encode key names on disk for better cross-platform compatibility.
• Plugins
• MacOS support
• New permanently unstable "internal" plugin type for ultra-privileged plugins with access to all internals.
• Plugin configuration support.
• IPFS can be initialized with an existing config: ipfs init /path/to/existing/config
• Pins can be streamed with ipfs pin ls --stream
• Windows improvements
• Additional antivirus workarounds in flatfs.
• Double-clicking ipfs.exe starts the daemon.
• ipfs add -r now properly recognizes hidden files on windows.
• And a ton of bug fixes.

And even more!

Thank You!

Feel free to get in touch: @momack28 on Twitter

Questions?

More improvements are coming! 🚀