1 of 22

Zero to Hero, Learn dWeb Storage in 30mins

Australia Crypto Convention

Jenks Guo�Developer Advocate�Filecoin Foundation

My name is Jenks, I work for a very interesting and unique organisation called Filecoin Foundation. Filecoin Foundation is the steward of Fielcoin network, a storage network’s mission is to store humanities most important data. ��Today, on this special day at Gold Coast, in the sunshine state. I would like to speak to you all about decentralized web storage, what it is, why it is important to web3, and how IPFS & Filecoin tackle this challenge. And, in my mind, built one of the most important foundations for the decentralized web.

Moreover this is a workshop, workshop means you need to work for it. Today, in addition to me working hard on the stage, you will also get to do something. The work to do, if you choose to follow along, is to upload a picture of your choice to IPFS & Filecoin’s decentralized storage network. Your picture, which is your content, will be immutable, censorship-free, indestructible and semi-permanent. It will be there publicly until kingdom come.

So after this workshop, you are not only equipped with knowledge about decentralized web storage, you also have something tangible to show your friends & colleagues an url of your dweb image and tell them you worked hard for it.

Sounds good? Okay, let’s get started.

2 of 22

What is a CDN?

Data storage is important to us

Personal Cloud Storage

$61.42 billion market

CAGR 37.6%

Largest in North America

Key players:

Amazon, Apple, Google, Seagate, Box, Microsoft, Dropbox …

Photo by Edgar Moran on Unsplash

When was the last time you lost your precious photos & personal data? Years ago, a colleague of mine traveled to cuba for work with his personal phone, the company has administrative access to his phone and found no records of him being assigned to the country. So to avoid getting into trouble with Cuban authorities, because of very strict software & data import laws, his phone was erased without his consent. Years of his family photos were gones in a snap, precious memories perished.

My entire photo gallery since 2008 is now backed up on a fragile 2TB western digital external hard drive that could fail any time. To back this up on multiple clouds, it will incur a considerable amount of cost every year.

Data storage is very important to our lives. The importance is growing with the rise of demand for efficient storage, increased frequency of data access, data sharing and usage of portable devices such as tablets and smartphones capable of higher bandwidth, processing power to process data intensive applications and, of course, higher resolution of photos and videos.

The market size of personal cloud storage is currently at $61.42 billion USD, and is growing at a compound annual growth rate of 37.6%. This rate is forecasted to last till 2027. North america is the dominant region.This data came from Market Research Future this year.

Dominant Key Players on Personal Cloud Storage:

Amazon
Apple
Google
Seagate
Box
Microsoft
Dropbox

Storing data is a good business because the demand keeps growing.

3 of 22

What is a CDN?

Data storage is important to the internet

Cloud Storage Stats

2000 Exabyte

$78 Billion Market

CAGR 18%

Data storage is incredibly important to the internet. If we think about it, just about everything we interact with online, has something to do with storage. A website is a bunch of frontend code that is downloaded to your browser, and your browser makes it look pretty by rendering it. A mobile app needs to be downloaded from the app store to run on your local device. The pictures & videos we view online, need to be stored somewhere and then be retrieved by websites or an app.

Research by Statistica.com shows that in 2021 the total capacity of cloud data center storage in the world is over 2000 Exabytes. One exabyte equals 1 million terabytes. The world is using about 470 Exabytes of data right now. Cloud storage is a 78 billion USD market and is growing at a 18.% compound growth rate per year.

4 of 22

We are not good at preserving data online

Average website lifespan is 2.5 years

We are storing a lot of things online, but we are not good at keeping it. When you stop paying your storage providers, or when the developer stops paying cloud services, the data and content online will be deleted and the space is reused for something else. So they can perish easily.

Seen this somewhere online? When content is not there, a web server will return a 404 error.

This happens more often than you think on a global scale. According to Forbes, the average lifespan of a website is 2 years and 7 months. This average figure might be a generalization but it is shocking to think that the webpages you are viewing right now, in more than 2 and a half years, might not be there anymore.

Preserving data online is hard because I think there are two problems:

We are relying on people or organisations to persist data
Our current centralised internet design is not good for data persistence

5 of 22

Web 1.0

Web 2.0

Web 3.0

Internet

Web 3.0 is the Read-Write-Trust-Verifiable Web

The world is watching the evolution of the internet. From the beginning of the internet, it was used to wire up computers and transfer files point to point.

Later in the web 1.0 era, when establishing servers was cheaper, many of the informational read-only static websites were created for people to consume content. Changing the content relies on the owners of the servers and it was not so easy to do so.

The web 2.0, which is considered to be today’s internet, comes with the dominance of the platform websites. Platforms like facebook and twitter, allows people to create content and share easily. However the custody of the data is now in the hands of the platform businesses. Although it is very convenient and helps democratization of internet technology, we are increasingly seeing data being centralized into certain platforms and people losing control over their data. It also creates lots of silos. Such as your picture on Facebook doesn't show up on Linkedin. It leads to a lot of duplication of data and content needs to downloaded and served usually multiple times adding burden to the wires.

So a group of very smart people got together and thought this is not cool. We need a better version of the internet because it is such an important part of our life now. So the Web 3.0 movement was created, the intention is to return the data and control of the internet back to the hands of the users, normal users like you and me. This is something very new, but there is a lot of investment and lots of resources being poured into it, and it is really happening right now.

6 of 22

Web 3.0 is the combination of …

Distributed Web�

Work offline�
Faster�
Safer�
No need for central point of coordination

Permanent Links

Blockchain�

Publicly Verifiable�
Permissionless�
Incentive alignment

Semantic Web�

Graph of Knowledge

Programmable understanding of meaning

What is Web3 exactly? It is a combination of a few things & attributes:

First is a concept of Distributed Web:

Works offline, that means you don’t always need to be connected to the main network all the time. I will explain more in the IPFS section of my speech later.
It is faster
It is safer
It should require not central point of coordination
It should give you the ability to create permanent links, so you can access it independent of the location

Second it is Blockchain empowered:

Blockchain technology helps online content to be publicly verified
It should be permionless, so anyone can join the network
It should have incentive mechanism & crypto-token economy to help with longevity & the right participation in the network

The last, but not least is, the concept of Semantics Web:

Programmable understanding of the meaning. Allow machines to understand the meaning of the objects online, this way they can be more helpful to us as they understand how they should be used and intent.
This will also allow building better knowledge graphs, because it breaks silos.

Imaging in the semantic web, the search engine understands that Gold Coast is a location as a single object and doesn’t care which website hosts information about Gold Coast. So it represents it as facts and is able to show all of the interactions you may want to have with it, such as navigate there, find flight tickets and explore the beauty of it.

7 of 22

8 of 22

Your picture will be public & might be permanently stored on dweb!

patient-surf-6690.on.fleek.co

Take a photo with your phone and upload the file via my demo app

Well, before I start. This is an afternoon session, I don’t want to send you to nap by talking too much.

I mentioned earlier this is a workshop. So let’s do the workshop part of this talk.

You can see on the screen that there is a QR code and a web address. This QR code will take you to a decentralised web application that is deployed to IPFS and Filecoin to allow you to upload a file, any file to the decentralized storage network.

To make sure the content is fresh, may I ask you to take a photo with your phone and upload the photo via this web page to dweb. If you are brave enough, you can take a selfie, but I need to remind you that this picture is going on IPFS & Filecoin, which means it will be publicly available ( if anyone has the link) can see it and it will persist on the network for a very long time because the system is designed to persist data. I can’t guarantee that I could take it down even if you requested me to, and threatened to take my life.

By the end of it, after the upload, don’t close the webpage. Instead grab a copy of the url that it shows. That url is a IPFS Content ID that you can visit in the future, that CID is a finger print of your picture and no one can temper that link or swap the image out. Because the url itself contains an immutable reference to every bit of that image.

Sounds good? If you are not asleep yet, I suggest you do it.

Look at this, a whole conference room working hard in a workshop. Who says the speaker can’t get the audience to do something? The speaker gets to have some fun as well!

While you are doing that. I will just say a little bit more about this site. This site is actually open-source, it was written as a demonstration application for developers to learn about IPFS & a service of Filecoin called Web3.storage. Web3.storage is like box or dropbox for web 3, it provides easy UI, API & SDK interfaces for developers to integrate to IPFS and Filecoin quickly. It is a centralized tool, but once files are uploaded to IPFS it goes into dweb and can be accessed anywhere. So if the user has a dweb enabled browser like Brave or Opera, they can instead of typing in http://xxxxx, use ipfs://CID to visit the files stored.

The site is also deployed to the IPFS and Filecoin using a tool called Fleek.co. Fleek.co is another project integrated with Filecoin. They are like Netlify or Web3 and are able to provide continuous deployment and integration to website builders. This application was deployed on Fleek, for less than 10 clicks and in a few minutes. It is a really really good experience for developers. So please consider it, or ask your developer to consider it for your next dweb project.

So have you done it? For the ones who are quick, you might already have gotten a linke that ends with dweb.link and that will be your IPFS picture served by a gateway called dweb.link. Because your phone browser is not likely to be web3 enabled, so Dweb.link plays the role of bridging web2 to web3.

If your image does not load yet, please don’t worry, it will eventually. The gateway will eventually find the content at Web3.storage’s IPFS node and serve the content back to you.

9 of 22

WTF Just Happened?

WTF happened just then?

So what just happened you might be asking? It seems that you have uploaded a photo somewhere, and were able to read it right after with a funky link. Big deal?

While it seems simple from a user’s perspective, there are many significant things happening under the bonnet.

First the website you accessed via the QR code is hosted on worlds largest peer to peer file system called IPFS, there is no centralized server that can be shutdown to stop the site from running.

The website does not store any information, rather it sends your file to a service called Web3.storage which is a Filecoin product.

Web3.storage runs an IPFS node and your content is pinned on the peer to peer network to be visited anywhere. A copy of it is also stored on Filecoin for persistent storage so it will not disappear for a long time. Few of the 4000+ storage providers of Filecoin are using their enterprise grade storage to store your photo right now.

10 of 22

10

Computer

file://�path/to/index.html

Web2�IP + port

http://domain.com/�path/to/index.html

ipfs://[CID]/�path/to/index.html

IPFS�Content ID

IPFS addresses content by what it is, instead of where it is

It replaces a folder or file location with a Content ID

IPFS �Interplanetary File System

11 of 22

12 of 22

13 of 22

14 of 22

Any�decentralised�and reliable�alternatives?

…

15 of 22

Filecoin is storage designed for Web3 from the ground up

Compatible with IPFS & Web3

Use IPFS as a building block to share all the benefits of content addressing and CIDs

Integrates with Web3 networks like NEAR and many others��Will soon be compatible with EVM (storage + smart contracts!) through FVM.

Verifiable, via cryptographic proofs

Ensures that providers are actually storing data and keeping it safe

Uses block rewards as incentives & collateral slashing as penalties��Gives you control over the permanence of your data

A massive, decentralized network

18,000,000 TB of capacity across the globe provided by thousands of storage providers

Governed by consensus, instead of a single corporation

Filecoin is storage designed for web3 from the ground up.

Compatible with IPFS & Web3

Use IPFS as a building block to share all the benefits of content addressing and CIDs

Integrates with Web3 networks like NEAR and many others

Will soon be compatible with EVM (storage + smart contracts !) through FVM.

Verifiable, via cryptographic proofs

Ensures that providers are actually storing data and keeping it safe

Uses block rewards as incentives & collateral slashing as penalties

Gives you control over the permanence of your data

A massive, decentralized network

18,000,000 TB of capacity across the globe provided by thousands of storage providers

Governed by consensus, instead of a single corporation

Community driven - by a foundation funded for 50 years of open community development

A filecoin storage deal can be made as many times as the user wants and the replication can be as redundant as the user wants to be. By default the storage deal is either 6 month or 18 months long. The deals are time bound to allow adjustment so the storage can always be competitive compared to cloud storage but also able to support a sustainable economic model.

16 of 22

18.379 EiB

4059

179.70 PiB

87M

September 8, 2022

7.16M

17 of 22

IPFS ❤️ Filecoin

17

IPFS for fast, flexible retrieval (gateways, local nodes, browsers)

Filecoin for persistence and verifiability

18 of 22

Fleek hosting

18

Designed for

Fast, modern, censorship-proof websites and web-apps on the open web
Works with most modern frameworks (Docker, Gatsby, React, Webflow, Hugo, Next, Jekyll, etc.)

How it works

Connect your Github repository
Add build settings
Deploy your site to IPFS
Backed up to Filecoin

Usage

Github Actions
CLI
GraphQL API

fleek.co/hosting

19 of 22

NFT.Storage

Designed for

NFT devs who want free, multi- generational, decentralized storage

How it works

Compute CID of data locally that can be used in an NFT as a pointer to your content
Once data is uploaded, available to retrieve via IPFS and backed up to Filecoin (>8x redundancy)
Long-term decentralization in the works (smart contract-enforced redundancy, DAO-based funding)
Store everywhere else that makes you comfortable

Usage

Javascript client library
HTTP API + remote pinning service
Web interface

https://nft.storage/

Another one of my other favourite tools is NFT.Storage.

Storing your NFT metadata immutably & permanently, as you may know is integral to keeping the main value proposition of NFT’s - their non-fungibility, and this is exactly what NFT.Storage has been built for, as the name suggests :P

So NFT.Storage was specifically created as a public good to archive and persist NFT data & it takes care of the complexity around firstly creating an IPFS CID for your NFT metadata, and then makes an automatic deal to store this content persistently on Filecoin (with 8 or more times redundancy) and a multi-generational timeframe.

Over 45 million NFTs are currently stored on NFT.storage and it’s used by both OpenSea and Magic Eden to name a few.��It’s also all currently free and the team is also currently working on a feature to include perpetual renewal of filecoin storage deals which can be triggered (for eg) in smart contracts - so there will be no need to trust a centralised entity with this either.

If you are interested in using this, I’ve also got a little starter kit demo project on github and a video on this topic which you might want to take a look at. I think Ally has already put them in our sponsors discord channel.

They have SDKs in just about any programming language I know.

20 of 22

Web3.Storage

Designed for

General Filecoin & IPFS storage
Familiar and simple interfaces
Production-level storage and retrieval reliability + performance

How it works

Compute CID of data locally and upload to edge worker
Data immediately available to the IPFS network
Automatically stores with 6+ Filecoin providers around the world
1TiB always free

Usage

Javascript and Go client libraries
HTTP API + remote pinning service
Web interface

https://web3.storage/

21 of 22

Estuary

What is it

An open-source allows easy sending of public data to Filecoin network
Provides an end-to-end solution on how IPFS, Filecoin & other tech can be used together
A reference build for others

How it works

Using libp2p protocol for establishing secured and multiplexed connection between peers
Lotus-gateway for interact with Filecoin Chain
Lots of deals automation

Usage

Install and run
Run Dockerized instances
Hosted version

https://estuary.tech

22 of 22

Summary

Jenks Guo�Developer Advocate�Filecoin Foundation�Twitter: @jenksguo