1 of 57

A Collaborative, Secure, and Private�InterPlanetary Wayback Web Archiving System Using IPFS

Mat Kelly

Old Dominion University

Norfolk, Virginia, USA

@machawk1

David Dias

Protocol Labs

Planet Earth

@daviddias

https://github.com/oduwsdl/ipwb

https://ipfs.io

IIPC Web Archiving Conference • June 15, 2017 • London, UK

w/ Sawood Alam, Michael L. Nelson, and Michele C. Weigle

2 of 57

Outline

InterPlanetary File System Motivation & Design
InterPlanetary Wayback Motivation & Design
How IPFS/IPWB relate, relevancy to Web archiving
Advances in IPFS/IPWB
Demo(s)

3 of 57

IPFS

Motivation & Design

4 of 57

5 of 57

6 of 57

7 of 57

8 of 57

9 of 57

wip

10 of 57

wip

11 of 57

wip

12 of 57

wip

13 of 57

wip

14 of 57

15 of 57

wip

16 of 57

wip

17 of 57

wip

18 of 57

wip

19 of 57

20 of 57

21 of 57

22 of 57

23 of 57

24 of 57

25 of 57

26 of 57

27 of 57

28 of 57

29 of 57

wip

Add talks here

30 of 57

31 of 57

32 of 57

33 of 57

34 of 57

35 of 57

36 of 57

37 of 57

38 of 57

39 of 57

Motivation & Design

40 of 57

Motivation

Persistence of archived Web data dependent on resilience of organization and availability of data
Remove massive redundancy in Web archive files of exact duplicate content
Determine feasibility of pushing WARCs into IPFS

41 of 57

Design

Extending the CDXJ Format
Indexing and IPFS Dissemination Procedure
Replay and IPFS Pull Procedure

index

replay

42 of 57

Design - CDXJ Format

com,example)/index.html 20170301192639 {"mime_type": "text/html", "status_code": "200"}�com,example)/images/frog.png 20170301192639 {"mime_type": "image/png", "status_code": "200"}

https://github.com/oduwsdl/ORS/wiki/CDXJ�Alam et al. “Web Archive Profiling Through CDX Summarization”, TPDL 2015

See:

43 of 57

Design - CDXJ Format

com,example)/index.html 20170301192639 {"locator": "urn:ipfs/QmPdyY6Pm66iWtGpTc7PqK11hvsnYSKMVL57G69RiNjGcm/QmNZ6mKSSAXAmXEocQj5gT4y4kdcr5D2C173ubWJ6PSKEZ", "mime_type": "text/html", "status_code": "200"}�com,example)/images/frog.png 20170301192639 {"locator": "urn:ipfs/QmUeko8zM7Xanwz6F9GtRH4rLAi4Poj3EMECGsci3BRQfs/QmPhMnX74cwqx2xgj9d3N3gTra8CzafXwSbUwU8xagMfqR", "mime_type": "image/png", "status_code": "200"}

44 of 57

Design

45 of 57

ipwb Design - “Indexing” Process

Extract HTTP Response from WARC

HTTP header and entity body (payload) separately

Push header and payload to IPFS, retain hashes
Construct CDXJ record containing:

URI of original resource (URI-R)
Datetime
Locator: urn:ipfs/headerHash/payloadHash

Repeat for each WARC-Response record
Save locally as CDXJ file

HTTP header block

HTTP payload block

46 of 57

Design - Replay

Identify CDXJ line w/ URI-R + datetime
Fetch content for header and payload from IPFS using locator
Reassemble content into HTTP response, serve to browser
Repeat for each embedded resource requested

47 of 57

Advancements

48 of 57

49 of 57

50 of 57

51 of 57

52 of 57

53 of 57

Privacy, Collaboration, and Security

Encryption on indexing/dissemination, decryption on replay��

com,mywebsite)/photos/vacation 20170605083914 {

"locator": "urn:ipfs/QmdmV...P9Hf/QmRDB...1Bz2P",

"encryption_method": "xor", “encryption_key”: � “my#Gre4t#Encrypti0n#K3y!”, "mime_type": "text/html", "status_code": "200"}

54 of 57

Privacy, Collaboration, and Security

IPWB CDXJs may be transferred for our users’ replay��
CDXJ-by-hash recursive fetch/replay

Share hash of CDXJ then $ ipwb replay hash to replicate experience

index

replay

...

QmVvshF...dXJ3

Push CDXJ to IPFS

55 of 57

Other ipwb Advancements

Rerouting (instead of Rewriting) for Archival Replay*

IPWB replay registers ServiceWorker

Intercepts requests from archival replay to live Web

Prevents live Web from “leaking into” the archive on replay�

Memento Support

Replay system serves TimeMap, Timegate, and Datetime (memento) resolution endpoints
http://localhost/timemap/http://mywebsite.com/photos/vacation
http://localhost/memento/20170605092450/http://mywebsite.com/photos/vacation

* To be presented at JCDL 2017 in Toronto, Canada, June 19-23, 2017

56 of 57