Recording high-fidelity online news sites with Webrecorder

Dodging the Memory Hole�November 15, 2017

Ilya Kreymer and Anna Perricci

About Webrecorder

Create high-fidelity, interactive captures of any web pages you browse

http://webrecorder.io

Webrecorder Player App

Webrecorder Project key facts

• Robust tools
• Free to use
• Fully open source
• Using open standards
• Growing user community
• Quickly evolving

A project by

with generous support from

Webrecorder Team

Why is it hard to preserve online news?

• News sites push the limits of current technology�
• Interactive content!�
• Lots of video/audio and ad content�
• Traditional crawlers can’t run Javascript�
• Rapidly changing

High fidelity web archiving

• Record any web page loaded in the browser

• Archive interactive content (only available after user input)

​

• Same system for recording and playback (web browser)

Webrecorder creates high fidelity web archives including

elements that crawler based systems

often fail to capture

such as interactive content

Collecting at human scale

• Webrecorder: web archiving for all!�
• Collecting is done by a person via web browser one page at a time�
• Can import and augment collections created by crawlers

​

The payoff for careful capture is an �accurate representation of the original

​

What about social media?

• Webrecorder can capture content from social media sites, and works especially well with Instagram and Twitter�
• Some websites deliver content individualized for each user
• Webrecorder can record the content you see when you are logged in to a social media profile

Account login is optional

• One does not need to login to use Webrecorder to capture web content
• Users can download the captures right away (as a WARC file) & save them locally

​

• For continued access to archived content online & to be able to add to a collection, one must log in to a free account

​

Access & sharing options

​

• User created collections can be kept private or made public through Webrecorder.io

​

• Public collections can be viewed by anyone
• Finer access controls are being considered

​

Preconfigured browsers

• Using a preconfigured browsers to capture and replay web content that may not be supported in future web browsers
• e.g. Flash�
• Access with a preconfigured browser ensures greater faithfulness to the original look and feel of web pages
• Browsers use HTTP proxy mode = even better fidelity

Webrecorder Player

• Desktop application for OSX, Window and Linux�
• User friendly application to browse any web archive (saved in standard WARC format)�
• Can browse web archives offline, no internet connection required!

Did you happen to hear about what happened with Gothamist and DNA Info?

2 ways Webrecorder can help

1.�Content on the live web can be captured, saved & accessed (Webrecorder.io). These web archives can also be downloaded for offline access

2. �Archived web content can be imported into Webrecorder (via extraction from other web archives). Webrecorder collections can be saved in Webrecorderio.io or downloaded for offline access

Preparing for further outreach

​

• Preparing for upcoming trainings for journalists on
• capturing their own finished work
• using web archiving to capture source materials (to ease or alleviate some link rot issues)�
• Full set of user documentation to be published in 2018 following user interface redesign

Demo!

Webrecorder Player - NEW Release Preview

• Uses HTTP/S Proxy Mode for better fidelity
• Support for Flash plugin built-in
• Cache WARC indexes for faster reloads
• New Look and Side bar
• Coming Soon!

​

Open Source Tools

• Webrecorder: https://github.com/webrecorder/webrecorder
• Webrecorder Player:

https://github.com/webrecorder/webrecorderplayer-electron/releases

• Core Replay Engine: https://github.com/ikreymer/pywb

​

Webrecorer Sample Collections

https://webrecorder.io/wrsc

Next steps for Webrecorder

• Wrapping up current major Mellon funding
• Further technical and organizational development in progress
• Ethics and Archiving the Web
• March 22-24, 2018 in New York City
• Last chance to apply to participate: http://rhizome.org/editorial/2017/oct/24/open-call-national-forum-on-ethics-and-archiving-the-web/
• Stay tuned for further developments with Webrecorder

Q & A

Using Webrecorder Today

​

Use Our Hosted Service

• Sign-up at https://webrecorder.io/ for a free account

Run your own Webrecorder instance:

• Install from https://github.com/webrecorder/webrecorder-deploy

Download Webrecorder Player for your Desktop:

• Download from https://github.com/webrecorder/webrecorderplayer-electron

A project by

with generous support from

Thank you

additional outreach support