Project Fugu�Reaching parity with native
Kenneth Christiansen @kennethrohde
Web Platform Architect, Intel
Hi! I am Kenneth
@kennethrohde
@kennethrohde
@kennethrohde
@kennethrohde
@kennethrohde
I the web!
@kennethrohde
The web is awesome
Great reach, low friction
safe
ephemeral
(deep) linkable
indexable
composable
@kennethrohde
@kennethrohde
@kennethrohde
The Web Platform is an amazing place to be
@kennethrohde
@kennethrohde
Audio & Video Capture
Advanced Camera Controls
Recording Media
Real-Time Communication
USB
Credentials
Payments
Network Type & Speed
Online State
Vibration
Battery Status
Local Notifications
Push Messages
Home Screen Installation
Foreground Detection
Permissions
File Reading
Touch Settings
Speech Recognition
Pointing Device Adaptation
Offline Mode
Background Sync
Geolocation
Device Position
Device Motion
Proximity Sensors
Virtual & Augmented Reality
Fullscreen
Screen Orientation & Lock
Wake Lock
Presentation Features
Virtual & Augmented Reality
Fullscreen
Screen Orientation & Lock
Presentation Features
Vibration
Device Memory
Bluetooth
Real-Time Communication
Touch Gestures
Storage Quotas
but sometimes you
need more power
@kennethrohde
We have Web Assembly!
@kennethrohde
wasm is a new portable, size- and load-time-efficient format suitable for compilation to the web
@kennethrohde
Kind of
an efficient, low-level bytecode
for the web
@kennethrohde
@kennethrohde
Web Assembly solves some real problems!
@kennethrohde
But we are still lacking some capabilities!
@kennethrohde
How do I work efficiently with local files?
@kennethrohde
@kennethrohde
@kennethrohde
@kennethrohde
Project Fugu
@kennethrohde
@kennethrohde
@kennethrohde
Delicious if prepared correctly, deadly if not
@kennethrohde
Enable web apps to do anything native apps can, �by exposing the capabilities of native platforms to the web platform, while maintaining user security, privacy, trust, and other core tenets of the web.
@kennethrohde
Enable web apps to do anything native apps can, �by exposing the capabilities of native platforms to �the web platform, while maintaining user security, privacy, trust, and other core tenets of the web.
bit.ly/powerful-apis
@kennethrohde
crbug.com?q=proj-fugu
@kennethrohde
The process
Traditionally it takes a long time to standardize web features
https://w3ctag.github.io/explainers.md
@kennethrohde
Write explainer
Solicit feedback and iterate
Formalize spec and look for public support
Origin Trial >> Ship It
Identify need and use-cases
Features being worked on
Closing the app gap
@kennethrohde
Hardware connectivity to USB and Bluetooth
Bluetooth Low Energy (BLE) is a wireless technology aimed at applications in healthcare, fitness, beacons, security, and toys.
USB is a wired general purpose technology
Accessing such devices shouldn’t requires downloading native apps.
Early efforts, started before Project Fugu
Use Cases
Connect to toys and use them from the web.
@kennethrohde
bluetooth.rocks/batmobile
Web�Bluetooth
Watch my earlier talk
on how to use Web Bluetooth and Web USB
@kennethrohde
Web USB and�Web Bluetooth
Web USB and
Web Bluetooth
Shipped!
@kennethrohde
Identify �Need
Write �Explainer
Solicit �Feedback
Formalize �Spec
Ship�It
@kennethrohde
Accessing Serial and HID Devices
Bridge the physical world and the web by allowing web apps to communicate with devices such as microcontrollers, 3D printers, handheld barcode scanners and other peripheral devices.
Use Case
Connect to an Arduino or other microcontroller kit and program its microcontroller right from a web app.
@kennethrohde
Serial API
PhilippHenkel (https://commons.wikimedia.org/wiki/File:ArduinoUno.svg)
const requestOptions = {
// Filter on devices with the Arduino USB vendor ID.
filters: [{ vendorId: 0x2341 }],
};
// Request an Arduino from the user.
const port = await navigator.serial.requestPort(requestOptions);
// Open and begin reading.
await port.open({ baudrate: 115200 });
const reader = port.reader.getReader();
while (true) {
const {done, data} = await reader.read();
if (done) break;
console.log(data);
}
Proposed API
Serial API
@kennethrohde
Identify �Need
Write �Explainer
Solicit �Feedback
Formalize �Spec
Ship�It
Use Case
Logic for gamepads and other devices can be moved to JavaScript, providing more support than is possible from the browser.
@kennethrohde
WebHID
Use Case
The HID specification describes a wide array of devices that could be supported, including virtual reality controls, flight simulators, medical equipment, and more.
WebHID would allow these devices to be used without requiring additional drivers or modification to the browser.
@kennethrohde
WebHID
WebHID
@kennethrohde
Identify �Need
Write �Explainer
Solicit �Feedback
Formalize �Spec
Ship�It
@kennethrohde
Sharing Content and Receiving Shares
Sharing to other apps and receiving shares from other apps are core features of many experiences.
Sharing should just be universally supported, no matter if native or web.
Use Cases
The web should be able trigger shares to the system share sheet, making sharing super simple!
@kennethrohde
Web Share
if ('share' in navigator) {
const shareOpts = {
title: 'Unlocking New Capabilities for the Web',
text: 'One of the most exciting talks at I/O',
url: 'https://www.google.com',
}
navigator.share(shareOpts)
.then(() => {...})
.catch((e) => {...});
}
@kennethrohde
Receiving Shares
Native applications can act as share targets and receive content like text or files.
Web apps like social media sites should be able to register themselves as share targets to the system.
@kennethrohde
Receiving Shares
Native applications can act as share targets and receive content like text or files.
Web apps like social media sites should be able to register themselves as share targets to the system.
"share_target": {
"action": "/share-target/",
"method": "GET",
"enctype": "application/x-www-form-urlencoded",
"params": {
"title": "title",
"text": "text",
"url": "url"
}
}
Extends Web App Manifest
Web Share and Web Share Target
Shipped!
@kennethrohde
Identify �Need
Write �Explainer
Solicit �Feedback
Formalize �Spec
Ship�It
Web Share Target (v2)
In development
@kennethrohde
Identify �Need
Write �Explainer
Solicit �Feedback
Formalize �Spec
Ship�It
Use Cases
Integrate your media with the device’s control panel to make the experience smooth.
@kennethrohde
Media Session �API
Use Cases
Control video playback in full screen mode.
Use your desktop media keys to control your media without the need to focus the tab where the media is playing.
Keyboard Media Buttons
navigator.mediaSession.setActionHandler('previoustrack', () => {
// User hit "Previous Track" key.
});
navigator.mediaSession.setActionHandler('nexttrack', () => {
// User hit "Next Track" key.
});
navigator.mediaSession.setActionHandler('play', () => {
// User hit "Play" key.
});
navigator.mediaSession.setActionHandler('pause', () => {
// User hit "Pause" key.
});
Media Session API
Shipped!
@kennethrohde
@kennethrohde
Identify �Need
Write �Explainer
Solicit �Feedback
Formalize �Spec
Ship�It
@kennethrohde
Detecting Barcodes �and Faces
A lot of things in the world have barcodes that you can scan with your phone camera, and when your camera detects a face, it will highlight it with a boundary box.
Web apps should be able to tap into our devices’ built-in detectors.
Use Cases
Scanning all sorts of barcodes �(QR codes, EAN-13 codes, Aztec codes, etc.) to identify products, make payments, and many more.
@kennethrohde
Shape Detection API
Use Cases
Scanning all sorts of barcodes �(QR codes, EAN-13 codes, Aztec codes, etc.) to identify products, make payments, and many more.
@kennethrohde
Shape Detection API
Use Cases
Scanning all sorts of barcodes �(QR codes, EAN-13 codes, Aztec codes, etc.) to identify products, make payments, and many more.
@kennethrohde
Shape Detection API
if ('FaceDetector' in window) {
const faces = await new FaceDetector().detect(img);
} else {
// Use fallback
}
if ('BarcodeDetector' in window) {
const codes = await new BarcodeDetector().detect(img);
} else {
// Use fallback
}
Shape Detection API
@kennethrohde
Identify �Need
Write �Explainer
Solicit �Feedback
Formalize �Spec
Ship�It
Use Cases
Subtly alert users of incoming events like a long-running task that has finished or an app’s state, like the number of unread messages.
@kennethrohde
Badging
Mock - demonstration purposes only.
// Setting a badge
window.Badge.set(42);
// Set the badge to a simple dot on the icon
window.Badge.set();
// Alternatively, explicitly clearing it
window.Badge.clear();
window.Badge.set(0);
Badging API
@kennethrohde
Identify �Need
Write �Explainer
Solicit �Feedback
Formalize �Spec
Ship�It
@kennethrohde
Accessing the �users contacts
Native apps can easily ask for a access to a user’s contacts in their address book.
Web apps should be able to
Contacts Picker
Security Consideration
Give developers the features they need while ensuring users are in control of their own and their contacts’ information and understand what they share.
The contacts picker will only work when served from a secure host, and it can only be shown with a deliberate user gesture.
Users can also explicitly choose which contacts they want to share, and they see what is being shared before they share it.
@kennethrohde
const options = {
multiple: false,
properties: ['name', 'email', 'tel'],
};
try {
const contacts = await navigator.contacts.select(options);
// Do something with the contacts
} catch (e) {
// Handle error
}
Proposed API
[{
"email": ["kenneth.r.christiansen@intel.com"],
"name": ["Kenneth Christiansen"],
"tel": []
}]
Proposed API
Contacts Picker
@kennethrohde
Identify �Need
Write �Explainer
Solicit �Feedback
Formalize �Spec
Ship�It
Font Access API
Photo by Annie Spratt on Unsplash
Simplify use of locally installed logotype fonts, company standard fonts, or specially licensed fonts that are only available on the desktop.
Proposed API
const fonts = await navigator.fonts.query({local: true});
for (let font of fonts) {
console.log(font.family, font.postScriptName);
addFontToSelector(font);
}
Local Font Access
@kennethrohde
Identify �Need
Write �Explainer
Solicit �Feedback
Formalize �Spec
Ship�It
Spec
Coming soon
Demo
Coming soon
@kennethrohde
Reading and writing files
Web apps can open files and upload them to a server or work with them locally, but when you want to save, you have to download them and overwrite your local copy.
Web apps should be able to just read and write local files upon permission being granted.
Use Cases
Directly edit a file or multiple files without the need to upload and then download them.
@kennethrohde
File System
The File Picker
@kennethrohde
File System
Proposed API
const fileRef = await self.chooseFileSystemEntries(fileOpts);
// Read from the file
const file = await fileRef.getFile();
const image_data = await file.arrayBuffer();
// Write to the file
const writer = await fileRef.getWriter();
await writer.write(0, new_image_data);
File System
@kennethrohde
Identify �Need
Write �Explainer
Solicit �Feedback
Formalize �Spec
Ship�It
Project Fugu�Intel contributions
Kenneth Christiansen
Web Platform Architect, Intel
Rijubrata Bhaumik
Software Engineer, Intel
@kennethrohde
Generic Sensors
"The goal of the Generic Sensor API is to promote consistency across sensor APIs, enable advanced use cases thanks to performant low-level APIs, and increase the pace at which new sensors can be exposed to the Web by simplifying the specification and implementation processes."
The starting point: Existing APIs had various issues
@kennethrohde
Implemented
Generic Sensor Extra Classes flag
Implementation in Chrome
@kennethrohde
85
Grouping
@kennethrohde
Sensors are either low-level, derived or fusion (used data from multiple)
Sensor
Accelerometer
Gyroscope
Ambient Light
OrientationSensor
...
AbsoluteOrientationSensor
RelativeOrientationSensor
LinearAccelerationSensor
@kennethrohde
@kennethrohde
@kennethrohde
AmbientLightSensor is almost ready to ship
Works on Android as well as desktop, incl Windows/Edge
Use Cases:
https://w3c.github.io/ambient-light/#usecases-requirements
Privacy mitigations (Spec, Experiments)
Close to ship ALS
@kennethrohde
Learn more
@kennethrohde
@kennethrohde
Near Field Comms
Read and write to NFC tags using the NFC data exchange format (NDEF)
Old NFC demo has been rewritten from scratch
https://kenchris.github.io/webnfc-groceries/
Screenshots:
https://photos.app.goo.gl/5K7WdjyiDDRxajNv6
Next steps
@kennethrohde
93
Near Field Communication
Pretty cool, even
Sir Tim Berners-Lee is excited
@kennethrohde
@kennethrohde
Web NFC demo
Proposed API
const reader = new NFCReader();
reader.onreading = event => {
for (let record of event.message.records) {
// ...
}
};
const ctr = new AbortController;
reader.scan({ signal: ctr.signal });
Proposed API
const reader = new NFCReader();
reader.onreading = event => {
for (let record of event.message.records) {
if (record.recordType === "json") {
console.log(`JSON: ${record.toJSON().myProperty}`);
}
if (record.recordType === "opaque" && record.mediaType.startsWith('image/')) {
const blob = new Blob([record.toArrayBuffer()], {type: record.mediaType});
const img = document.createElement("img");
img.src = URL.createObjectURL(blob);
img.onload = () => window.URL.revokeObjectURL(this.src);
document.body.appendChild(img);
}
}
}
};
const ctr = new AbortController;
reader.scan({ signal: ctr.signal });
Proposed API
const writer = new NFCWriter();
writer.push({
records: [
{
recordType: "json",
mediaType: "application/json",
data: {
name: "Benny Jensen",
title: "Banker"
}
},
{
recordType: "json",
mediaType: "application/json",
data: {
name: "Zoey Braun",
title: "Engineer"
}
}]
}, {target: "tag", ignoreRead: false});
NFC
@kennethrohde
Identify �Need
Write �Explainer
Solicit �Feedback
Formalize �Spec
Ship�It
Spec
bit.ly/33AySh2
Demo
bit.ly/2ZdaNgO
@kennethrohde
Keeping an App Alive
Native apps can use wake locks to prevent a device from sleeping or to keep a device’s screen on.
Web apps should be able to do �the same.
Use Cases
Prevent a device from sleeping so a navigation app can get you from point A to point B and can notify you of turns you need to take.
Keep the screen of a device on while playing chess, so you can think about your next move as long as you want.
Wake Lock
@kennethrohde
Wake Lock
Two different kinds �of wake locks
A screen wake lock prevents the device’s screen from turning off so that the user can see the information that’s displayed on screen.
A system wake lock prevents the device’s CPU from entering standby mode so that your app can continue running.
@kennethrohde
Wake Lock
Keep a device awake
The Wake Lock API provides two types of wake locks: screen and system. One may imply the effects of the other. For example, a screen wake lock implies that the app—and therefore the system—should �continue running.
function tryKeepScreenAlive(minutes) {
navigator.wakeLock.request("screen").then(lock => {
setTimeout(() => lock.release(), minutes * 60 * 1000);
});
}
tryKeepScreenAlive(10);
Wake Lock
@kennethrohde
Identify �Need
Write �Explainer
Solicit �Feedback
Formalize �Spec
Ship�It
Capture media
Be able to capture media from video feed as well as add filters etc
Media Capture extensions
Mostly driven by my co-workers Alexandra Zhuravleva and Rijubrata Bhaunik who have also made a lot of demos
https://github.com/riju/WebCamera
@kennethrohde
Media Capture extensions
Media Stream Image Capture
@kennethrohde
Identify �Need
Write �Explainer
Solicit �Feedback
Formalize �Spec
Ship�It
Explainer
Coming soon
Updates Post
https://developers.google.com/web/updates/2016/12/imagecapture
Feedback
Media Capture depth extensions
@kennethrohde
Media Capture depth extensions
@kennethrohde
Media Capture depth extensions
Media Capture Depth Stream Extensions
@kennethrohde
Identify �Need
Write �Explainer
Solicit �Feedback
Formalize �Spec
Ship�It
Audio & Video Capture
Advanced Camera Controls
Recording Media
Real-Time Communication
USB
Credentials
Payments
Network Type & Speed
Online State
Vibration
Battery Status
Local Notifications
Push Messages
Home Screen Installation
Foreground Detection
Permissions
File Reading
Touch Settings
Speech Recognition
Pointing Device Adaptation
Offline Mode
Background Sync
Geolocation
Device Position
Device Motion
Proximity Sensors
Virtual & Augmented Reality
Fullscreen
Screen Orientation & Lock
Wake Lock
Presentation Features
Virtual & Augmented Reality
Fullscreen
Screen Orientation & Lock
Presentation Features
Vibration
Device Memory
Bluetooth
Real-Time Communication
Touch Gestures
Storage Quotas
Existing
Audio & Video Capture
Advanced Camera Controls
Recording Media
Real-Time Communication
USB
Credentials
Payments
Network Type & Speed
Online State
Vibration
Battery Status
Local Notifications
Push Messages
Home Screen Installation
Foreground Detection
Permissions
File Reading
Contacts
SMS
Serial
Task Scheduling
Touch Settings
Speech Recognition
Pointing Device Adaptation
Offline Mode
Background Sync
Inter-App Communication
NFC
Geolocation
Geofencing
Device Position
Device Motion
Proximity Sensors
Virtual & Augmented Reality
Fullscreen
Screen Orientation & Lock
Wake Lock
Presentation Features
Wake Lock
Proximity Sensors
Ambient Light
Virtual & Augmented Reality
Fullscreen
Screen Orientation & Lock
Presentation Features
Task Scheduling
Vibration
Device Memory
Bluetooth
Real-Time Communication
Touch Gestures
Run on Startup
Clipboard Access
Unreliable Socket Communications
Storage Quotas
Clipboard Access
Unlimited Storage
WebHID (HID)
App Icon Badging
Local Fonts Access
2020
@kennethrohde
developers.chrome.com/origintrials
@kennethrohde
bit.ly/new-fugu-request
Tentative launches for in-flight APIs
M73
M72
M74
M75
M79
M76
M77
M78
Q1
Q2
Q3
Q4
Image Clipboard Support
Shape Detection
App Icon Badging
Wake Lock
Contacts Picker
Native File System Access
Phone Number Sign Up
Unlimited Quota
Scheduled Notifications
Web Serial / Web HID
Local Font Access
File Handling
M75
M76
M77
M78
M79
2019
OT
Updated plan from week 30
Resources
Web Capabilities codelab: codelabs.developers.google.com/codelabs/web-capabilities
Capabilities landing page: developers.google.com/web/updates/capabilities
@kennethrohde
Thank you!
Thanks to Pete LePage and Thomas Steiner for allowing me to borrow some of their slides
@kennethrohde