W3C WebRTC
WG Meeting
April 27, 2021
8:00 - 10:00 AM Pacific Time
1
Chairs: Bernard Aboba
Harald Alvestrand
Jan-Ivar Bruaroey
W3C WG IPR Policy
2
Welcome!
3
About this Virtual Meeting
4
Issues for Discussion Today
5
Dr. Alex Gouaillard
It is with a heavy heart that we must give you the news that our friend, colleague and leader Dr. Alex passed away on Thursday, April 8th 2021. He was involved in a motor vehicle accident in Thailand that took his life.
All of us are in shock. And for anyone who knew him, whether for weeks, months or years, he was a brilliant human being and as memorable a personality as you will meet in life. He will forever have a place in our hearts.
Dr. Alex was among the largest contributors and advocates for WebRTC technology. And although we are still mourning his loss, we will honour his memory by working to fulfill his vision for the future of WebRTC.
6
WebRTC-PC Extension Process (Harald)
Proposal:
7
Test Proposal from Two Meetings Ago
8
Status Update
9
Issues for Discussion Today
10
PR 37: requestKeyFrame API (Bernard)
11
MediaCapture Transform:
Proposal to adopt
Status of Implementation
MediaCapture Transform Issue 4
Is Readable/WritableStream the right approach?
We believe it is.
14
MediaCapture Transform Issue 6
Memory management for incoming frames.
What if the upstream track produces more frames than can be consumed?
Existing mechanisms:
MediaStreamTrackProcessor also has an input buffer whose maximum size can be set by the application.
15
MediaCapture Transform Issue 20
Add "real-time" warning/note to MediaStreamTrackGenerator
What if a MediaStreamTrackGenerator (which is a track) generates more frames than the sinks can consume. How to notify the application about it?
Potential solutions:
16
MediaCapture Transform Issue 1
Add a high-quality face/body tracking API to discourage poor/discriminatory implementations
17
Proposal: Adopt the spec as a WG document
What WG Adoption Means
Feedback on Transform (Youenn)
A transform for both audio and video?
Media capture transform vs. Audio worklet
Proposal: focus solely on raw video
19
Feedback on Transform (Youenn)
Processing ideally done off the main thread
Proposal: envision off-the-main thread alternatives, for instance:
20
Feedback on Transform (Youenn)
How could it look like?
21
// main-page.js�const worker = new Worker('my-head-detection-worker.js');�const stream = await navigator.mediaDevices.getUserMedia({ video: true);�const track = stream.getVideoTracks()[0].clone();�worker.postMessage({type:'head', track: track}, [track]);�worker.onmessage = (event) => console.log(event.data));
// my-head-detection-worker.js�async function computeHeadPosition(frame)�{� ...�}�onmessage = (e) => {� if (e.data.type === 'head+encode') {� const encoder = new VideoEncoder({ output : (data) => { ... } });� const track = e.data.track;� track.onmute = (event) => { self.postMessage('head detection disabled');� track.onunmute = (event) => { self.postMessage('head detection enabled');� track.onframe = (event) => {� const position = await computeHeadPosition(event.frame); � if (position) self.postMessage('we got a head at ' + position);� encoder.encode(event.frame);� };� }�};
Frame hopping from camera thread to worker thread without ever going through main thread
Feedback on Transform (Youenn)
Existing APIs heavily based on MediaStreamTrack
MediaStreamTrack and WebCodecs
22
Feedback on Transform (Youenn)
How could it look like?
23
// encode path�const stream = await navigator.mediaDevices.getUserMedia({ video: true });�const encoder = new VideoEncoder({ output : (data) => {� transferEncodedData(data);�} });�encoder.encode(stream.getVideoTracks()[0]);�
�// decode path�const decoder = new VideoDecoder({ type: 'track', output : (track) => {� video.srcObject = new MediaStream([track]);�});
Frame hopping from camera thread to web codec thread without ever going through main thread
Frame hopping from web codec thread to video rendering thread without ever going through main thread
Feedback on Transform (Youenn)
The controlling channel
24
Feedback on Transform (Youenn)
Potential issues with control signals current API
25
Feedback on Transform (Youenn)
What about additional downlink signals
Example
26
Feedback on Transform (Youenn)
Should we expose frames using a ReadableStream?
27
Feedback on Transform (Youenn)
Tentative summary
28
Feedback on Transform (Jan-Ivar)
What Youenn said, except adding streams to his examples, for back-pressure.
Back-pressure != buffering. For real-time too regardless of drop vs buffer strategy.
29
PR 64: Remove redundant API (Youenn)
30
Issue 99: Use WebCodec APIs (Youenn)
Two APIs for very similar concepts
Main differences
Two options
31
#155 getViewportMedia (Jan-Ivar)
Agreement! (Consensus?)
Agree on: Shape of opt-in-to-capture doc header.
Problems w/COEP: require-corp; html-capture. Opts into riskier (not safer) profile (confusing). Requires Fetch Metadata request header as well (Sec-Fetch-COEP)�
Proposal from @camillelamy: Off-by-default Document policy (WICG)
32
getViewportMedia: Mitigate User information harvesting
getViewportMedia: cropping
Proposal: Capture intersection of viewports of TLBC’s active document & iframe.
Transfer MediaStreamTrack
instead of allowing capture�by others. Lets top-level page�delegate iframe capture to�iframes without opting into
being captured itself. Seems�most conservative (to start).
Solves cropping at the target. �Avoids passing IDs or�cropping coordinates.�Fine-grained enough.
Page
<iframe>
<script>
await navigator.mediaDevices.getViewportMedia();
CSS
Benefits of getViewportMedia from Arbitrary Frame (Elad)
Consider an application that combines presentations and VC. One frame each.
The VC frame captures the entire tab and crops itself away.
This allows the VC app to use restrictOwnAudio to exclude its own audio from the capture.
The VC can now transmit the captured media content without creating an audio feedback loop.
35
Media Content
Presentation Software
VC
Capture Handle - Exposition
36
Capture Handle - Puzzle Pieces
37
Capture Handle - Possible Hack
What could we do with no changes to the browser?
Problems?
---
(1) QR code used to keep the illustration simple. Similar solutions yield similar problems.
38
Capture Handle - Suggested Solution
On the captured app:
On the capturing app:
39
Capture Handle - Suggested Solution (Captured App)
// Code on Slides-3000
navigator.mediaDevices.setCaptureHandleConfig({
exposeOrigin: true,
handle: JSON.stringify({
description: "See slides-3000.com for our API description. " +
"It supports such amazing actions as nextSlide, " +
"prevSlide, goFullScreen, etc.",
sessionId: mySessionId(),
})
});
40
Capture Handle - Suggested Solution (Capturing App)
// Code on VC-MAX
const mediaStream = await navigator.mediaDevices.getDisplayMedia();
const captureHandle =
mediaStream.getVideoTracks()[0].getSettings().captureHandle;
if (captureHandle.origin == "preso.slides-3000.com") {
const sessionId = JSON.parse(captureHandle.handle).sessionId;
Let commsChannel = ...; // Specific to Slides-3000.
commsChannel.postMessage(sessionId, "goFullScreen");
...
} else if (captureHandle.origin == “avoid.doodle.com”) {
...
}
41
Capture Handle - Interesting Note
Application don’t actually have to expose their origin. If the session ID is sufficiently unguessable, a separate API could be set up exposing validate().
const captureHandle =
mediaStream.getVideoTracks()[0].getSettings().captureHandle;
if (slides3000Helper.validate(captureHandle)) {
...
}
If a site exposes its origin, though, an RTT to the identification service could be saved.
Note that the captured site chooses to allow exposing the origin; the browser sets the actual value to be read on the capturing side.
42
Capture Handle - Stop Worrying, Love the Bomb.
Privacy/Security concerns? Probably not.
43
Screenshot API - Who? Why?
Who takes screenshots of the current tab’s viewport?
Why? Often - to gather user feedback on defects.�Is it important? Depends - is customer-retention important? (Yes it is!)
44
Most Web-based Google Products
Sites using html2canvas
Screenshot API - State of the Art
45
Screenshot API - Workaround #1 (gDM)
Idea: Use getDisplayMedia, grab a single frame.
Deficiencies:
46
Screenshot API - Workaround #2 (gVM)
Idea: Use getViewportMedia, grab a single frame.
Deficiencies:
47
Screenshot API - Workaround #3 (DOM Redrawing)
Idea: Walk the DOM and redraw it onto a Canvas.
Popular solution:
Deficiencies:
48
Screenshot API - Workaround #4 (Extensions)
Idea: Extensions can take a screenshot...
Deficiencies:
49
Screenshot API - Workaround #5 (Manual)
Idea: Ask the user to upload their own screenshot.
Deficiencies:
50
Screenshot API - Goals and Non-Goals
Non-Goals
Goals
51
Screenshot API - Security Requirements 1/2
There is consensus that getViewportMedia is safe enough if gated by:
This is an upper-bound on the requirements for a screenshot API.
Is the bound tight? Can we relax any of these using replacements?
52
Screenshot API - Security Requirements 2/2
Common-sense requirements:
Can adding any of the following stand in for COI and opt-in headers?
The above mechanisms aren’t bulletproof, but maybe in combination…?
53
Conditional Audio Suppression
Suppose this meeting weren’t virtual. We’d all be sitting in a room, taking turns presenting from our laptops to a projector and a set of speakers. The next speaker presents a tab. Do they want the audio to flow out of both their laptop’s speakers as well as the PA system’s? Probably not.
Solution? Define a new constraint - suppressLocalAudioPlayback.
When this constraint is applied, audio from captured source is not played out over the local speakers.
This can also help with echo cancellation on a single-machine setup, as playing the locally captured audio in a VC as though it came from another participant simplifies things for the echo canceller.
54
For extra credit
55
Name the bird!
Thank you
Special thanks to:
WG Participants, Editors & Chairs
The bird
56