W3C WebRTC
WG Meeting
February 27, 2020
8 AM Pacific Time
1
Chairs: Bernard Aboba
Harald Alvestrand
Jan-Ivar Bruaroey
W3C WG IPR Policy
2
Welcome!
3
About this Virtual Meeting
Information on the meeting:
4
KITE Test Results (Dr. Alex)
5
KITE Test Results (Dr. Alex) - Browsers
Lots of work. Critical mass only in EU. Difficult to maintain (cosmo effort). Needs update => IETF 107. Madrid better (july).
6
KITE Test Results (Dr. Alex) - SFUs
7
Issues for Discussion Today
8
Issues for Discussion Today
9
Issue 642: Turn off device on disabled track. (Jan-Ivar)
Camera & mic may be temporarily disabled using track.enabled. Sole use case:
micMute.onclick = () => audioTrack.enabled = !micMute.checked;
endCall.onclick = () => pc.close();
camMute.onclick = () => videoTrack.enabled = !camMute.checked;
This is semantically complete (says it all) & works in all browser permission models.
But some browsers don’t turn off the camera HW light...
Users demand privacy.
Better privacy on camera mute in Firefox 60
👁 😬
Issue 642: Turn off device on disabled track. (Jan-Ivar)
Sites want this behavior! Here’s what they do to work around it (a good example of where a spec not being explicit about behavior causes web compat headaches).
camMute.onclick = async () => { // Our users demand camera HW light off in other browsers. Let’s hack!
try {
videoTrack.enabled = !camMute.checked;
await wait(3000); // because users
if (camMute.checked == !videoTrack) return;
if (camMute.checked) {
videoTrack.stop(); // kill the light!
} else if (videoTrack && videoTrack.readyState != “live”) { // revive the camera!
selfView.removeTrack(videoTrack);
videoTrack = null; // avoid racing with ourselves
videoTrack = (await navigator.mediaDevices.getUserMedia(stashedConstraints)).getVideoTracks()[0]);
await micTransceiver.sender.replaceTrack(videoTrack);
}
} catch (e) { /* 🤷🏼♂️ */ }
}
😢
🤮
Refresher on how Firefox works (fiddle):
Proposal: Enforce web compat around this behavior in all browsers
Spec today¹: ”when a track becomes either muted or disabled, and this brings all tracks connected to the device to be either muted, disabled, or stopped, then the UA MAY, using the device's deviceId, deviceId, set [[devicesLiveMap]][deviceId] to false”
1) This is only a close approximation to hardware lights, inasmuch as having physical and logical “privacy indicators” align is POLA.
12
MUST
Issue 655: How to avoid wide-lens & telephoto on new phones (jib)
Flagship phones have multiple back cameras now. How to distinguish wide-lens?�Wide-lens usually means fixed-focus, represented by 0 in android, so might this work?
{video: {focusDistance: {exact: 0}}} // pick wide-lens video camera� {video: {focusDistance: {min: 0.0001}}} // avoid wide-lens video cameras
But relies on a side-effect: Apparently no rule says all wide lenses have fixed focus.
Also, what if the phone has wide-lens & telephoto? E.g. Samsung S10 specs:
AFAICT these are 35mm equivalent focal lengths, “a measure that indicates the angle of view”, because photography.
🤷🏼♂️
Issue 655: How to avoid wide-lens & telephoto on new phones (jib)
Proposal A: A new (“35mm equivalent”) focalLength constraint:�� {video: {focalLength: {min: 0.026}}} // avoid all wide-lenses < 26mm� {video: {focalLength: {min: 0.052}}} // pick telephoto�
Proposal B: A new angleOfView constraint:�� {video: {angleOfView: {max: 68}}} // avoid all wide-lenses < 26mm� {video: {angleOfView: {max: 38}}} // pick telephoto
Conversions: = 67.8872 = 37.2
Bonus Q: Where to specify this? Mediacapture-main or Mediacapture-image?
Issue 652: In-browser device picker (Jan-Ivar)
In 2020, exposing all your devices to the web beyond the one you’re using, is not POLA. It goes beyond fingerprinting, revealing actual private information users did not intend to share about what they own and have plugged in. This may include devices personalized with names, high-entropy prototype devices, perhaps even embarrassing (adult) devices�
It’s not "the minimal information needed to achieve user goals".
PING wants privacy-by-default in-browser device picker:
Web developers need this API to work in all browsers before it’s usable.
Issue 652: In-browser device picker (Jan-Ivar)
The TAG joins PING in wanting this. w3ctag/design-principles#152:
Discourage device enumeration, prefer less powerful alternatives
“The Rule of Least Power suggests that we should go with the least powerful API which meets its use cases.”
“we ... recommend that, when API designers need to expose devices in some manner, they consider [in order]:
Issue 652: In-browser device picker (Jan-Ivar)
On availability-style API: Nothing prevents UAs from letting users customize browser default cam/mic & expose a single device per kind. No spec work seems needed for this.
On picker-style API: How far do we go?
Goal 1: Get rid of in-content device selection & label in enumerateDevices #640
Goal 2: Somehow prevent browsers from granting all devices of a kind by default.
Let’s focus on goal 1. We may never reach consensus on goal 2.
What’s a minimal approach? What are the minimal API changes needed?
Issue 652: In-browser device picker — minimal approach (Jan-Ivar)
Use cases. New visitor: (browsers that do per-device grants will need a picker)
await navigator.mediaDevices.getUserMedia({video: true});
await navigator.mediaDevices.getUserMedia({video: {facingMode: “environment”}});
await navigator.mediaDevices.getUserMedia({video: {width: {min: 1280}}});
Repeat visitor: (picker undesired, unless previous device removed and multiple choices)
await navigator.mediaDevices.getUserMedia({video: {deviceId}});
await navigator.mediaDevices.getUserMedia({video: {deviceId: {exact: deviceId}}});
await navigator.mediaDevices.getUserMedia({video: true});
Changing (or adding a 2nd) camera view in a world without labels: (picker is required)
await navigator.mediaDevices.getUserMedia({video: {existingDeviceId}});
await navigator.mediaDevices.getUserMedia({video: {existingDeviceId: [...ids]}});
Lazy getter, or don’t know about stream.clone(): (picker would be web compat bug) ❌
await navigator.mediaDevices.getUserMedia(sameConstraints);
Issue 652: In-browser device picker — minimal approach (Jan-Ivar)
New visitor/repeat visitor: Firefox already has an in-browser device picker for this.
await navigator.mediaDevices.getUserMedia({video: true, audio: true});
Lists {min … max}, w/ideal chosen by default.
But it only appears if permission is absent.
Strong emphasis on permission and defaults.
No known obstacles to other browsers adding something comparable if they want to.
Issue 652: In-browser device picker — minimal approach (Jan-Ivar)
Changing camera: We need an in-browser picker shown regardless of permission:
button.onclick = async () => {
if (numberOfVideoInputDevices < 2) return;
const constraints = {
video: {existingDeviceId: cameraTrack.getSettings().deviceId}
};
cameraTrack = (await navigator.mediaDevices.getUserMedia(constraints)).getVideoTracks()[0];
button.innerText = cameraTrack.label;
}
A new existingDeviceId constraint
tells UA this is a request for a device
other than the (non-ideal) specified,
and it always prompts the user.
(It’s up to UX whether to default to or
exclude the specified device as a choice.�We dunno if JS intends to replace or supplement it)
Share another camera�jan-ivar.github.io wants to share a different camera. Choose which camera to share.
Issue 652: In-browser device picker (Jan-Ivar)
Pros (of minimal API):
Cons:
Issue 652: In-browser device picker (Jan-Ivar)
Better booleans: chooseUserMedia unless we change the semantics of getUserMedia.
Option A: Morph already-expressive gUM w/transition plan inspired by {sdpSemantics}:
await navigator.mediaDevices.getUserMedia({video: true, semantics: “browser-chooses”});
await navigator.mediaDevices.getUserMedia({video: true, semantics: ”user-chooses”});
Step 1: Implement picker in all browsers
Step 2: Announce {semantics} and default to “browser-chooses” in all browsers
Step 3: Sites prepare to avoid redundant pickers (preferably using exact, clone())
Step 4: Flip default to “user-chooses” in all browsers
Step 5: (Optional) Deprecate {semantics} in all browsers (mention caveat)
Option B: Concede status quo with above syntax. Option C: Concede status quo with:
await navigator.mediaDevices.getUserMedia({video: true}); // browser chooses
await navigator.mediaDevices.chooseUserMedia({video: true}); // user chooses
Issue 652: In-browser device picker (Jan-Ivar)
Issue 640: Only reveal labels of devices user has given permis.. (Youenn)
If difficult to enforce per-device exposure rule, enforce per-device-type exposure
Still possible to use groupId to get microphone corresponding to camera
Issue 639: Enforcing user gesture for getUserMedia (Youenn)
26
Extensions for Discussion Today
27
Insertable Streams (Harald)
28
Insertable Streams - WebIDL API
partial dictionary RTCConfiguration {
boolean forceEncodedVideoInsertableStreams = false;
boolean forceEncodedAudioInsertableStreams = false;
};
partial interface RTCRtpSender {
RTCInsertableStreams createEncodedVideoStreams();
RTCInsertableStreams createEncodedAudioStreams();
};
partial interface RTCRtpReceiver {
RTCInsertableStreams createEncodedVideoStreams();
RTCInsertableStreams createEncodedAudioStreams();
};
dictionary RTCInsertableStreams {
ReadableStream readable;
WritableStream writable;
};
enum RTCEncodedVideoFrameType {
"Empty", "key", "delta",
};
interface RTCEncodedVideoFrame {
readonly attribute RTCEncodedVideoFrameType type;
readonly attribute unsigned long long timestamp;
attribute ArrayBuffer data;
readonly attribute ArrayBuffer additionalData;
};
29
Insertable Streams - example use
let senderTransform = new TransformStream({
async transform(chunk, controller) {
let view = new DataView(chunk.data);
let newData = new ArrayBuffer(chunk.data.byteLength + 4);
let newView = new DataView(newData);
// Invert and pad the bits in the frame
for (let i = 0; i < chunk.data.byteLength; ++i)
newView.setInt8(i, ~view.getInt8(i));
// Set the padding bytes to zero.
newView.setInt8(chunk.data.byteLength + i, 0);
chunk.data = newData;
controller.enqueue(chunk);
},
});
let senderStreams = videoSender.getEncodedVideoStreams();
// After ICE and offer/answer exchange.
senderStreams.readable
.pipeThrough(senderTransform)
.pipeTo(senderStreams.writable);
30
Insertable Streams - Relation to other efforts
WebCodec
TransferableStreams
31
Insertable Streams - Change from previous API proposal
32
Insertable Streams - Next Steps
33
Capture Timestamp (Henrik)
Problem 1: How to calculate the end-to-end delay?
34
RTCRtpContributingSource.timestamp tells you time of playout.
Just need time of capture…
Capture Timestamp (Henrik)
Problem 1: How to calculate the end-to-end delay?
RTP header extension abs-capture-time gives you…
Problem 2: How to measure A/V sync?
35
Capture Timestamp (Henrik)
Example!
36
Problem:
Capture Timestamp (Henrik)
Example!
With clockOffset you can calculate captureTimestamp in receiver’s clock, ergo you can calculate e2eDelay.
37
Capture Timestamp (Henrik)
Example!
38
Capture Timestamp (Henrik)
But this doesn’t work if there’s a middlebox! :’(
39
clockOffset transforms sender’s clock to receiver’s clock.
But captureTimestamp is in capturer’s clock.
Capture Timestamp (Henrik)
abs-capture-time’s estimatedClockOffset gives us the sender’s estimate of the clockOffset between the capturer’s clock and the sender’s clock. Thus:
sender’s captureTimestamp
=�capturer’s captureTimestamp + estimatedClockOffset
This works regardless of how many “hops” between the capturer and the receiver, by adding to estimatedClockOffset each “hop”.
40
Capture Timestamp (Henrik)
For more info, see webrtc-extensions PR#33 which added:
Based on abs-capture-time RTP header extension:
41
RTP Header Extensions (Harald)
This section:
a=extmap:14 urn:ietf:params:rtp-hdrext:toffset
a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:13 urn:3gpp:video-orientation
a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:12 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay
a=extmap:11 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type
a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-timing
a=extmap:8 http://tools.ietf.org/html/draft-ietf-avtext-framemarking-07
a=extmap:9 http://www.webrtc.org/experiments/rtp-hdrext/color-space
a=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid
a=extmap:5 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id
a=extmap:6 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id
The need:
42
RTP header extensions (Harald)
43
Content-Hints (Harald)
Status update!
44
For extra credit
45
Name that bird!
Thank you
Special thanks to:
WG Participants, Editors & Chairs
The bird
46