W3C WebRTC
WG Meeting
March 15, 2022
8 AM - 10 AM
1
Chairs: Bernard Aboba
Harald Alvestrand
Jan-Ivar Bruaroey
W3C WG IPR Policy
2
Welcome!
3
About this Virtual Meeting
4
W3C Code of Conduct
5
Virtual Interim Meeting Tips
This session is (still) being recorded
6
Understanding Document Status
7
Poll about TPAC 2022
8
Issues for Discussion Today
Time control:
9
WebRTC-SVC &
WebRTC-Extensions (Bernard)
Start Time: 8:10 AM
End Time: 8:30 AM
10
Issues/PRs for Discussion
WebRTC-SVC
WebRTC-Extensions
11
Issue 68: Clarify behavior of getParameters()
Before negotiation has completed, getParameters() returns the scalabilityMode value for each encoding in encodings, assuming it was successfully set by addTransceiver() or setParameters(). If no scalabilityMode value was provided for an encoding in encodings, or if a value was not successfully set, then getParameters() will not return a scalabilityMode value for that encoding.
After negotiation has completed, getParameters() returns the currently configured scalabilityMode value for each encoding in encodings. This may be different from the values requested in addTransceiver() or setParameters(). If the configuration is not satisfactory, setParameters() can be used to change it.
If an encoding in encodings had no scalabilityMode value provided to addTransceiver() or setParameters(), getParameters() returns the default scalabilityMode of the most preferred codec. The most preferred codec and the default scalabilityMode for each ecodec are both implementation dependent. The default scalabilityMode SHOULD be one of the temporal scalability modes (e.g. "L1T1","L1T2","L1T3", etc.).
12
PR 69: Clarify behavior of getParameters()
Before the initial negotiation has completed, getParameters() returns the scalabilityMode value for each encoding in encodings, as last set by addTransceiver() or setParameters(). If no scalabilityMode value was provided for an encoding in encodings, or if a value was not successfully set, then getParameters() will not return a scalabilityMode value for that encoding.
After the initial negotiation has completed, getParameters() returns the currently configured scalabilityMode value for each encoding in encodings. This may be different from the values requested in addTransceiver() or setParameters(). If the configuration is not satisfactory, setParameters() can be used to change it.
If addTransceiver() or setParameters() did not provide a scalabilityMode value for an encoding in encodings, then after the initial negotiation has completed, getParameters() returns the default scalabilityMode of the most preferred codec for that encoding. The most preferred codec and the default scalabilityMode for each ecodec are both implementation dependent. The default scalabilityMode SHOULD be one of the temporal scalability modes (e.g. "L1T1","L1T2","L1T3", etc.).
13
Issue 98: Disabling hardware acceleration
14
Issue 98: Potential Approaches
partial dictionary RTCRtpCodecCapability {
HardwareAcceleration hardwareAcceleration = "no-preference";
};
15
Issue 98: Potential Approaches (cont’d)
16
Issue 99: Should RTCRtpHeaderExtensionCapabilities offer an “enabled” member? (Harald)
17
Discussion (End Time: 8:30 AM)
18
Avoiding the “Hall of Mirrors” (Elad)
Start Time: 08:30 AM
End Time: 09:00 AM
19
Hall of Mirrors - Reminder (Elad)
The “Hall of Mirrors” effect can be observed when an application captures a surface, then draws it back to the area being captured.
20
Hall of Mirrors - Problem (Elad)
Users can accidentally trigger one of three variants of the HoM effect:
Many video-conferencing applications display a preview of the captured content back to the user, triggering the HoM.
Adverse effects include:
21
Hall of Mirrors - Rejected “Solution” (Elad)
Rejected: “Maybe the user agent should just exclude the current tab from the list of available tabs?”
Note that this (rejected) solution exclusively focuses on tab-capture.
This would break legitimate applications which intentionally record the current tab, for example recording gameplay or video demonstrations. Such applications would normally not show the local user a preview, which means they don’t have a problem with HoM.
22
Hall of Mirrors - Suggested Solution (Elad)
Recall that getDisplayMedia() receives the following dictionary.
We could extend it:
dictionary DisplayMediaStreamConstraints {
(boolean or MediaTrackConstraints) video = true;
(boolean or MediaTrackConstraints) audio = false;
boolean excludeCurrentTab;
};
The default behavior remains unchanged (in all browsers currently supporting tab-capture). But applications can trigger the new behavior, which avoids HoM by denying the user the possibility of selecting the current tab.
23
Hall of Mirrors - Security Discussion (Elad)
“That influences user choice! It could be abused for social engineering!”
We have discussed (often) the dangers of allowing an application to push the user towards either:
The suggested solution does neither. There is no degradation in security.
24
Hall of Mirrors - Default Value (Elad)
Three options for a default value:
I suggest we go with #3. This is a hint. It may be provided or omitted, and the user agent MAY regard or ignore it.
If a value for includeCurrentTab is specified, it is a hint. The user agent MAY regard this hint when deciding whether to include the current tab in the list of surfaces it offers to the user.
25
Hall of Mirrors - Potential Scope Expansion (Elad)
It is possible to go beyond tab-capture.
The security argument made in an earlier slide still holds, albeit less patently.
26
Discussion (End Time: 09:00 AM)
27
Display Surface Hints (Elad)
getViewportMedia (Jan-Ivar)
Start Time: 09:00 AM
End Time: 09:20 AM
28
Display Surface Hinting - Let’s Resolve (Elad)
We have been discussing displaySurface-hints for a long. The latest manigestation of this discussion is issue #184, which has been under discussion for 9 months by now.
This is a highly-requested control knob. We should be able to accommodate Web-developers in a timely manner with such small changes. We should converge on the least controversial proposal.
User agents MAY change the order or prominence of offered choices in response to an application's preference, as indicated by the {{displaySurface}} constraint.
Let’s ship it.
29
getViewportMedia() update: ready for Call for Adoption
Document up at https://w3c.github.io/mediacapture-viewport/ (UD)
Recap: (resolutions from April & September)
Discussion (End Time: 09:20 AM)
31
MediaCapture-Extensions
Start Time: 09:20 AM
End Time: 09:50 AM
32
PRs for Discussion
33
34
dictionary DetectedFace {
required float probability;
FrozenArray<DetectedFaceLandmark> landmarks;
};
partial dictionary MediaTrackSupportedConstraints {
boolean faceDetectionMode = true;
boolean faceDetectionLandmarks = true;
boolean faceDetectionMaxNumFaces = true;
boolean faceDetectionNumContourPoints = true;
boolean faceDetectionNumLandmarkPoints = true;
};
Face detection example
35
// Check if face detection is supported by the browser
const supports = navigator.mediaDevices.getSupportedConstraints();
if (supports.faceDetectionMode &&
supports.faceDetectionNumContourPoints) {
// Browser supports face contour detection.
} else {
throw('Face contour detection is not supported');
}
// Open camera with face detection enabled
const stream = await navigator.mediaDevices.getUserMedia({
video: {
faceDetectionMode: 'contour',
faceDetectionNumContourPoints: {exact: 4}
}
});
const [videoTrack] = stream.getVideoTracks();
// Use a video worker and show to user.
const videoElement = document.querySelector('video');
const videoWorker = new Worker('video-worker.js');
videoWorker.postMessage({track: videoTrack}, [videoTrack]);
const {data} = await new Promise(r => videoWorker.onmessage);
videoElement.srcObject = new MediaStream([data.videoTrack]);
// video-worker.js:
self.onmessage = async ({data: {track}}) => {
const generator = new VideoTrackGenerator();
parent.postMessage({videoTrack: generator.track}, [generator.track]);
const {readable} = new MediaStreamTrackProcessor({track});
const transformer = new TransformStream({
async transform(frame, controller) {
for (const face of frame.detectedFaces) {
console.log(
`Face @ (${face.contour[0].x}, ${face.contour[0].y}), ` +
`(${face.contour[1].x}, ${face.contour[1].y}), ` +
`(${face.contour[2].x}, ${face.contour[2].y}), ` +
`(${face.contour[3].x}, ${face.contour[3].y})`);
controller.enqueue(frame);
}
}
});
await readable.pipeThrough(transformer).pipeTo(generator.writable);
};
PR 49: Background Concealment Blur
36
const stream = await
navigator.mediaDevices.getUserMedia({video: true});
const [videoTrack] = stream.getVideoTracks();
// Try to conceal background.
const videoCapabilities = videoTrack.getCapabilities();
if (videoCapabilities.backgroundBlur) {
await videoTrack.applyConstraints({
advanced: [{backgroundBlur: videoCapabilities.backgroundBlur.max}]
});
} else {
// Background concealment is not supported by the platform or by the camera.
// Consider falling back to some other method.
}
// Show to user.
const videoElement = document.querySelector("video");
videoElement.srcObject = stream;
value of 0.0 indicates no background blur and increasing values indicate increasing background blur
PR 57: Face detection, BG blur, etc example
37
const generator = new VideoTrackGenerator();
parent.postMessage({videoTrack: generator.track}, [generator.track]);
const {readable} = new MediaStreamTrackProcessor({track});
const transformer = new TransformStream({
async transform(frame, controller) {
// Detect faces or retrieve detected faces.
const detectedFaces =
customFaceDetection
? await detectFaces(frame)
: frame.detectedFaces;
// Blur the background if needed.
if (customBackgroundBlur) {
const newFrame = await blurBackground(frame, detectedFaces);
frame.close();
frame = newFrame;
}
// Correct the eye gaze if needed.
if (customEyeGazeCorrection && (detectedFaces || []).length > 0) {
const newFrame = await correctEyeGaze(frame, detectedFaces);
frame.close();
frame = newFrame;
}
controller.enqueue(frame);
}
});
await readable.pipeThrough(transformer).pipeTo(generator.writable);
};
PR 53: Lighting Correction
38
Lighting correction is a boolean setting controlling whether face and background lighting balance is to be corrected.
PR 55: Face Framing
39
Face framing is a boolean setting controlling whether framing is to be improved by cropping to faces.
PR 56: Eye gaze correction
40
Eye gaze correction is a boolean setting controlling whether the eye gaze is to be corrected.
Discussion (End Time: 09:50 AM)
41
Thank you
Special thanks to:
WG Participants, Editors & Chairs
42