W3C WebRTC
WG Meeting
November 21, 2023
8 AM - 10 AM
1
Chairs: Bernard Aboba
Harald Alvestrand
Jan-Ivar Bruaroey
W3C WG IPR Policy
2
Welcome!
3
About this Virtual Meeting
4
W3C Code of Conduct
5
Virtual Interim Meeting Tips
This session is (still) being recorded
6
Understanding Document Status
7
Issues for Discussion Today
Time control:
8
Grab Bag
Start Time: 08:05 AM
End Time: 08:40 AM
9
For Discussion Today
10
Issue 121: Background Blur: Unprocessed video should be mandatory
11
Issue 129: [Audio-Stats] Disagreement about audio dropped counters
The WG decided to add the following metrics (April and October Virtual Interims):
Allows calculating dropped frames and glitch ratios, a measure of audio quality.
Drops can happen by OS:
Drops can happen by User Agent:
BUT… Audio glitches are rare and difficult for an app to affect on a case-by-case basis.
12
Issue 129: [Audio-Stats] Disagreement about audio dropped counters
There’s an alleged problem:
Google disagrees with both the premises and the conclusion. Apps needs to monitor quality:
Decision needed: Should audio frame drops be exposed to JS?
13
Issue 146: Exposing decode errors/SW fallback as an event
14
Issue 92: Align exposing scalabilityMode with WebRTC “hardware capabilities” check
15
Issue 92: Align exposing scalabilityMode with WebRTC “hardware capabilities” check
16
MediaCapabilities on MacBook Air M2 (Safari Tech Preview 183)
17
chrome://gpu info
Issue 92: Align exposing scalabilityMode with WebRTC “hardware capabilities” check
18
PR 212: Describe data
19
PR 175: Add RTCIceTransport method to remove pairs
Promise<undefined> removeCandidatePair(RTCIceCandidatePair pair);
20
Issue 2170: Data channel default binaryType value is 'blob'
21
Discussion (End Time: 08:40)
22
SDP Codec Negotiation (Harald)
Start Time: 08:40 AM
End Time: 09:00 AM
23
Issue 186: New API for SDP negotiation
After discussion with Jan-Ivar who thought the API was too complex, a revised proposal:
The utility helper merely executes the transceiver function for any newly created transceiver.�Functionality is the same as before.
Debated point: API on transceiver or on transform?
Present proposal - API on transceiver
Encoder
Transform
Packetizer
SDP negotiation module
Browser
capabilities
Offer/Answer
Packets
Application
Transceiver
API
Configuration
Packets
With API on transform
Encoder
Transform
Packetizer
SDP negotiation module
Browser
capabilities
Offer/Answer
Packets
Application
Transceiver
API
Configuration
Packets
Open issue: MIME type or PT?
28
Discussion (End Time: 09:00)
29
RtpTransport
Start Time: 09:00 AM
End Time: 09:20 AM
30
RtpTransport
31
RtpTransport
32
Review Current State
33
RtpTransport
34
BWE API
My proposal: Build BWE API on RtpTransport�(such as this simplified version of Harald's proposal)
partial interface RtpTransport {
attribute BandwidthInfo bandwidthInfo;
attribute EventHandler onbandwidthestimate;
};
partial interface BandwidthInfo {
readonly attribute long availableOutgoingBitrate;
};
35
BWE API
You may also want a way to get to it with existing m-lines:
partial interface RtpSender {
readonly attribute RtpTransport rtpTransport;
};
partial interface RtpReceiver {
readonly attribute RtpTransport rtpTransport;
};
36
RtpTransport
37
Forwarding API
My proposal: Use RtpTransport for forwarding�
38
RtpTransport
39
Using existing m-lines
40
What are you prevented from sending?
41
What are you prevented from sending?
42
What are you prevented from sending?
Proposal: allow disabling bumper lanes easily, and keep the bumper lanes simple and easy to implement
43
Is built-in demux worth it?
44
Is built-in demux worth it?
Proposal: build in RTP demux, but not RTCP demux�(most of the benefit for a fraction of the complexity)
�
45
How do we "mix" outgoing traffic?
46
RtpTransport
47
Workers
48
RtpTransport
49
WHATWG Streams
50
WHATWG Streams
RtpTransport.receiveRtcp({type: 206})
51
WHATWG Streams
Proposal: Don't use WHATWG streams right now, but maybe use it if/when have somewhere they make sense
52
RtpTransport
53
Bonus #1: "Modify" Packets
PC
Net
Web�App
X
to/from�Network
to/from�PC
Bonus #2: RTP over WebTransport
const wt = new WebTransport(url);
const writer = wt.datagrams.writable.getWriter();
let rtpPackets = myPacketizer.packetize(frame, mtu);
await sendRtpPackets(writer, rtpPackets);
async function sendRtpPackets(writer, rtpPackets) {
for (const rtpPacket of rtpPackets) {
await writer.ready;
writer.write(rtpPacket.toArrayBuffer()).catch(() => {});
}
}
Discussion (End Time: 09:20)
56
Three Thumbs Up - Mute (Elad & Guido)
Start Time: 09:20 AM
End Time: 09:40 AM
57
Three Thumbs Up - Setup
What happens when multiple entities offer similar functionality to the user?
Entities:
Functionality:
58
Users’ Mental Model
59
Reality
60
I spoke but nobody could hear me.
I was not muted.
61
Definition: Upstream
For media:
Or:
HW -> OS -> UA -> Web app
62
Ideally…
63
Prioritization
Exposing upstream-state comes before changing it.
Generally speaking, if you don’t know what the upstream state is, when will you ask to change it?
Concretely for mic-mute, if the Web app doesn’t know the user muted the mic in the OS, how will it solicit a user’s gesture to unmute?
64
The `mute` event
Would the existing `mute` event suffice? No.
65
Proposal
enum MuteSource {"unspecified", "user-agent", "operating-system"};
interface MuteReason {
readonly MuteSource source;
readonly boolean potentiallyActionable; // Explanation follows.
};
partial interface MediaStreamTrack {
sequence<MuteReason> getMuteReasons();
};
66
Multiple Reasons
Note that the previous slide anticipated the possibility of multiple concurrent mute-reasons.
67
Privacy Concerns
Would exposing any of this information be problematic privacy-wise?
The one concrete concern mentioned thus far - incoming calls.
Solution - provide little information:
source == “operating-system”.
Worst-case scenario, the app shows suboptimal app-UX while the user is not looking, offering to unmute what cannot be muted.
Any other concerns?
68
requestUnmute()
Leaving the topic of state-exposure, let’s briefly discuss state-control.
partial interface MediaStreamTrack {
Promise<undefined> requestUnmute();
};
Requires user-gesture.
May result in a user-agent prompt, an operating-system prompt, or both.
69
What is potentiallyActionable?
Web applications don’t want to offer users impossible actions[*].
[*] But it’s acceptable, albeit suboptimal, to do so while the users aren’t looking, as is likely the case when the users receive a phone call.
70
Discussion (End Time: 09:40)
71
Mediacapture-screen-share (Elad)
Start Time: 09:40 AM
End Time: 09:55 AM
72
For Discussion Today
73
Contradictory hints - example #1
// Audio generally not requested,
// but system-audio marked as desired.
navigator.mediaDevices.getDisplayMedia({
audio: false,
systemAudio: "include",
});
74
Contradictory hints - example #2
// Audio requested, including an explicit
// request for system-audio,
// but monitors asked to be excluded.
navigator.mediaDevices.getDisplayMedia({
audio: true,
systemAudio: "include",
monitorTypeSurfaces: "exclude"
});
75
Contradictory hints - example #3
// Application requested monitors to be
// displayed most prominently,
// while simultaneously asking for monitors
// to not be offered.
navigator.mediaDevices.getDisplayMedia({
video: { displaySurface: "monitor" },
monitorTypeSurfaces: "exclude"
});
76
Handling of contradictory hints - proposal
Above is a snapshot of the getDisplayMedia algorithm in the spec.
Proposal:
77
Issue 281: Missing OS permissions - problem description
Sometimes screen-sharing is blocked because the user presses cancel; maybe the user changed their mind.
Sometimes screen-sharing is blocked because the OS permissions are configured to block. In that case, some applications might wish to explain that to the user.
78
Issue 281: Missing OS permissions - proposal
79
(The ctor sets the name attribute to the value “NotAllowedError”.)
Issue 281: Possible objection
Possible objection: “Shouldn’t the UA inform the user?”
Answers:
80
Discussion (End Time: 09:55)
81
WebRTC Extended Use Cases
Start Time: 09:40 AM
End Time: 09:55 AM
82
For Discussion Today
83
Status of Section 3.2: Low Latency Streaming
84
Section 3.2.2: Low Latency Broadcast w/fanout
85
PR 123: Use Case Goals
86
PR 123: Section 3.2.2: Clarify Use Cases
87
Section 3.2.1: Game Streaming
88
Section 3.2.1: Game Streaming
89
Issue #103: Feedback related to WebRTC-NV Low Latency Streaming Use Case
90
PR 125: Clarify Requirement N37
91
PR 118: Clarify Game Streaming requirements (Section 3.2.1)
92
PR 118: Clarify Game Streaming requirements (Section 3.2.1)
93
ID | Requirement | Description | Benefits to Cloud Gaming | Cloud Gaming Specific? |
N48 | Recovery using non-key frames | WebRTC must support a mode allows video decoding to continue even after a frame loss without waiting for a key frame. This enables addition of recovery methods such as using frames containing intra coded macroblocks and coding units - WebRTC Issue: 15192 | Players can continue to game with partially intelligible video. Fast recovery from losses on the network | Can be used by any application where video corruption is preferred to video freezes |
N49 | Loss of encoder -decoder synchronicity notification | The WebRTC connection should generate signals indicating to encoder about loss of encoder-decoder synchronicity (DPB buffers) and sequence of the frame loss.(RFC 4585 section-6.3.3: Reference Picture Selection Indication) - Delete of RPSI (Mar/2017) | Fast recovery from losses on network. Helps application to choose right recovery method in lossy network. |
PR 118: Clarify Game Streaming requirements (Section 3.2.1)
94
ID | Requirement | Description | Benefits to Cloud Gaming | Cloud Gaming Specific? |
N50 | Configurable RTCP transmission interval | The application must be able to configure RTCP feedback transmission interval (e.g., Transport-wide RTCP Feedback Message). | Gaming is sensitive to congestion and packet loss resulting in higher latency. Consistent RTCP feedback helps application to adapt video quality to varying network (BWE and packet loss). | In general, short latency is very important, but consistent latency is even more important for the cloud gaming. |
N51 | Improve accuracy of Jitter buffer control | The user agent needs to provide the jitter buffer to account for jitter in the pipeline up to the frame render stage - WebRTC Issue: 15535 | Increases accuracy of jitter buffer adaptation and helps maintain consistent latency |
N48 Recovery using non-key frames
95
→ Is there enough consensus to add this requirement to WebRTC requirements list? Exactly how it is solved (if solvable) can be discussed later, we are working with the use cases and requirements in this document.
The application must be able to control video decoding to continue even after a frame-loss without waiting for a key frame. This enables fast recovery from lossy network conditions.
N49: Loss of encoder -decoder synchronicity notification
96
→ Ongoing discussions are about "how" to implement this. Is there consensus about the requirement. We would like to conclude the PR?
: The application must be able to generate signals that indicate to the encoder the loss of encoder-decoder synchronicity (DPB buffers) and the sequence of frame loss using the platform-agnostic protocols. This helps the application choose the right recovery method in a lossy network.
N50: Configurable RTCP transmission interval
→ Ongoing discussions are about "how" to implement this. Is there consensus about the requirement. We would like to conclude the PR? Why can't we have a requirement on enabling quicker reacting (transport wide) congestion control (presumably enabled by frequent RTCP reports)?
: The application must be able to configure RTCP feedback transmission interval (e.g., Transport-wide RTCP Feedback Message). This helps the application adapt the video quality to the varying network and maintain consistent latency.
97
N51:Improve accuracy of Jitter buffer control
→ Is there enough consensus to add this requirement to WebRTC requirements list?
The user agent needs to provide the jitter buffer to account for jitter in the pipeline up to the frame render stage. This helps the application adapt the video quality to the varying network and maintain consistent latency.
98
Discussion (End Time: 09:55)
99
Wrapup and Next Steps
Start Time: 09:55 AM
End Time: 10:00 AM
100
Title Goes Here
101
Thank you
Special thanks to:
WG Participants, Editors & Chairs
102
PR 123: Use Cases Removed
103