W3C WebRTC WG
TPAC Meeting
September 24, 2024
09:00 - 12:30
1
Chairs: Bernard Aboba
Harald Alvestrand
Jan-Ivar Bruaroey
W3C WG IPR Policy
2
W3C Code of Conduct
3
Safety Reminders
While attending TPAC, follow the health rules:
Please be aware of and respect the personal boundaries of your fellow participants
For information about mask and test availability, what to do if you are not feeling well, and what to do if you test positive for COVID, see:
4
About Today’s Meeting
5
Other TPAC 2024 Meetings of Interest
6
For Discussion Today
Time control:
7
Virtual Meeting Tips (Zoom)
8
State of the WG (Harald)
Start Time: 09:10 AM
End Time: 09:30 AM
9
External Environment
10
Activity since TPAC 2023
11
Implementation activity
12
Things that seem stable (and used)
13
Major new or expanded topics
14
Discussion (End Time: 09:30)
15
WebRTC-PC Recycling to REC (Dom)
Start Time: 09:30 AM
End Time: 09:50 AM
16
17
Amendments aligning with existing implementations
18
Untestable amendments
Unobservable IDL changes:
Untestable in WPT:
19
Interoperable modifications
20
Next steps
(this can be done every 6 months if useful)
21
Discussion (End Time: 09:50)
22
Codec Issues (Henrik Boström & Harald)
Start Time: 09:50 AM
End Time: 10:10 AM
23
Making Codecs Transceiver-specific (Harald)
24
Transceiver Specific Codecs (Harald)
25
Codec directionality (Henrik)
Codecs on sendrecv m= section are “what we can receive”*
*fine print: While this is what we prefer to receive (not send),�this may still influence what we send! Important for backwards compatibility.
Conclusion: Preferences can include both send and receive codecs.
26
Codec directionality (H..
RFC 3264 section 5.1 to the rescue…? TL;DR:
Unicast: codec filter on direction is straightforward.
SendRecv: can we mix sendrecv with recvonly codecs?
27
Codec directionality (Henrik)
We should: filter codec preferences based on direction.
Proposal A: Avoid footgun.
Proposal B: Support “{H265, H264}”.
Proposal C: Max power, max footgun.
Proposal D: Change the rules.
28
Discussion (End Time: 10:10)
29
IceTransport Extensions (Sameer)
Start Time: 10:10 AM
End Time: 10:30 AM
30
Issue 209 - App control over ICE checks
Allow an App to observe, influence, and initiate outgoing ICE checks
But first, lots of questions…
And then…
31
How do ICE checks work today?
32
How could things be improved?
Example scenario:
33
Possible with existing API or stats?
34
Possible to express with config?
35
What is the new API?
partial interface RTCIceTransport {
// Send an ICE check.
Promise<RTCIceCheckRequest> checkCandidatePair(RTCIceCandidatePair pair);
// Fired before ICE agent sends an ICE check, cancellable.
attribute EventHandler /* RTCIceCheckEvent */ onicecandidatepaircheck;
}
interface RTCIceCheckEvent : Event { // Cancellable
readonly attribute RTCIceCandidatePair candidatePair;
// Resolves when the check is actually sent. Rejected => send failure.
readonly attribute Promise<RTCIceCheckRequest> request;
}
interface RTCIceCheckRequest {
readonly attribute ArrayBuffer transactionId;
readonly attribute DOMHighResTimeStamp sentTime;
// Resolves when response is received. Rejected => timeout.
readonly attribute Promise<RTCIceCheckResponse> response;
}
interface RTCIceCheckResponse {
readonly attribute DOMHighResTimeStamp receivedTime;
readonly attribute boolean retransmitted;
// No error => success.
readonly attribute RTCIceCheckResponseError? error;
}
36
Mechanics of the API
37
How to use the new API?
const pc = …;
const ice = pc.getTransceivers()[0].sender.transport.iceTransport;
ice.onicecandidatepaircheck = async(event) => {
if (shouldNotCheck(event.candidatePair)) {
event.preventDefault(); // prevent a check
return;
}
const request = await event.request;
handleCheck(request);
}
// send a check
const request = await ice.checkCandidatePair(alternatePair);
handleCheck(request);
function handleCheck(request) {
try {
const response = await request.response;
if (response.error) {
// … do something with error …
return;
}
const rtt = response.receivedTime - request.sentTime;
if (!response.retransmitted) {
// … do something with rtt …
}
} catch(error) {
// … do something with timeout …
}
}
38
Future extensibility
partial interface RTCIceCheckRequest { partial interface RTCIceCheckResponse {
readonly attribute RTCStunAttributes attributes; readonly attribute RTCStunAttributes attributes;
} }
dictionary RTCStunAttributes { dictionary RTCUnrecognizedStunAttribute {
boolean useCandidate; required ArrayBuffer type;
sequence<ArrayBuffer> unknownAttributes; ArrayBuffer value;
sequence<RTCUnrecognizedStunAttribute> unrecognizedAttributes; }
}
Promise<RTCIceCheckRequest> checkCandidatePair(RTCIceCandidatePair pair, optional RTCStunAttributes attributes);
Promise<RTCIceCheckRequest> checkCandidatePair(RTCIceCandidatePair pair, optional double timeout, optional unsigned short maxRetransmissions);
39
Discussion (End Time: 10:30)
40
Break
Start Time: 10:30 AM
End Time: 11:00 AM
41
Encoded Source (Guido Urdaneta)
Start Time: 11:00 AM
End Time: 11:20 AM
42
Use Case
43
Scenario
44
Server
RTCRtpEncodedSource proposal
45
Input PC1
Input PC2
Output PC1
P2P Node
Use Encoded Transform to read frames from Input PCs
Custom processing to produce output frames (discard duplicates, adjust metadata, append app-specific metadata,...)
Write to Output PCs using
Encoded Source
To
Peer O1
From
Peer I1
From
Peer I2
Output PC1
To
Peer O2
Original proposal
46
Feedback from original proposal
47
Basic example
// main.js
const worker = new Worker('worker.js');
// Let relayPCs be the set of PCs used to relay frames.
for (let pc of relayPCs) {
const [sender] = pc.getSenders();
await sender.createEncodedSource(worker); // similar to replaceTrack()
}
// Let recvPc1, recvPc2 be the receiving PCs.
recvPc1.ontrack = ({receiver}) =>
receiver.transform = new RTCRtpScriptTransform(worker);
recvPc2.ontrack = ({receiver}) =>
receiver.transform = new RTCRtpScriptTransform(worker);
48
Basic example
// worker.js
let sourceWriters = [];
onrtcsenderencodedsource = ({source: {writable}}) => {
sourceWriters.push(writable.getWriter());
}
onrtctransform = async ({transformer: {readable, writable, options}}) => {
function transform(frame, controller) {
if (shouldForward(frame)) { // app-defined (e.g., drop duplicates)
for (let writer of sourceWriters)
writer.write(getOutputFrame(frame)); // app-defined (e.g., adjust metadata)
}
controller.enqueue(frame); // put original frame back in receiver
}
await readable.pipeThrough(new TransformStream({transform})).pipeTo(writable);
}
49
Errors and signals - developer feedback
50
Signals - Key Frame requests
[Exposed=DedicatedWorker] interface RTCRtpSenderEncodedSource {
// Accepts RTCRtpEncoded{Video|Audio}Frame, rejects on incorrect frames
readonly attribute WritableStream writable;
attribute EventHandler onkeyframerequest;
};
51
Bandwidth signals
52
Bandwidth and other signals
[Exposed=DedicatedWorker] interface BandwidthInfo {
readonly attribute long allocatedBitrate; // bits per second
readonly attribute long availableOutgoingBitrate;
};
[Exposed=DedicatedWorker] interface RTCRtpSenderEncodedSource {
...
readonly attribute BandwidthInfo bandwidthInfo;
attribute EventHandler onbandwidthestimate;
readonly attribute unsigned long long droppedFrames;
readonly attribute double expectedSendQueueTime; // milliseconds
};
53
Example
// worker.js
async function maybeRelayFrame(frame, writer, bandwidthInfo) {
// Append extra redundancy data to the payload if there is enough bandwidth
if (bandwidthInfo.allocatedBitrate > kMinBitrateForExtraRedundancyData) {
appendExtraData(frame);
}
await writer.write(frame);
}
54
API Shape
partial interface RTCRtpSender {
Promise<undefined> createEncodedSource(
Worker worker, optional any options, optional sequence<object> transfer);
}
partial interface DedicatedWorkerGlobalScope {
attribute EventHandler onrtcsenderencodedsource;
}
[Exposed=DedicatedWorker] interface RTCRtpSenderEncodedSourceEvent : Event {
readonly attribute RTCRtpSenderEncodedSource encodedSource;
};
55
API Shape
interface BandwidthInfo {
readonly attribute long allocatedBitrate; // bits per second
readonly attribute long availableOutgoingBitrate;
};
[Exposed=DedicatedWorker] interface RTCRtpSenderEncodedSource {
readonly attribute WritableStream writable; // Accepts RTCRtpEncoded{Video|Audio}Frame
attribute EventHandler onkeyframerequest;
readonly attribute BandwidthInfo bandwidthInfo;
attribute EventHandler onbandwidthestimate;
readonly attribute any options;
};
56
Pros and cons
57
Discussion (End Time: 11:20)
58
Timing Model I (Bernard, Markus & Youenn)
Start Time: 11:20 AM
End Time: 11:40 AM
59
For Discussion Today
60
Issue 87: What is the timestamp value of the VideoFrame/AudioData from a remote track? (Bernard/Markus)
61
62
63
64
dictionary VideoFrameCallbackMetadata {
required DOMHighResTimeStamp presentationTime;
required DOMHighResTimeStamp expectedDisplayTime;
required unsigned long height;
required unsigned long presentedFrames;
DOMHighResTimeStamp captureTime;
DOMHighResTimeStamp receiveTime;
};
65
Issue 96: What is the impact of timestamp for video frames enqueued in VideoTrackGenerator? (Youennf)
66
Issue 96: What is the impact of timestamp for video frames enqueued in VideoTrackGenerator? (Youennf)
Issue 80: Expectations/Requirements for VideoFrame and AudioData timestamps (Bernard)
67
68
Issue 86: Playback and sync of tracks created by VideoTrackGenerator (Bernard/Markus)
69
Issue 86: Playback and sync of tracks created by VideoTrackGenerator (Cont’d)
Discussion (End Time: 11:40)
70
RtpTransport (Peter Thatcher)
Start Time: 11:40 AM
End Time: 12:00 PM
71
Reminder about use cases
Control/customization of:
Example Use Cases
Status Update
Things That Have Been Figured Out
Things In Progress
Conclusion
Discussion (End Time: 12:00)
78
Corruption Stats + Encoder Complexity (Erik Språng)
Start Time: 12:00 PM
End Time: 12:20 PM
79
Stats issue 787: Corruption Likelihood Metric
The purpose is to provide a metric that indicates the estimated probability that an inbound video stream is experiencing corruptions.
We’re targeting outright bugs that cause visual artifacts that is otherwise not visible in any existing stats, it is not intended as general quality metric. The typical use case is finding problems early and be able to root cause it to e.g. browser versions, hardware setups, configuration/experiment rollouts - without having to rely on user feedback reports.
80
double totalCorruptionProbability;
double totalSquaredCorruptionProbability;
unsigned long long corruptionMeasurements;
Stats issue 787: Corruption Likelihood Metric
Example implementation: http://www.webrtc.org/experiments/rtp-hdrext/corruption-detection
RTP header extension used as side-channel, transmitting randomly selected image samples for validation
81
Stats issue 787: Corruption Likelihood Metric
82
Packet
Extension
With the encoded images, piggy-back a few (13 with one-byte extensions) randomly selected samples as raw values.
Stats issue 787: Corruption Likelihood Metric
83
Packet
Extension
Compare to the raw decoded samples values on the receive side.
Stats issue 787: Corruption Likelihood Metric
84
Large QP - average over large area
Low QP - average over small area
The extension has fields to indicate filter size and allowed error thresholds, so that expected distortions from lossy compression can be suppressed.
Stats issue 787: Corruption Likelihood Metric
85
The stats value is not intended to be tightly coupled to this implementation.
Described as probability of corruption in range [0.0, 1.0] to be generic enough to be used with other implementations.
Future iterations could include:
Stats issue 787: Corruption Likelihood Metric
86
Issue 191: Add API to control encode complexity
87
enum RTCEncodeComplexityMode {
"low",
"normal",
"high"
};
partial dictionary RTCRtpEncodingParameters {
RTCEncodeComplexityMode encodeComplexityMode = "normal";
};
Specifies the encoding complexity mode relative to "normal" mode:
Issue 191: Add API to control encode complexity
88
Intended use cases:
Discussion (End Time: 12:20)
89
Wrapup and Next Steps
Start Time: 12:20
End Time: 12:30
90
Next Steps
91
Spillover
92
Use Case 2: SFU “Lip Sync”
93
94
95
Use Case 3: SFU Audio Sync
96
Timing Use Case II: BYOC
97
Timing Use Case 1 (WebRTC)
Packet Reception | RTP: Audio and Video for the Internet (flylib.com)
98
99
Mapping Media Timelines to a Common Clock
Packet Reception | RTP: Audio and Video for the Internet (flylib.com)
100
Synchronization of Media
Packet Reception | RTP: Audio and Video for the Internet (flylib.com)
101
Lip Sync at the Receiver
Packet Reception | RTP: Audio and Video for the Internet (flylib.com)
102
Thank you
Special thanks to:
WG Participants, Editors & Chairs
103