W3C WebRTC
WG Meeting
January 19, 2021
8:00 AM - 9:30 AM Pacific Time
1
Chairs: Bernard Aboba
Harald Alvestrand
Jan-Ivar Bruaroey
W3C WG IPR Policy
2
Welcome!
3
A Request...
4
About this Virtual Meeting
5
Issues for Discussion Today
6
Issues for Discussion Today
7
Raw Media Insertable Streams Status Report (Harald)
Beyond Insertable Streams
Encoded data - next steps towards the Lego Laser
Insertable Streams is a wires-out-the-side model
This means that in addition to the exposed stream, there is a reverse stream of feedback data that is not shown to the user
It is not arbitrary where to connect the wires - JS can’t create frames, and has to send frames back to the sender that created them.
This model was explicitly abandoned for Breakout Box (nonencoded data)
A Chained Boxes model is cleaner and more flexible
Requires the feedback channel to be explicit
In Encoded Data, a lot more feedback is needed than for non-encoded
We also should not delay dealing with the SDP issue any longer
First Sketch of An API
interface RTCRtpBaseSender {� // Transport and stats attributes�}
interface RTCRtpSender : RTCRtpBaseSender {� // everything to do with tracks goes here�}
interface RTCRtpFrameSender : RTCRtpBaseSender {
Stream writable; // Stream of EncodedFrame and control objects� Stream readableControl; // Stream of control objects�}��In RTCRtpTransceiver, the “sender” attribute has type (RTCRtpFrameSender or RTCRtpSender)
Requesting and Receiving Frame Streams
When sending, we can use AddFrameStream() rather than AddTrack.
When receiving, we can set a configuration attribute (per type) called whatSignalDoIWant with possible values “track” or “stream-of-frames”; one leads to firing ontrack; the other leads to firing onframestream, with appropriate parameters.
The SDP question
When the format of the data is defined by the application, the SDP negotiated has to be cared for by the application too. For instance, in a pipeline that goes
WebCodec(AV2) -> PeerConnection
Negotiating any codec but AV2 would be silly - but the codec negotiated also defines the encapsulation, which is different from the frame format.
Suggested:
A possible application example: Jitter buffer in JS
Issue #48 - WebRTC insertable streams as transform (Youenn)
Ongoing Safari experiment
�Experimental API allows attaching transforms to senders and receivers
typedef (SFrameTransform or RTCRtpScriptTransform) RTCRtpTransform;
partial interface RTCRtpSender {
attribute RTCRtpTransform? transform;
};
partial interface RTCRtpReceiver {
attribute RTCRtpTransform? transform;
};
Issue #48 - WebRTC insertable streams as transform (Youenn)
Summary from last meeting
Questions that were raised
Let's look at JS examples to answer these questions
#48 - WebRTC insertable streams - Example 1
RTCRtpScriptTransform in window
RTCRtpScriptTransformer in worker
// HTML page�const pc = new RTCPeerConnection();�const worker = new Worker("worker-module.js");��const sender = pc.addTrack(track, stream);�sender.transform = new RTCRtpScriptTransform(worker, "myEncryptFrameTransform");�sender.transform.port.postMessage("Hello Transformer");�sender.transform.onmessage = (e) => console.log(e.data);
// worker-module.js - stream based variant�function encryptFrame(frame)�{� ...�}�onrtctransform = (e) => {� const transformer = e.transformer;�z}�function createTransform()�{� return new TransformStream({ transform : encryptFrame });�}
// worker-module.js - frame based variant�function encryptFrame(frame)�{� ...�}�onrtctransform = (e) => {� const transformer = e.transformer;� transformer.port.postMessage("Starting");� transformer.onframe = async (e) => {� transformer.write(await encryptFrame(e.frame)); � }�}
#48 - WebRTC insertable streams - Example 2�RTCRtpScriptTransform in worker, stream transferable
Dedicated RTCRtpScriptTransform context
// HTML page�const pc = new RTCPeerConnection();�const worker = new Worker("worker-module.js");��const transformer = await new Promise(resolve => worker.onmessage = (e) => resolve(e.data);�const sender = pc.addTrack(track, stream);�sender.transform = transformer;�
// worker-module.js - transform based variant�function encryptFrame(frame)�{� ...�}�async function transformFrame(frame, context)�{� context.enqueue(await encryptFrame(frame));�}�const transform = new RTCRtpScriptTransform({ transform });�self.postMessage(transform, [transform]);
#48 - WebRTC insertable streams - Example 3
Extending API for feedback control and more
// HTML page�const pc = new RTCPeerConnection();�const worker = new Worker("worker-module.js");��const sender = pc.addTrack(track, stream);�sender.transform = new RTCRtpScriptTransform(worker, "myEncryptFrameTransform");�sender.transform.port.postMessage("Hello Transformer");�sender.transform.onmessage = (e) => console.log(e.data);
// worker-module.js�onrtctransform = (e) => {� const transformer = e.transformer;� transformer.port.postMessage("Starting");� transformer.readable.pipeThrough(createTransform(transformer))� .pipeTo(transformer.writable);� transformer.onbitratechange = (e) => {� ...� };�}�
// worker-module.js�function encryptFrame(frame, transformer)�{� if (needsKeyFrame && frame.type != 'key') {� tranformer.requestKeyFrame();� return;� }� ...�}�function createTransformer(transform)�{� return new TransformStream({ transform :� frame => encryptFrame(frame, transformer)� });�}
#48 - WebRTC insertable streams - Evaluation
Expose streams or frames in workers
Expose a feedback channel
Evaluate API complexity
#48 - WebRTC insertable streams - Proposal
Proposal 1
Proposal 2
Proposal 3
Elad Alon
Would introducing one specific video-editing capability in the browser actually make sense?
Cropping MediaStreamTracks
Previously on WEBRTC WG, 90210
The benefits and drawbacks of adding video-editing capabilities to the browser have been discussed in the past. A new argument needs to be made, in order for this discussion to be worth our time.
I believe I have such an argument, which applies to one particular video-editing capability - cropping.
The Browser’s Mandate [Citation Needed]
Interesting Use-Case
Consider this application for editing and displaying slides. It shares a cropped version of itself with remote users.
It may also display to the local user some private content, such as speaker notes.
The application’s promise to the user:
The user’s private content (e.g. speaker notes) will be cropped away. Consistently and robustly. Even if the user resizes the window or changes the zoom-level. Not a single frame containing private information will ever be shared remotely.
The browser must ensure the application has the means to deliver on such a reasonable promise.
This Particular Promise is Hard to Keep
Insufficient “Solution” - OnResize Event Handlers
Registering OnResize event handlers - not a solution.
Insufficient “Solution” - Buffering
The application could buffer the frames. It can wait to ensure no resize-event is received shortly after a frame is captured, then consume the frame by forwarding it to a PeerConnection.
This is not a good solution. It would introduce delay. (Noteworthy: the shorter the delay - the less robust the guarantee by the application.)
Conclusion: The Browser Must Intervene
The browser should provide the application either of the following:
Shape of APIs to Come
Avoiding particulars, the general shape we envision for the API is:
Code Sample
<div crop_id=’b9eb9d62’> content… </div>
let ms = await navigator.mediaDevices.thisTabMedia();
let mst = await ms.getVideoTracks();
mst.cropTo(‘b9eb9d62’); // Or programmatically.
Note that the crop ID ‘b9eb9d62’ is either defined in the context in which cropTo() is called, or it can be transmitted there via postMessage(), etc.
It is up to debate whether the crop_id should be (a) an unguessable token or (b) a user-defined string, with any simple string being fair game.
Issues for Discussion Today
33
Issue 52: Invalid TURN credentials: What Function Should Fail? (Henrik)
TURN credentials are set with pc.setConfiguration(). For non-parse errors like invalid credentials or unable to reach host, errors would only be discovered later.
Problem: Not clear if/where invalid TURN credential failures are surfaced.
pc.onicecandidateerror already covers unable to reach server:
If no host candidate can reach the server, errorCode will be set to the value 701 which is outside the STUN error code range. This error is only fired once per server URL while in the RTCIceGatheringState of "gathering".
Proposal:
Issue 63: Enabling opus stereo audio without SDP munging (stereo=1) (Henrik)
SDP munging is currently required to send stereo audio.
Problem:
Issue 63: Enabling opus stereo audio without SDP munging (stereo=1) (Henrik)
Proposal:
Q: What if I want mono? Do I have to SDP munge stereo=0?
A: No, use getUserMedia({audio:{channelCount:1}}), WebAudio, etc.
Follow-up:
Issue 230: Add support for WebRTC Data Channel in Worker
Web sites do use data channel to transmit data but process the data in workers
Potential solution: make data channels transferable
Reduced problem scope
Issue 230: Add support for WebRTC Data Channel in Worker
What is needed to make data channels transferable?
Specification check
Implementation check (based on webrtc.org code base)
Issue 230: Add support for WebRTC Data Channel in Worker
Conclusion
Alternative
Is there interest in any of these possibilities?
For extra credit
40
Name this bird!
Thank you
Special thanks to:
WG Participants, Editors & Chairs
The bird
41